← Blog

How many cosmic rays can GPT-2 survive?

October 27, 2026

Every so often a cosmic ray hits a memory chip and flips a bit. A language model is just its parameters sitting in memory, so I wanted to know how many flips it can take. I flipped uniformly random bits in GPT-2's 124M parametersWeights, biases and LayerNorm gains: about 2 billion bits in fp16. The input and output embeddings share one matrix, so it counts once. and checked whether it still worked, 2,580 times across six experiments.

Each parameter is a 16-bit float (fp16): scientific notation in base 2. One sign bit, five exponent bits that set the power of two, and ten mantissa bits that hold the significant digits. A mantissa flip nudges the digits; an exponent flip changes the scale.

One part of those parameters is the token embedding table. GPT-2 has a vocabulary of 50,257 tokens, and the table gives each one its own row of 768 numbers. It makes up about a third of the model, so many flips land in it. Each run records exactly which number every flip changed, so when a flip lands in this table we know which row it hit, and therefore which token. Here is one run, flip by flip:

What counts as broken

Each run is scored by perplexity on the first 4,096 tokens of the WikiText-2 test set:Four 1,024-token windows, so N = 4,092 predicted tokens.

PPL = exp( −¹⁄_N ∑_i log p(x_i | x_<i) )

Perplexity can be thought of as the number of equally likely tokens the model is choosing between. Guessing uniformly over the vocabulary scores 50,257; always being right scores 1. Clean GPT-2 scores 28.2.

A run counts as broken if its perplexity more than doubles or its logits contain NaN or Inf. The threshold barely matters, because the damage is all or nothing: 98.6% of runs ended within 10% of clean or more than 10× above it. The run above is typical: two harmless flips, then one that destroys it.

A dozen flips is enough

Out of about 2 billion bits, twelve random flips break GPT-2 half the time, and 300 broke every run. fp32 has twice the bits but only holds out to 22.

It's one bit

A mantissa flip changes a parameter by at most 50%, and the model doesn't notice. A sign flip rarely matters either. But flipping the top exponent bit multiplies the value by 2¹⁶, turning a typical 0.02 into 1,311, and that outlier propagates through every later layer. A single such flip breaks GPT-2 82% of the time.One flip of bit 14 in a random parameter, 50 runs. Since a random flip hits it with probability 1/16, about a dozen flips is the typical breaking point.

That's what happened in the run above: the lethal flip hit the embedding row for “ heated”. GPT-2 uses the same matrix to score its next token, comparing its final state against every row, so with one number in that row at 1,287 instead of 0.02, “ heated” wins every time.

Where the flip lands barely matters. LayerNorm is 0.03% of the parameters but breaks the model after about as many flips as the rest, so per bit it's roughly 1,000× more fragile. A newer model isn't tougher either: Qwen2.5-0.5B, four times the size, breaks after about 8.

A cheap fix

Since the damage comes from parameters becoming huge, the fix is to store each tensor's clean minimum and maximum and clamp back into that range, with NaNs set to 0.Those two numbers per tensor have to live somewhere safe, such as ECC memory. On the same flips, the median breaking point moves from 12 to 1,390.

Where the flips come from

The cosmic rays themselves are mostly protons, accelerated by supernova shock waves elsewhere in the galaxy. They never reach the ground: 15 to 20 km up, one shatters a nitrogen or oxygen nucleus, and the fragments shatter more, making an air shower of pions, muons, protons and neutrons. The neutrons are what reach memory. Uncharged, they don't lose energy ionizing the air, and they pass straight through a laptop. At sea level about 13 per square centimeter per hour are energetic enough to matter.Above 10 MeV, in New York City: the reference flux chipmakers use for soft-error testing (JEDEC JESD89).

Occasionally one hits a silicon nucleus in the chip and breaks it apart. The charged fragments tear through the crystal and free electrons along the way; a 5 MeV alpha particle frees over a million. A DRAM cell holds its bit as a few tens of thousands of electrons on a capacitor, so a nearby track can drain or fill it. Nothing is damaged, hence “soft” errors.The first ones were traced in 1978 at Intel, to alpha particles from trace uranium and thorium in chip packaging (May and Woods). A year later IBM's Ziegler and Lanford predicted cosmic rays would cause them too.

So how long would it last?

The answer combines one number from the experiment with one from the literature: GPT-2 breaks after a median of about 12 flips, as measured above, and published soft-error rates say how often flips arrive. Those rates are quoted in FIT, failures per billion device-hours, per megabit, and GPT-2 is 1,899 Mbit. At 0.56 FIT/Mbit, measured on production servers, a flip arrives every 107 years, so 12 of them take about 1,300. At 1,000 FIT/Mbit, typical of older lab studies, a flip arrives every 22 days, so GPT-2 lasts about nine months.Per-bit rates fell as cells shrank, and field measurements came in well below older beam tests.

Those are sea-level numbers. Higher up there's less air to absorb the shower: the flux is about 3.5× in Denver and 300× at cruising altitude. Move across the sky to see how long GPT-2 lasts in memory without error correction.Sea-level rates: 0.56 FIT/Mbit measured on production servers (Li et al., 2007) and 1,000 FIT/Mbit, within the range of older lab studies. Flux: Denver over 3× and 30,000 ft about 40× Denver (Actel, 2002), 40,000 ft 300× (NASA, 2014), interpolated in between. Orbit: 10⁻⁷ to 10⁻⁶ upsets per bit per day (Wang et al., 2024).

Latitude and the Sun matter too. Earth's magnetic field deflects cosmic rays most strongly at the equator, so the flux near the poles is roughly double, and at solar maximum the Sun's field cuts ground-level neutron counts by 10 to 20 percent. Orbit has no air at all: satellites take the primaries directly, and most of their upsets happen over the South Atlantic Anomaly, where the inner Van Allen belt dips to a few hundred kilometers.

Server memory has ECC, extra check bits on each 64-bit word that correct any single flipped bit, and DDR5 corrects single-bit errors inside each chip, so on the ground this is mostly a curiosity.Mostly. In a 2003 election in Schaerbeek, Belgium, a candidate received exactly 4,096 (2¹²) extra votes on an electronic voting machine, blamed on a single flipped bit. But the risk grows with size: a 70B model in bf16 has 560 times GPT-2's bits and, at the modern rate, takes a flip about every 70 days. In orbit, an unprotected model doesn't last the afternoon.

Code, data, rate sources and every run's flip list are on GitHub.