Half of Your Weighted List Is Not Weighted

A weighted list in which half the rows carry the same minimum weight is not a weighted list. It is a uniform list with a weighted prefix — and nothing in the file, in the schema, or in the test suite says so.

We were running one of those. In a corpus of Swedish given names, 997 of 1,865 female entries (53.5%) and 1,115 of 2,015 male entries (55.3%) sat at weight 1. Their true frequencies were not equal and not close to equal: inside that flattened block the real counts ran from 36 to 567 for women and 31 to 479 for men. A name held by 567 people was being drawn exactly as often as a name held by 36.

The file was valid. The weights summed correctly. The rows were sorted descending. Every validator passed with zero failures and zero warnings. The list looked, by every check we had, like a properly weighted list.

The mechanism: one byte

The weights had been quantised into the range 1–255 so that each one would fit in a byte. The transform is the obvious one:

weight = round(count * CEILING / max_count)
if (weight < 1) weight = 1          // never drop a row entirely

That produces a bucket of width max_count / CEILING. Any entry smaller than half a bucket rounds to zero and is then clamped to 1; any entry smaller than one and a half buckets rounds to 1 as well. So everything below 1.5 * max_count / CEILING lands on the floor, indistinguishable.

In our data the largest female count was 96,428. A bucket is therefore 96,428 / 255 = 378.1, and the collapse threshold is 1.5 × 378.1 = 567.2. The largest real count found on weight 1 was 567. The mechanism is not inferred; it reproduces the boundary to the digit.

Reversing the mapping shows the shape clearly:

The top of the list is fine. Weight 255 holds exactly one name; weight 100 holds one; weight 50 holds three. The head is represented with essentially full precision. Every bit of the damage is at the bottom, where the buckets are wider than the entire remaining range.

The size of the damage is set by the single largest entry

This is the part that makes the defect hard to anticipate, and it generalises to every fixed-ceiling quantisation.

The collapse threshold is 1.5 * max / CEILING. It does not depend on how many rows you have, how skewed the tail is, or how carefully you built it. It depends on one number: the largest value in the list. The most popular item in your data decides how much of the rest of your data ceases to exist as distinct.

We measured that directly, by discarding the top K rows and re-quantising what remained at the same ceiling of 255:

Twenty rows out of 1,865 — one percent of the file — are responsible for the other 53.5% being flattened. If you are quantising anything Zipf-shaped, which includes names, words, cities, products, error codes and almost every frequency table anybody ships, this is your situation. The ratio you have to represent is max / min, and for this list it is 2,679. A byte offers 255 levels. The arithmetic was lost before anyone chose the transform.

The other lever is the ceiling itself, and it is brutally non-linear:

Going from 255 to 1,023 — from one byte to ten bits — takes the collapse from 53.5% to 0.4%. The cost of the defect and the cost of the fix are wildly asymmetric, which is worth knowing before you spend an afternoon arguing about whether the tail matters.

Why nothing caught it

Here is the uncomfortable half of the story. We re-measured the list after replacing the quantised weights with the real counts, and almost every summary statistic you would reach for says nothing happened:

Top-ten share moved by 0.2 percentage points. Perplexity moved by 5%. The total variation distance between the broken distribution and the correct one is under two and a half percent. If your acceptance test is "does the aggregate shape look right", the broken file passes, and it passes convincingly.

It passes because aggregate measures are dominated by the head, and the head was never damaged. The tail holds 8.34% of the probability mass in the broken version against 7.14% in the correct one. Errors confined to 8% of the mass cannot move a summary statistic much, no matter how wrong they are inside that 8%.

But look at what is wrong inside it. We sampled pairs of names at random and asked how often two names with genuinely different frequencies had been assigned the same weight: 31.1% of all pairs for women, 33.2% for men. A third of the list lost its ordering entirely. Per name, the correction ranges from ×0.10 to ×1.52 — an order of magnitude for individual rows, inside a distribution whose global distance from the truth is 2.4%.

That is the general lesson and it is worth stating on its own line: a distribution-level metric cannot detect a defect confined to the tail, because the tail is by definition the part that contributes least to distribution-level metrics. If you audit weighted data only in aggregate, tail defects are invisible by construction — not because you were careless, but because you chose an instrument that averages over exactly the region where the problem lives.

Being honest about severity

Two things pull in the other direction and both belong here.

First, inside the collapsed block the true distribution was itself fairly flat. Measured as perplexity, the real counts of those 997 names give an effective 944 alternatives out of a possible 997 — 94.7% of perfectly uniform. Flattening a nearly-flat block to exactly flat is a smaller crime than the ×15.8 span makes it sound. The span is a statement about the two extreme rows, not about the block.

Second, quantisation preserved rank order for the head faultlessly and never inverted it anywhere; it only created ties. Ties are a weaker failure than inversions. A list that says "these 997 items are equally likely" is wrong; a list that says "the rarest is likelier than the commonest" would be worse.

So the honest summary is: the aggregate error is small, the per-item error is large, and which of those matters depends entirely on what the list is for. For picking a plausible name to display, a 2.4% distributional shift is nothing. For anything where an individual item's probability is the output — a sampler feeding a simulation, a weighted A/B allocation, a recommender's prior, a load balancer's shard weights — a third of your entries being tied at the floor is the whole story.

We fixed ours because the fix was free: the row set did not change, no entry was added or removed, and the stored counters simply became 32-bit. The list went from 86 distinct weights to 1,084.

The diagnostics

None of this requires access to the original source. Everything below can be computed from the shipped file alone, in a few lines, and any one of them would have caught our defect years earlier.

1. Distinct weights over row count. This is the single best signal and it is almost free.

distinct = count(unique(weights))
ratio    = distinct / rows

Ours read 86 distinct values across 1,865 rows — 4.6%. After the fix: 1,084 over 1,865, 58%. For frequency data derived from counts, a healthy ratio is tens of percent. Anything in low single digits means the weights are not counts; they are labels.

2. Share of rows on the minimum weight. A Zipfian list has some mass at the floor — there really are many rare items. What it does not have is a floor holding half the file.

floor_share = count(w == min(weights)) / rows

Above roughly 20% this needs an explanation. At 53.5% the answer is not "the data is like that".

3. The largest equivalence class. Group by weight and take the biggest group. Ours: 997 rows sharing one value, then 269 sharing the next, then 113. Three values covered 74% of the file.

4. The suspicious ceiling. If max(weights) is exactly 255, 127, 100, 1,000 or 65,535, the number is not a measurement. It is the edge of a container. This is the cheapest check of the lot and it is a near-certain diagnosis when combined with (1).

5. The dynamic range you actually need. Compute max / min over the source counts, before quantising. That is the number of distinct levels a faithful linear encoding requires. Ours was 2,679 against 255 available. Comparing those two numbers takes ten seconds and tells you the answer before you write the encoder.

6. The real span inside the floor class. This one needs the source, and it is the number that converts a suspicion into a measurement: join the shipped weights back to the original counts, take only the rows at the minimum weight, and report max / min of their true values. Ours: ×15.8 and ×15.5. That is the quantity of information destroyed, stated in the units anyone will care about.

A useful property of this list: checks 1–5 run on the artefact you ship, with no source access, no history and no context. They are the kind of check you can run over every data file in a repository in an afternoon, which is the only kind that actually gets run.

If you must quantise

Sometimes the byte is genuinely fixed — a wire format you do not own, an embedded target, a column type inherited from a schema you cannot migrate. The transform still does not have to be linear.

Quantise in log space. Frequencies are multiplicative; a linear scale spends its resolution where the data has none. round(k * ln(count)) distributes levels evenly across ratios instead of across absolute values, and 255 levels covering a factor of 2,679 gives about a 3% relative error everywhere rather than perfect precision at the top and total collapse at the bottom.

Or store the rank and keep the counts elsewhere. If the consumer only needs relative order and a rough shape, ship the order and reconstruct weights from a parametric curve.

Or split the field. A byte of mantissa and a nibble of exponent is still cheaper than a 32-bit counter and represents the range exactly.

And whichever you choose, record the dynamic range you are discarding. Write it in the file header, not in a commit message. "This column is quantised; the source spans a factor of 2,679 and this encoding resolves 255 levels" is a sentence that would have made the defect self-reporting.

The mistake was never "we used a byte". It was that the compression was lossy in a way nobody had quantified, and a lossy transform whose loss is unmeasured is indistinguishable from a correct one right up until somebody joins the output back to the input.

The checklist

1. Count distinct weights and divide by rows. Under a few percent means the weights are labels, not measurements.
2. Measure the share of rows sitting on the minimum weight. Half the file at the floor is not a data property.
3. Look for a ceiling that is a container boundary — 255, 127, 65,535, 100, 1,000. Measurements do not land on those.
4. Compute max / min on the source counts before choosing an encoding. That is the number of levels you need; compare it to the number you have.
5. After quantising, join back and report the real span inside the floor class. This is the only number that states the damage in the units that matter.
6. Do not accept aggregate agreement as proof. Top-10 share, entropy and total variation distance are all dominated by the head and were all within noise on a file where a third of the pairs had lost their ordering.
7. Quantise in log space if the range is multiplicative, which for anything Zipf-shaped it is.
8. Write the discarded range into the artefact. An unmeasured lossy transform looks exactly like a lossless one.
9. Remember that the head sets the floor. Twenty of 1,865 rows decided that 997 others would be tied. Nothing about that is visible in the twenty.

The defect had been shipping for as long as the corpus existed. It survived a validator that checks row counts, sort order, encoding, byte-order marks, weight sums against a 32-bit bound, and the top-10 concentration share against a population figure — 261 files, zero failures, zero warnings. Every one of those checks was correct, and none of them was looking at the only thing that was wrong: whether the numbers in the weight column were still numbers, or had become categories.

For the companion failure — where a zero in the data meant two different things and no structural validator could tell them apart — see When Zero Means Two Different Things. For the general method of auditing frequency data against its source, see How to Audit Name Frequency Data. For what the head of these distributions looks like when it is intact, see Surname Concentration Curves. For checks that pass because they never looked, see The Check That Goes Blind Exactly Where You Need It.

Data as of 2026-07-21

All figures were measured on 21 July 2026 against the corpus files on disk and against a pre-change backup of the same files, and were re-derived independently for this article rather than quoted from the change record.

Sources and notes:

• The Swedish figures are our own calculation. They are derived by us from given-name counts published by Statistics Sweden (SCB); the processed material, the weights and every percentage in this article are ours and not SCB's, and SCB is not the source of the processed figures. Only the underlying counts are attributable to the agency.
• Female list — 1,865 rows. Quantised: 86 distinct weights, minimum 1, maximum 255, sum 11,954, 997 rows (53.5%) on weight 1, top-10 share 13.05%. Real counts: 1,084 distinct weights, minimum 36, maximum 96,428, sum 4,445,328, top-10 share 13.28%.
• Male list — 2,015 rows. Quantised: 100 distinct weights, minimum 1, maximum 255, sum 14,744, 1,115 rows (55.3%) on weight 1, top-10 share 13.21%. Real counts: 1,074 distinct weights, minimum 31, maximum 81,754, sum 4,655,153, top-10 share 13.42%.
• The collapsed block — for the female list, the 997 rows stored at weight 1 have true counts from 36 to 567, a span of ×15.75; quartiles 228 / 291 / 393. For the male list, 1,115 rows from 31 to 479, span ×15.45. The block holds 8.34% of the quantised mass against 7.14% of the true mass (female) and 7.56% against 6.44% (male).
• The mechanism — reconstructed by joining quantised weights to true counts and comparing against w * max / 255. The mean true count per stored weight tracks the linear prediction to within about 1% at every level from 2 upward, and the observed maximum on weight 1 is 567 against a predicted collapse boundary of 567.2. The row set is identical before and after: 0 added, 0 removed, 0 renamed.
• Distributional distance — total variation distance between the quantised and true distributions is 2.446% (female) and 2.252% (male). Largest per-name increase ×1.52, largest decrease ×0.10, in both lists. Perplexity 542 → 514 and 482 → 456. Names required to reach half the mass: 92 → 89 and 76 → 74.
• Tied pairs — estimated by sampling 200,000 random pairs of rows from a fixed seed and counting pairs whose true counts differ but whose quantised weights are equal: 31.1% (female), 33.2% (male).
• Within-block flatness — perplexity of the true counts of the 997 floor rows is 944, i.e. 94.7% of the 997 that a perfectly uniform block would give; for the male block, 1,057 of 1,115, or 94.8%. This is the figure that keeps the ×15.8 span in proportion, and it is reported here because omitting it would overstate the defect.
• Ceiling and head sensitivity — both tables were computed on the true female counts by re-applying round(count * C / max) with a floor of 1, for each ceiling C, and by removing the top K rows before re-quantising. Dynamic range of the female list: max / min = 2,678.6; max / median = 188.
• Validator state — the corpus validator reports 261 files, 0 failures, 0 warnings, both before and after the change. It checks row counts, descending sort, UTF-8 without a byte-order mark, absence of carriage returns, weight sums within a 32-bit bound, and top-10 concentration against a population share. None of those is capable of detecting the defect described here, and that is not a criticism of the validator — it is the point of the article.
• What was changed — the weight column only. No entry was added, removed or reordered, and no other locale was touched. The stored counters are now 32-bit; the collapse threshold at that width is below 1 and no row can reach the floor by rounding.