Features

The sampling coverage study

How often each sampling bound covers the true misstatement, by sample size and misstatements found, on generated populations: method, criterion and results.

Updated November 18, 2026 · 9 min read

The coverage study measures how often each sampling bound of the analytics engine covers the true misstatement. It builds populations whose misstatement is known to the cent, draws 2,000 samples from each one for every method and sample size, and counts the samples whose bound covers the truth. A 95 % upper bound should cover it in at least 95 % of the samples.

Its results decide what Blast evaluates: a method the study does not accept gives no bound. This page describes version 2 of the study, frozen on 23 October 2026, the latest version.

Populations and misstatements

Five population families, each built with 2,000, 20,000 and 200,000 items:

FamilyPopulation
P1Log-normal invoices (μ 7, σ 1.2, median about 1,100)
P2Heavy tail: 95 % log-normal with σ 1, 5 % large items (μ 9, σ 2.5)
P3Homogeneous: log-normal with σ 0.3
P4P1 with 5 % credit balances
P5Skewed balances (σ 1.9), with the largest 2.3 % of items left out, as if examined in full

Twelve misstatement patterns, planted in 0.2 % to 50 % of the items:

PatternMisstated itemsSize of each misstatement
E-unichosen at randoma share of the book value between 0 and 1
E-fullchosen at randomthe whole book value
E-smallchosen at randomup to 20 % of the book value
E-sizechosen in proportion to the amountbetween 0 and 1
E-invchosen in inverse proportion to the amountbetween 0 and 1
E-mixedchosen at random80 % overstatements, 20 % understatements up to half the book value
E-over1chosen at random10 % of them larger than the book value (variables estimators only)
E-strat-highall in the highest stratumbetween 0 and 1 (stratified designs only)
E-strat-lowall in the lowest stratumbetween 0 and 1 (stratified designs only)
E-archosen at randomoverstatements only: mostly the whole amount on small items, small shares on large ones
E-balchosen at randomsmall misstatements, as many in each direction
E-underchosen at randomtwice as many understatements as overstatements, some larger than the book value

A family is one population with one pattern, for example P2 with E-full.

Methods and sizes

Sample sizes 30, 50, 75, 100, 150, 200, 300, 500 and 1,000, at 95 % confidence, one-sided:

the Stringer bound with Poisson and with binomial factors, on fixed-interval selections;

the cell bound, on cell selections;

classical PPS, on both selections;

the mean-per-unit, difference, ratio and regression estimators;

attribute bounds, binomial and hypergeometric.

The monetary unit and variables methods are also measured stratified, with 3 and 5 strata, on P1, P2, P4 and P5.

Criterion

For each method, sample size n and number of misstatements found m, the study counts the samples of each family. It groups m in classes: 0, 1, 2, 3, 4, 5, 6–7, 8–10, 11–15, 16–20, 21–30, 31–50 and over 50.

A family judges a cell when the cell holds at least 1,000 samples with a bound.

It accepts the cell when the bound covers the truth in at least 95.0 % of them and the one-sided 95 % Clopper-Pearson lower bound of that coverage is at least 94.0 %.

A cell is in the validity domain when at least 3 families judge it and none rejects it.

The Stringer, cell and attribute bounds are judged over every m at once, by family and n, and must be accepted everywhere. Their cells given m are published but are not a condition. When populations with different misstatement rates are pooled, a sample that finds few misstatements comes mostly from draws that missed them, where the bound is too low. For that reason no bound of this kind passes cell by cell given m.

Results

Method: Stringer bound, Poisson factors · Lowest coverage, over every m, any family and n: 97.7 % · Validity domain: none needed

Method: Stringer bound, binomial factors · Lowest coverage, over every m, any family and n: 97.0 % · Validity domain: none needed

Method: Cell bound · Lowest coverage, over every m, any family and n: 97.7 % · Validity domain: none needed

Method: Stringer bound, stratified · Lowest coverage, over every m, any family and n: 99.9 % · Validity domain: none needed

Method: Cell bound, stratified · Lowest coverage, over every m, any family and n: 99.7 % · Validity domain: none needed

Method: Attribute bound, binomial · Lowest coverage, over every m, any family and n: 97.3 % · Validity domain: none needed

Method: Attribute bound, hypergeometric · Lowest coverage, over every m, any family and n: 96.3 % · Validity domain: none needed

Method: Classical PPS · Lowest coverage, over every m, any family and n: 17.5 % at n = 30, 63.0 % at n = 1,000 · Validity domain: empty

Method: Classical PPS, stratified · Lowest coverage, over every m, any family and n: 31.8 % at n = 30, 65.3 % at n = 1,000 · Validity domain: empty

Method: Mean per unit, difference, ratio, regression · Lowest coverage, over every m, any family and n: between 15.1 % and 20.5 % at n = 30 · Validity domain: empty

Given m, 20 of the 114 cells of the Stringer bound with Poisson factors and 25 of the 114 cells of the cell bound fall below the criterion in at least one family. The lowest is 86.1 %, for the Stringer bound at n = 1,000 when one misstatement is found (P5 with E-full).

The Stringer and cell bounds are cautious. On average, the Stringer bound with Poisson factors is 2.82 times the true overstatement at n = 30, 1.68 times at n = 100 and 1.13 times at n = 1,000. The cell bound is 2.66, 1.60 and 1.11 times.

When a classical PPS sample finds no misstatement, its bound is the known misstatement alone, so it never covers a population that holds one. Its weak side is the upper bound of overstatements. The difference, ratio and regression bounds are 0 when no misstatement is found. The mean-per-unit bound falls short on skewed populations: at n = 1,000, its lower bound covers 94.5 % to 95.0 % of the samples on P3 and 63 % to 65 % on P2.

Comparison with the Financial Audit Manual

The GAO/CIGIE Financial Audit Manual (GAO-18-601G, updated April 2020, section 480.39 and its footnote 5) says that a monetary unit sample with a large number of misstatements should generally be evaluated as a classical PPS sample, with an audit sampling specialist. Its footnote gives the general rule: 10 misstatements for a sample of 75 to 100 items, 10 % for 100 to 300 items and 30 above 300, with a minimum of 75 items for classical PPS. The June 2026 edition of the manual (GAO-26-108577) keeps the same rule.

Blast publishes the stricter of the study and the manual. Samples that meet the manual's rule still fall below the criterion in some families: at n = 1,000 with more than 50 misstatements found, 4 of 118 families reject the classical PPS bound. Its domain therefore stays empty. The manual gives no such rule for the variables estimators, so the study alone sets theirs.

What the study does not say

The study measures coverage on the populations above, at 95 % confidence. It says nothing about a population they do not describe. The Samples tab accepts other confidence levels, which the study has not measured. Coverage given m is pooled over the population sizes and misstatement rates of a family, so an unfavourable size or rate can be hidden by the others. Three choices the tab offers are not measured: the Stringer bound on a cell selection, the random monetary units selection and systematic samples.

You can download four files of version 2: the plan (plan.json), the verdict of every cell (verdicts.csv), the thresholds the engine reads (thresholds.json) and the validity domain (validity.json). The raw counts are not published; the first command below writes them again.

Rerun the study

Each version of the study is frozen with its plan (populations, grid, criterion and the study seed, 20261023), the raw counts of every cell, the verdict of every cell, the thresholds the engine reads and a report. Every sample has its own seed derived from the study seed, and the counts use integer arithmetic only, so any machine writes the same counts. The campaign of version 2 took about 2 hours 20 minutes on a 14-core machine.

From the analytics engine's source folder:

cargo run --locked --release --example sampling_coverage -- \ --plan ../../../../docs/quality/sampling-coverage/v2/plan.json \ --out target/sampling-coverage-v2 cargo run --locked --release --example sampling_coverage -- \ --derive target/sampling-coverage-v2 --validity stats/validity/v2.json \ --artefact ../../../../docs/quality/sampling-coverage/v2

The first command runs the campaign. The second derives the verdicts, the thresholds and the report from its counts.

A new version of an evaluation or draw method fails the engine's tests until a new version of the study is frozen with it. Earlier versions are kept, so an evaluation recorded under an earlier version replays the same.