Tutorial

This walks through the whole loop: install, point the evaluator at a token stream, read the verdict honestly, and reproduce the numbers. It assumes nothing beyond Python 3.11+.

The one idea to hold onto: matching the manuscript is necessary, not sufficient. The tool tells you whether your hypothesis is not excluded — never that it is the answer. Half of learning to use it is learning which axes are informative and which are traps.

1. Install and fetch reference data

python -m venv .venv && . .venv/bin/activate
pip install -e .                 # core evaluator: numpy / pandas / requests only
python -m ms408.acquire          # pinned, sha256-checked reference data (consume-only, L19)

The reference bands themselves ship inside the package, so evaluate() works immediately; acquire is needed for the worked example, the data-gated tests, and verify.

2. Evaluate a token stream

Your input is a list of whitespace-separated word tokens — a cipher’s output, a generator’s output, a transliteration variant, a candidate plaintext rendering. From the shell:

python -m ms408 my_tokens.txt          # pretty table
python -m ms408 --json my_tokens.txt   # machine-readable verdict
cat my_tokens.txt | python -m ms408 -  # from stdin

Or in Python:

from ms408 import evaluate

verdict = evaluate(open("my_tokens.txt").read().split())
print(verdict["hard_axes_in_band"], "of", verdict["hard_axes_total"], "hard axes match")
for axis, r in verdict["axes"].items():
    print(f"{axis:12} value={r['value']!s:>9}  band={r['band']}  in={r['in_band']}")
    # r["caveat"] carries the honest hedge; r["soft"] / r["confounded"] / r["token_sensitive"]

Match the token budget. The bands are built at ~10,000 tokens. h2, ed1, zipf are fairly budget-robust, but ttr is intrinsically token-count-sensitive, so compare at a similar budget. See how examples/evaluate_naibbe.py takes a matched block sample.

3. Read the verdict honestly

The verdict groups axes by how much weight they can bear:

Two verdicts that look similar but aren’t:

• All hard axes in band. Your hypothesis reproduces the manuscript’s load-bearing statistics. That is the strongest the tool can say — and it is still only “not excluded”. It does not identify a mechanism (many different processes can share a statistical envelope). Treat it as a licence to investigate further, not a result.
• Zero hard axes in band. Usually your hypothesis is genuinely off. But check which axes failed and whether any were confounded before you conclude “excluded” — see §4.

4. The cautionary tale: a real cipher scoring 0/3

Run the worked example (needs the Naibbe data from acquire):

python examples/evaluate_naibbe.py

Greshko’s Naibbe cipher (2025) — a genuine, decipherable verbose-homophonic cipher — scores 0/3 hard. The naive read is “0/3 → the manuscript isn’t this cipher”. That read is wrong, and it is exactly the mistake the tool is built to prevent:

• The dI axis has collapsed to ~0, but dI is confounded: most of the collapse is Greshko’s respacing of the plaintext, applied before enciphering. Word-boundary Latin dI (0.076) sits inside the manuscript’s band. An out-of-band dI here is an artifact.
• On the deconfounded mid-level syntax axes — the load-bearing ones — verbose+homophonic ciphers are not robustly separable from the manuscript (E30/E31). So this cipher family is inconclusive, not excluded.
• The ed1 mismatch reflects this cipher’s homophone tables, not a law every verbose-homophonic cipher must obey.

The lesson: a raw count is not a conclusion. You have to know which axes are informative for the mechanism you’re testing. This is why the tool separates hard from confounded/soft and prints “necessary, not sufficient” on every verdict. Full analysis: LIMITS.md, reports/e29_naibbe_discriminators.md.

5. Reproduce the numbers

Everything the tool reports is reproducible from committed code:

python -m ms408.verify          # recompute the VMS point signature; check self-consistency
python -m ms408.verify --full   # also rebuild the reference bands and diff (bit-for-bit)

If you change an axis definition, rebuild the bands and re-pin the tests:

python -m ms408.experiments.e32_reference_bands   # rewrites src/ms408/data/reference_bands.json
# then update tests/test_verify.py::PINNED_* and confirm `ms408.verify --full` passes

Where to go next

• METHODOLOGY.md — the harness / firewall / grading / refutation discipline, and how to run the adversarial-review protocol on your own claim.
• LIMITS.md — the per-axis caveats in full.
• ../CONTRIBUTING.md — the rules for adding a discriminator or source.