path: root/tools/check_citations.py

#!/usr/bin/env python3
"""check_citations.py -- verify every "LT.txt:<n>" citation in lang/la.ini
actually resolves to the Latin text it claims, in docs/research/LT.txt.

Run via `make check-citations` (or directly: `python3 tools/check_citations.py
[--file LA_INI] [--lt-file LT_TXT]`). Exits 2 with a report if any citation
looks WRONG, is MALFORMED, or CANNOT BE VERIFIED; exits 0 (with a summary
line) only if every citation checked out cleanly; exits 0 with a loud
"SKIPPED" line if docs/research/LT.txt is not present (it is gitignored --
see below).

THIS SCRIPT WAS ITSELF FOUND TO BE SELF-POISONING (2026-08-19) AND HARDENED
------------------------------------------------------------------------
An earlier version pooled distinctive words from the whole COMMENT BLOCK
around a citation, including any double-quoted phrase the comment happened
to mention -- and comments routinely quote a WRONG historical value while
explaining a past fix (e.g. "CORRECTED: previously cited LT.txt:12459,
which is 'Dominica ultima Octobris', not this heading"). That quote landed
in the pool, so re-introducing the exact bug being documented -- citing
LT.txt:12459 again -- matched the very quote correcting it, and the script
reported "0 look wrong". A verification tool whose own documentation of a
fix defeats the check for that fix is worse than no tool: it manufactures
false confidence. Proven with a reproduction: reintroducing that one wrong
citation into a real copy of lang/la.ini produced zero findings on the
pre-hardening script. Four changes closed this, in order of how directly
each one addresses the reproduction:

  1. THE POOL IS SCOPED TO THE CITATION'S OWN ENTRY, never to the
     surrounding comment's quoted text. A citation is verified against the
     name it claims, not against anything quoted nearby -- see
     `distinctive_words` / the per-citation `pool_items` construction
     below. This alone closes the reproduced bug (see `test_check_citations.py`'s
     own `test_self_poisoning_quote_does_not_pass`).
  2. A POOL TOO THIN TO VERIFY FAILS CLOSED (round 1's shape; round 2
     replaced the mechanism -- see below -- but kept the discipline: no
     match is ever reported as a silent PASS just because nothing better
     was checked).
  3. NO MORE BLANKET +-2-LINE TOLERANCE. A citation is checked at the EXACT
     line it names. A heading that genuinely spans more than one physical
     line in the source must say so explicitly, "LT.txt:8609-8610" -- the
     allowance moves into the data, where a reader (and a future citation
     added nearby) can see it, instead of silently forgiving ANY citation
     within two lines of the truth. A bare "LT.txt:N" is checked at line N
     only.
  4. THIS SCRIPT NOW HAS ITS OWN TEST SUITE (test_check_citations.py,
     wired into `dune test` via tools/dune's own runtest rule) -- the
     single most important change. The reproduced bug survived as long as
     it did specifically BECAUSE nothing exercised this script's own
     logic against a known-wrong citation. The self-poisoning case above
     is now a permanent regression test.

ROUND 2 (2026-08-19): A REVIEW DEFEATED ROUND 1 AGAIN -- FIVE MORE FIXES
------------------------------------------------------------------------
Round 1's own hardening had three further holes, all found live against
this exact file, plus two smaller defects. Fixed in this order (matching
the order the defeats were found in, most dangerous first):

  1. "; PATTERN" SILENCED A WHOLE BLOCK, NOT JUST ITS OWN ENTRY. `check()`
     used to test `"PATTERN" in block_comment` -- one substring search over
     every comment in the block -- and `continue` past the ENTIRE block if
     it matched anywhere. In a block that mixes a PATTERN-marked entry with
     an ordinary entry carrying its own, genuinely wrong citation (exactly
     [season]'s own trailing-comment style: several entries back-to-back
     with no blank line between them), the wrong citation was never even
     looked at. Fixed by scoping PATTERN the same way citation pools are
     already scoped: a PASS over the block first collects which entries a
     PATTERN-bearing comment actually covers (the one preceding entry for a
     trailing comment, every entry in the block for a leading one -- the
     identical leading/trailing rule `check()` already uses for citation
     pooling), and only THOSE entries are excluded from checking. See
     `test_pattern_does_not_silence_a_different_entry_in_the_same_block`.
  2. EXPLICIT RANGES HAD NO UPPER BOUND. Round 1 replaced the old blanket
     +-2-line tolerance with an explicit per-citation range
     ("LT.txt:8609-8610") specifically so a wrap allowance would be
     visible in the data instead of invisible in the checker -- but set no
     cap on how WIDE that range could be. "LT.txt:8600-8650" passed if the
     claimed text appeared anywhere across fifty lines: the same
     "tolerance forgives a wrong line" defect the +-2 removal was meant to
     close, reintroduced at 25x the radius, now with the data's own
     apparent blessing. Fixed with `MAX_RANGE_WIDTH` (see below): a range
     wider than a genuine heading wrap (1-2 extra physical lines) is
     rejected as MALFORMED, naming the entry and the width, rather than
     silently accepted. See `test_range_wider_than_cap_is_malformed`.
  3. THE "TOO THIN TO VERIFY" GATE COUNTED WORDS, NOT RARITY. Round 1
     required a pool item to have >=2 distinctive words before it could
     even be checked, on the theory that a one-word match proves nothing.
     Measured against the real data: this flagged 11 CORRECT citations
     CANNOT VERIFY -- every one a short Latin hagionym whose Missal
     heading genuinely has only one non-stopword ("S. Antonii Abb.": only
     "antonii" survives the stopword filter) -- while a match on nothing
     but "classis" (507 occurrences across LT.txt) passed the >=2 bar
     freely whenever it shared a citation with three siblings ("I classis
     / II classis / III classis / IV classis"). Word COUNT was never the
     right proxy; word RARITY is. Replaced with `build_frequency_table` +
     `item_evidence`: a token seen n times in the whole corpus contributes
     1/n of evidence, an item's evidence is its single RAREST matched
     token (not a sum -- see `item_evidence`'s own docstring for why
     summing would reopen this exact class of bug), and `EVIDENCE_THRESHOLD`
     is the bar a match must clear to count as real proof rather than
     coincidence. See `test_rare_single_word_match_passes` and
     `test_common_word_only_match_is_cannot_verify`.
  4. A COMMA TYPO SILENTLY BECAME A DIFFERENT CITATION. "LT.txt:12,459" (a
     stray thousands-separator comma for the single number "12459") used
     to parse as TWO independent bare citations, "12" and "459" -- either
     of which might coincidentally match somewhere nearby while the
     intended line was never checked at all. `parse_citation_spec` now
     recognises the shape (a 1-2 digit token immediately followed by an
     exactly-3-digit token -- the only way a thousands-grouped LT.txt line
     number, which never exceeds 5 digits, can be split by one comma) and
     rejects the whole spec as MALFORMED rather than silently reinterpreting
     it. See `test_thousands_separator_typo_is_malformed`.
  5. THE SELF-TEST SUITE OVERSTATED ITS OWN COVERAGE. A review ran round
     1's seven fixture cases against the PRE-round-1 script: only two
     (`test_off_by_one_line_fails`, `test_self_poisoning_quote_does_not_pass`)
     actually failed on it -- the rest passed on both sides of round 1's
     fix and regression-tested nothing despite their names (a third,
     `test_wrap_range_required_not_just_first_line`, was added after the
     "seven" and also turns out to be a genuine regression test, confirmed
     the same way). Every test in this suite is now labelled REGRESSION
     (shown, not just claimed, to fail against a named prior version) or
     CHARACTERISATION (pins current behaviour; does not fail on the prior
     version, usually because it exercises a data shape or API surface
     that prior version did not have at all) -- see
     `test_check_citations.py`'s own module docstring for the full,
     per-test account and how each label was actually verified.

MALFORMED, PRECISELY (round 2's third finding class, alongside WRONG and
CANNOT VERIFY): a citation whose SYNTAX cannot be trusted even before its
content is checked -- a range wider than `MAX_RANGE_WIDTH`, a backwards
range, or the thousands-separator-typo shape above. Reported, counted, and
fails the run exactly like a WRONG citation: rejecting the syntax rather
than guessing at the author's intent is the whole point (see fix 2 and
fix 4 above) -- "the author either finds the real line or marks it PATTERN
honestly" is not achieved by the checker silently picking a reading.

WHAT THIS CHECKS, PRECISELY (a heuristic, not a proof)
-------------------------------------------------------
lang/la.ini's own comments cite a Missal heading in one of two shapes:

  1. A LEADING comment block, then a group of entries it covers, e.g.
     "; Ash Wednesday and the three days after it -- LT.txt:8686-8689."
     followed by four `key = value` lines. The pool for every citation
     found in such a comment is EVERY entry in the group (there is no
     positional correspondence encoded in the data between a particular
     cited line and a particular entry in the list).
  2. A TRAILING comment immediately under the ONE entry it explains, e.g.
     "advent = Tempus Adventus" then "; LT.txt:8609." on the next line,
     with no blank line -- [season]'s own style. The pool is that one entry
     alone.

A "PATTERN" marker follows the identical leading/trailing rule (see round
2 fix 1 above): it excludes only the entry (or entries) it is itself
attached to from citation-checking, never the rest of the block.

A citation is either a bare line number ("LT.txt:8609") or an explicit
range ("LT.txt:8609-8610", capped at `MAX_RANGE_WIDTH` lines) for a heading
that genuinely wraps across physical lines in the source; a comma-separated
list ("LT.txt:8618,8620,8622") is several independent citations, each
checked on its own -- unless the list itself looks like a thousands-typo
for one number (round 2 fix 4), in which case the whole spec is MALFORMED.
For a bare number the window is that one line; for a range it is the union
of every line in the range (inclusive). There is no other tolerance.

For each citation, the POOL is the set of candidate Latin phrases it could
be defending: the entry (trailing shape) or every entry in the group
(leading shape) -- nothing pulled from quoted prose elsewhere in the
comment (see the self-poisoning account above). Each pool item keeps its
OWN distinctive-word set (>=4 letters, not on the small stopword list
below, j/i and ae/oe/diacritics normalised) -- items are never flattened
into one shared bag, for the same reason quotes were removed: a citation
bundling two claims onto one line number must not pass on the strength of
an unrelated pool item's words.

A pool item is a candidate MATCH only if the cited window's text contains
ALL of its distinctive words -- otherwise it is not a match at all: it
never contributed to a PASS and is not what the citation is checked at
all. Among pool items that ARE contained in the window, the citation
PASSES if at least one clears `EVIDENCE_THRESHOLD` (see round 2 fix 3
above and `item_evidence`'s own docstring) -- i.e. contains a word rare
enough, across the whole LT.txt corpus, to be real evidence rather than
coincidence. A citation whose only contained items are all common-word-only
is CANNOT VERIFY, not a silent PASS: found, but not proven. A citation with
NO contained item at all -- the claimed words are simply not at the cited
line(s) -- is WRONG.

This is deliberately a LOOSE, word-overlap check within the (now exact and
bounded) window, not a byte-exact phrase match: la.ini spells abbreviations
out in full (Sanctissimi, not Ss.mi) and normalises j->i, and requiring a
byte-exact substring would either force every citation's prose to repeat
the raw OCR text verbatim (defeating the point of writing readable
comments) or produce false failures having nothing to do with a wrong line
number. What it proves is narrower than a byte-exact match, and is
disclosed as such: PASS means "the claimed name's distinctive words are
present, in full, at the exact line(s) cited, and at least one of them is
rare enough in the corpus to be real evidence" -- not that the citation is
the best possible line, only that it is not obviously wrong and is not
resting on a coincidence-prone common word.
"""
import argparse
import re
import sys
import unicodedata
from pathlib import Path

ROOT = Path(__file__).resolve().parent.parent
DEFAULT_LA_INI = ROOT / "lang" / "la.ini"
DEFAULT_LT_TXT = ROOT / "docs" / "research" / "LT.txt"

STOPWORDS = {
    "in", "de", "et", "ad", "post", "ante", "cum", "per", "seu", "infra",
    "vel", "si", "haec", "hoc", "hic", "qui", "quae", "quod", "quia",
    "tempus", "dominica", "dominicam", "dominicae", "feria", "feriae",
    "sabbato", "sabbatum", "die", "diebus", "eodem", "anno", "eius",
    "sancti", "sancta", "sanctae", "sancto", "sanctorum", "sanctus",
    "domini", "dominus", "octava", "octavam", "octavas",
    "missae", "missa", "proprium", "gregorianus", "cantus", "pdf",
    "forma", "longior", "brevior", "vide", "etiam", "dom", "prosper",
    "sacro", "actio", "electronica", "formam", "novissimae", "variationes",
    "copyright", "archivum", "liturgicum", "missale", "romanum", "index",
    "www", "http", "https", "htm", "html", "com", "romanum", "text",
}

# A citation's explicit range is capped at this many lines (n .. n+2). A
# genuine heading wrap in the source is one or two extra physical lines;
# anything wider is not a wrap, it is a search over a neighbourhood wide
# enough to coincidentally contain almost any short phrase -- exactly the
# blanket +-2 tolerance round 1 removed, reintroduced at a much larger
# radius by an unbounded range. See round 2 fix 2 in the module docstring.
MAX_RANGE_WIDTH = 3

# The evidence bar a pool item's RAREST matched word must clear to count
# as real proof (see `item_evidence` below). 1/200: a word occurring up to
# ~200 times across the whole ~118,000-token LT.txt corpus can still be
# the deciding, sole distinctive word of a short Missal calendar-table
# entry (measured: "omnium", the only survivor in "Omnium Sanctorum",
# occurs 139 times across the whole document, mostly in unrelated legal
# prose -- "of all" is common Latin furniture -- yet is genuinely the
# correct, sole citable word for that one heading). "classis" (507
# occurrences), the round 1 false-pass this rule specifically targets,
# sits comfortably over 2.5x past this bar and stays CANNOT VERIFY.
# Round-2's own measurement of every affected real citation in
# lang/la.ini is in the branch report.
EVIDENCE_THRESHOLD = 1.0 / 200


def normalize_word(w: str) -> str:
    w = w.lower()
    w = unicodedata.normalize("NFKD", w)
    w = "".join(c for c in w if not unicodedata.combining(c))
    w = w.replace("æ", "ae").replace("œ", "oe")
    w = re.sub(r"[^a-z]", "", w)
    w = w.replace("j", "i")
    return w


def distinctive_words(text: str) -> set:
    out = set()
    for tok in re.split(r"\s+", text):
        w = normalize_word(tok)
        if len(w) >= 4 and w not in STOPWORDS:
            out.add(w)
    return out


def build_frequency_table(lt_lines: list) -> dict:
    """Count how many times each normalised word occurs anywhere in the
    whole LT.txt corpus -- built ONCE per run (not per citation) and
    consulted by `word_evidence`/`item_evidence` below. This is what lets
    the checker tell a token that could only ever mean one heading (occurs
    once in 26,000+ lines) apart from common liturgical furniture (occurs
    hundreds of times) -- see round 2 fix 3 in the module docstring."""
    freq = {}
    for line in lt_lines:
        for tok in re.split(r"\s+", line):
            w = normalize_word(tok)
            if w:
                freq[w] = freq.get(w, 0) + 1
    return freq


def word_evidence(word: str, freq: dict) -> float:
    """A token seen n times contributes 1/n: a hapax (n=1) contributes 1.0
    -- about as conclusive as a word-overlap check can be -- and a word
    occurring hundreds of times contributes next to nothing. A plain
    reciprocal is chosen over a logarithmic/IDF scale deliberately:
    measured against this corpus, log-scaling compresses "occurs once" and
    "occurs 500 times" into a difference of a few units, too close to
    cleanly separate "essentially conclusive" from "unverified" with a
    single threshold; a reciprocal keeps them many orders of magnitude
    apart, which is the actual claim this rule makes."""
    n = freq.get(word, 0)
    return 1.0 / n if n > 0 else 0.0


def item_evidence(words: set, freq: dict) -> float:
    """A pool item's rarity evidence is the SINGLE RAREST word it matched
    on -- not a sum over all its words. Summing would let several
    merely-uncommon words add up to "enough" evidence between them, which
    is exactly the shape round 2 fix 3 exists to close (a match consisting
    only of common tokens must stay CANNOT VERIFY "regardless of how many
    words sit beside it") and exactly the self-poisoning failure mode from
    round 1 (quoted corrective prose tends to share several ordinary words
    with its neighbour, never one rare one). One genuinely rare word is
    real evidence; an accumulation of merely-uncommon words is not the
    same thing and must not be treated as if it were."""
    if not words:
        return 0.0
    return max(word_evidence(w, freq) for w in words)


class CitationRef:
    """One citation token: 'label' is what the data actually wrote
    ("8609" or "8609-8610"); 'lines' is the fully-expanded, sorted list of
    line numbers the label names -- a single-element list for a bare
    number, the whole inclusive range for an explicit wrap."""

    __slots__ = ("label", "lines")

    def __init__(self, label, lines):
        self.label = label
        self.lines = lines


def _looks_like_thousands_typo(tokens) -> bool:
    """'12,459' splits into ('12', '459') -- the classic shape of a human
    thousands-separator typo for a single 4-5 digit LT.txt line number:
    grouped from the right in chunks of 3, a 4-digit number groups as
    'N,NNN' and a 5-digit number as 'NN,NNN' (LT.txt never reaches 6
    digits, so a 3-digit leading group never arises from real grouping).
    Checked against every real comma-list citation in lang/la.ini as of
    round 2: none has a 1-2 digit token immediately followed by an
    exactly-3-digit token, so this shape is unambiguous enough in practice
    to reject outright as malformed rather than silently parsing as two
    unrelated short citations (round 2 fix 4)."""
    for a, b in zip(tokens, tokens[1:]):
        if re.fullmatch(r"\d{1,2}", a) and re.fullmatch(r"\d{3}", b):
            return True
    return False


def parse_citation_spec(spec: str):
    """'8618,8620,8622' -> three exact-line CitationRefs; '8609-8610' -> one
    CitationRef spanning both lines (an explicit wrapped heading, capped at
    MAX_RANGE_WIDTH lines). Returns (refs, malformed): `refs` is the list of
    successfully-parsed CitationRef objects; `malformed` is a list of
    {"token", "reason"} dicts for anything that did NOT parse into a
    trustworthy citation -- a backwards range, a range wider than the cap,
    the thousands-typo shape above, or an unparseable token. Round 1
    silently DROPPED all of these (no crash, but no report either); round 2
    surfaces every one instead, because a malformed citation naming a real
    entry deserves a human's attention exactly as much as a wrong one does
    (see round 2 fixes 2 and 4 in the module docstring)."""
    raw_tokens = [t.strip() for t in spec.split(",")]
    if _looks_like_thousands_typo(raw_tokens):
        return [], [
            {
                "token": spec,
                "reason": (
                    "looks like a thousands-separator typo for one number "
                    "(a 1-2 digit token immediately followed by a 3-digit "
                    "one) rather than a genuine list of citations -- "
                    "remove the comma, or split into real citations"
                ),
            }
        ]
    refs = []
    malformed = []
    for tok in raw_tokens:
        if not tok:
            continue
        m = re.fullmatch(r"(\d{2,6})-(\d{2,6})", tok)
        if m:
            a, b = int(m.group(1)), int(m.group(2))
            width = b - a + 1
            if a > b:
                malformed.append({"token": tok, "reason": f"backwards range (LT.txt:{tok})"})
            elif width > MAX_RANGE_WIDTH:
                malformed.append(
                    {
                        "token": tok,
                        "reason": (
                            f"range is {width} lines wide (max {MAX_RANGE_WIDTH}) -- "
                            "not a genuine heading wrap; find the real line or mark PATTERN honestly"
                        ),
                    }
                )
            else:
                refs.append(CitationRef(tok, list(range(a, b + 1))))
            continue
        m = re.fullmatch(r"(\d{2,6})", tok)
        if m:
            refs.append(CitationRef(tok, [int(m.group(1))]))
            continue
        malformed.append({"token": tok, "reason": "unparseable citation token"})
    return refs, malformed


CITATION_RE = re.compile(r"LT\.txt:\s*((?:\d{2,6}(?:-\d{2,6})?)(?:\s*,\s*\d{2,6}(?:-\d{2,6})?)*)")


def parse_blocks(la_ini_text: str):
    """Split la.ini into blocks on blank lines and [section] headers. Each
    block is a list of (kind, content) where kind is 'comment' or 'entry',
    content is the stripped comment text or (key, value)."""
    blocks = []
    cur = []
    for raw in la_ini_text.split("\n"):
        line = raw.rstrip("\n")
        stripped = line.strip()
        if stripped == "" or stripped.startswith("["):
            if cur:
                blocks.append(cur)
            cur = []
            continue
        if stripped.startswith(";"):
            cur.append(("comment", stripped[1:].strip()))
        elif "=" in stripped:
            k, _, v = stripped.partition("=")
            cur.append(("entry", (k.strip(), v.strip())))
        # anything else (shouldn't occur) is ignored
    if cur:
        blocks.append(cur)
    return blocks


def window_text_for(lt_lines, lines):
    lo, hi = lines[0], lines[-1]
    lo_c, hi_c = max(1, lo), min(len(lt_lines), hi)
    if lo_c > hi_c:
        return ""
    return " ".join(lt_lines[lo_c - 1 : hi_c])


def _pool_entries_for(entries_before, entries):
    """The leading/trailing pooling rule, shared identically by citation
    checking AND PATTERN scoping (round 2 fix 1): a TRAILING comment (one
    or more entries already seen in this block) pools against only the
    most recent entry; a LEADING comment (no entry seen yet) pools against
    every entry in the block. The same rule must decide both questions --
    a PATTERN marker and a citation attached to the same comment always
    cover the same entries, by construction."""
    return [entries_before[-1]] if entries_before else entries


def _pattern_marked_entries(block, entries):
    """Which entries in this block are excluded from citation-checking by
    their OWN "PATTERN" marker -- never by a PATTERN marker attached to a
    DIFFERENT entry in the same block. Returns a set of `id()` of the
    entry (key, value) tuples (safe: each entry tuple is a single object,
    shared by reference between `block` and `entries`, never copied).

    This is a first, standalone pass over the block, completed before any
    citation is evaluated, so a citation's own PATTERN status never
    depends on where in the block it happens to sit relative to its
    entry's PATTERN comment."""
    marked = set()
    entries_before = []
    for k, c in block:
        if k == "comment":
            if "PATTERN" in c:
                for e in _pool_entries_for(entries_before, entries):
                    marked.add(id(e))
        else:
            entries_before.append(c)
    return marked


def check(la_ini_text: str, lt_lines: list):
    """Returns a dict: checked (int), passed (int), findings (list --
    wrong citations), unverifiable (list -- no matched item clears the
    evidence bar), malformed (list -- citation syntax itself could not be
    trusted). findings, unverifiable and malformed are all things a human
    must look at; only 'passed' citations required no human attention."""
    checked = 0
    passed = 0
    findings = []
    unverifiable = []
    malformed = []
    freq = build_frequency_table(lt_lines)

    for block in parse_blocks(la_ini_text):
        entries = [c for k, c in block if k == "entry"]
        if not entries:
            continue

        pattern_entries = _pattern_marked_entries(block, entries)

        entries_before = []
        for k, c in block:
            if k == "comment":
                for m in CITATION_RE.finditer(c):
                    pool_entries = _pool_entries_for(entries_before, entries)
                    if all(id(e) in pattern_entries for e in pool_entries):
                        # Every entry this citation could be defending is
                        # itself PATTERN-marked: this "LT.txt:N" is not a
                        # provenance claim for any of them (e.g. citing the
                        # GRAMMAR another day's heading attests), so
                        # checking it would produce a meaningless failure.
                        continue
                    entry_desc = ", ".join(f"{k2}={v2}" for k2, v2 in pool_entries)
                    pool_items = [distinctive_words(v2) for _, v2 in pool_entries]
                    nonempty_items = [p for p in pool_items if p]
                    pool_words = sorted(set().union(*pool_items)) if pool_items else []

                    refs, bad_tokens = parse_citation_spec(m.group(1))
                    for bad in bad_tokens:
                        checked += 1
                        malformed.append(
                            {
                                "label": bad["token"],
                                "entries": entry_desc,
                                "reason": bad["reason"],
                            }
                        )
                    for ref in refs:
                        checked += 1
                        window_text = window_text_for(lt_lines, ref.lines)
                        window_words = distinctive_words(window_text)

                        if not nonempty_items:
                            # Every pool item is fully stopwords -- there is
                            # nothing to check either way. Absence of
                            # evidence is not evidence of correctness.
                            unverifiable.append(
                                {
                                    "label": ref.label,
                                    "entries": entry_desc,
                                    "reason": "no distinctive words in the entry text to check at all",
                                    "pool_words": pool_words,
                                }
                            )
                            continue

                        contained_items = [p for p in nonempty_items if p <= window_words]
                        strong_items = [
                            p for p in contained_items if item_evidence(p, freq) >= EVIDENCE_THRESHOLD
                        ]
                        if strong_items:
                            passed += 1
                        elif contained_items:
                            best = max(item_evidence(p, freq) for p in contained_items)
                            unverifiable.append(
                                {
                                    "label": ref.label,
                                    "entries": entry_desc,
                                    "reason": (
                                        f"matched, but every matched word is too common to trust "
                                        f"(best evidence {best:.4f}, need >= {EVIDENCE_THRESHOLD:.4f})"
                                    ),
                                    "pool_words": pool_words,
                                }
                            )
                        else:
                            n = ref.lines[0]
                            findings.append(
                                {
                                    "label": ref.label,
                                    "entries": entry_desc,
                                    "actual": (
                                        lt_lines[n - 1].strip()
                                        if 1 <= n <= len(lt_lines)
                                        else "(out of range)"
                                    ),
                                    "window": window_text.strip()[:160],
                                }
                            )
            else:
                entries_before.append(c)
    return {
        "checked": checked,
        "passed": passed,
        "findings": findings,
        "unverifiable": unverifiable,
        "malformed": malformed,
    }


def build_arg_parser():
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument(
        "--file",
        dest="la_ini",
        type=Path,
        default=DEFAULT_LA_INI,
        help=f"the la.ini-shaped file to check (default: {DEFAULT_LA_INI})",
    )
    p.add_argument(
        "--lt-file",
        dest="lt_txt",
        type=Path,
        default=DEFAULT_LT_TXT,
        help=f"the LT.txt transcription to check against (default: {DEFAULT_LT_TXT})",
    )
    return p


def main(argv=None):
    args = build_arg_parser().parse_args(argv)
    if not args.lt_txt.exists():
        print(
            f"SKIPPED: {args.lt_txt} is absent (docs/ is gitignored -- "
            "present locally only). Citations are NOT verified this run."
        )
        return 0
    la_ini_text = args.la_ini.read_text(encoding="utf-8")
    lt_lines = args.lt_txt.read_text(encoding="utf-8", errors="replace").split("\n")
    result = check(la_ini_text, lt_lines)
    checked = result["checked"]
    findings = result["findings"]
    unverifiable = result["unverifiable"]
    malformed = result["malformed"]

    if not findings and not unverifiable and not malformed:
        print(f"check-citations: {checked} LT.txt citations checked, 0 look wrong.")
        return 0

    print(
        f"check-citations: {len(findings)} of {checked} citations look wrong, "
        f"{len(malformed)} MALFORMED, {len(unverifiable)} CANNOT VERIFY (see below):\n"
    )
    if findings:
        print("WRONG:\n")
        for f in findings:
            print(f"  LT.txt:{f['label']} cited for [{f['entries']}]")
            print(f"    actual: {f['actual']!r}")
            print(f"    window: {f['window']!r}\n")
    if malformed:
        print("MALFORMED (citation syntax itself is untrustworthy):\n")
        for m in malformed:
            print(f"  LT.txt:{m['label']} cited for [{m['entries']}]")
            print(f"    {m['reason']}\n")
    if unverifiable:
        print("CANNOT VERIFY (a human must adjudicate these by hand):\n")
        for u in unverifiable:
            words = ", ".join(u["pool_words"]) if u["pool_words"] else "(none)"
            print(f"  LT.txt:{u['label']} cited for [{u['entries']}]")
            print(f"    {u['reason']}; pool words: {words}\n")
    return 2


if __name__ == "__main__":
    sys.exit(main())