From 174fe8b3fedf61cf1fa0dc7499573374133a8ca2 Mon Sep 17 00:00:00 2001 From: Lukasz Kasprzak Date: Wed, 19 Aug 2026 15:47:37 +0200 Subject: fix(tools): close five more ways to defeat check_citations.py Round 1 hardened check_citations.py against its own self-poisoning bug; a review defeated it again. Five fixes, in the order they were found: 1. PATTERN silenced a whole comment block, not just the entry it was attached to -- a wrong citation on a DIFFERENT, unmarked entry in the same block (e.g. [season]'s own back-to-back trailing-comment style) was never checked at all. Fixed by scoping PATTERN with the identical leading/trailing pooling rule citations already use: an entry is excluded only by its own marker, never a neighbour's. 2. Explicit per-citation ranges (introduced in round 1 to replace a blanket +-2-line tolerance) had no upper bound, reintroducing the same defect at a much larger radius (LT.txt:8600-8650 passed if the text appeared anywhere in fifty lines). Capped at MAX_RANGE_WIDTH (3 lines); anything wider is reported MALFORMED, naming the entry and the width, instead of silently accepted. 3. The "pool too thin to verify" gate counted words, not rarity -- it flagged 11 genuinely correct citations (short Latin hagionyms with only one non-stopword) CANNOT VERIFY, while a match on nothing but "classis" (507 occurrences) passed freely alongside three siblings. Replaced with a frequency table over the whole LT.txt corpus: a token's evidence is 1/(times seen), an item's evidence is its single rarest matched token (not a sum -- summing would let several merely-common words add up to "enough" between them, the same shape as the self-poisoning bug). 4. "LT.txt:12,459" (a comma typo for one number) parsed as two unrelated bare citations, 12 and 459, either of which could coincidentally match while the intended line was never checked. Detected as a thousands-separator-typo shape (a 1-2 digit token immediately followed by an exactly-3-digit one -- the only way a real LT.txt line number, which never exceeds 5 digits, splits under one comma) and rejected as malformed. 5. The self-test suite overstated its own coverage: of round 1's seven fixture cases, only two actually failed against the pre-round-1 script. Every test is now labelled REGRESSION or CHARACTERISATION, each verified by direct replay against the named prior version rather than asserted -- 14 of 33 are genuine regression tests. Both of the review's own defeats (block-wide PATTERN silencing, the 50-line range) are reproduced as dedicated fixtures and confirmed caught; both are also confirmed to slip through the pre-round-2 tool unchanged. Claude-Session: https://claude.ai/code/session_017ZBxCCRM2ojnBupp3SBxV9 --- tools/check_citations.py | 438 ++++++++++++++++++++++++++++++------- tools/test_check_citations.py | 493 ++++++++++++++++++++++++++++++++++++------ 2 files changed, 784 insertions(+), 147 deletions(-) (limited to 'tools') diff --git a/tools/check_citations.py b/tools/check_citations.py index 752977e..688b00e 100755 --- a/tools/check_citations.py +++ b/tools/check_citations.py @@ -4,9 +4,10 @@ actually resolves to the Latin text it claims, in docs/research/LT.txt. Run via `make check-citations` (or directly: `python3 tools/check_citations.py [--file LA_INI] [--lt-file LT_TXT]`). Exits 2 with a report if any citation -looks WRONG or CANNOT BE VERIFIED; exits 0 (with a summary line) only if -every citation checked out cleanly; exits 0 with a loud "SKIPPED" line if -docs/research/LT.txt is not present (it is gitignored -- see below). +looks WRONG, is MALFORMED, or CANNOT BE VERIFIED; exits 0 (with a summary +line) only if every citation checked out cleanly; exits 0 with a loud +"SKIPPED" line if docs/research/LT.txt is not present (it is gitignored -- +see below). THIS SCRIPT WAS ITSELF FOUND TO BE SELF-POISONING (2026-08-19) AND HARDENED ------------------------------------------------------------------------ @@ -30,14 +31,10 @@ each one addresses the reproduction: `distinctive_words` / the per-citation `pool_items` construction below. This alone closes the reproduced bug (see `test_check_citations.py`'s own `test_self_poisoning_quote_does_not_pass`). - 2. A POOL TOO THIN TO VERIFY FAILS CLOSED. Latin liturgical headings are - short and heavily stopword-laden ("Tempus Adventus", "I classis", - "albus"): after stripping stopwords, MANY single-entry pools collapse - to one word or none -- a one-word "match" proves nothing (it is as - likely to hit an unrelated nearby heading as the right one). Such a - citation is reported CANNOT VERIFY, not PASS, and it fails the target - exactly like a genuine mismatch: absence of evidence is not evidence - of correctness, and this script must not report it as one. + 2. A POOL TOO THIN TO VERIFY FAILS CLOSED (round 1's shape; round 2 + replaced the mechanism -- see below -- but kept the discipline: no + match is ever reported as a silent PASS just because nothing better + was checked). 3. NO MORE BLANKET +-2-LINE TOLERANCE. A citation is checked at the EXACT line it names. A heading that genuinely spans more than one physical line in the source must say so explicitly, "LT.txt:8609-8610" -- the @@ -52,6 +49,90 @@ each one addresses the reproduction: logic against a known-wrong citation. The self-poisoning case above is now a permanent regression test. +ROUND 2 (2026-08-19): A REVIEW DEFEATED ROUND 1 AGAIN -- FIVE MORE FIXES +------------------------------------------------------------------------ +Round 1's own hardening had three further holes, all found live against +this exact file, plus two smaller defects. Fixed in this order (matching +the order the defeats were found in, most dangerous first): + + 1. "; PATTERN" SILENCED A WHOLE BLOCK, NOT JUST ITS OWN ENTRY. `check()` + used to test `"PATTERN" in block_comment` -- one substring search over + every comment in the block -- and `continue` past the ENTIRE block if + it matched anywhere. In a block that mixes a PATTERN-marked entry with + an ordinary entry carrying its own, genuinely wrong citation (exactly + [season]'s own trailing-comment style: several entries back-to-back + with no blank line between them), the wrong citation was never even + looked at. Fixed by scoping PATTERN the same way citation pools are + already scoped: a PASS over the block first collects which entries a + PATTERN-bearing comment actually covers (the one preceding entry for a + trailing comment, every entry in the block for a leading one -- the + identical leading/trailing rule `check()` already uses for citation + pooling), and only THOSE entries are excluded from checking. See + `test_pattern_does_not_silence_a_different_entry_in_the_same_block`. + 2. EXPLICIT RANGES HAD NO UPPER BOUND. Round 1 replaced the old blanket + +-2-line tolerance with an explicit per-citation range + ("LT.txt:8609-8610") specifically so a wrap allowance would be + visible in the data instead of invisible in the checker -- but set no + cap on how WIDE that range could be. "LT.txt:8600-8650" passed if the + claimed text appeared anywhere across fifty lines: the same + "tolerance forgives a wrong line" defect the +-2 removal was meant to + close, reintroduced at 25x the radius, now with the data's own + apparent blessing. Fixed with `MAX_RANGE_WIDTH` (see below): a range + wider than a genuine heading wrap (1-2 extra physical lines) is + rejected as MALFORMED, naming the entry and the width, rather than + silently accepted. See `test_range_wider_than_cap_is_malformed`. + 3. THE "TOO THIN TO VERIFY" GATE COUNTED WORDS, NOT RARITY. Round 1 + required a pool item to have >=2 distinctive words before it could + even be checked, on the theory that a one-word match proves nothing. + Measured against the real data: this flagged 11 CORRECT citations + CANNOT VERIFY -- every one a short Latin hagionym whose Missal + heading genuinely has only one non-stopword ("S. Antonii Abb.": only + "antonii" survives the stopword filter) -- while a match on nothing + but "classis" (507 occurrences across LT.txt) passed the >=2 bar + freely whenever it shared a citation with three siblings ("I classis + / II classis / III classis / IV classis"). Word COUNT was never the + right proxy; word RARITY is. Replaced with `build_frequency_table` + + `item_evidence`: a token seen n times in the whole corpus contributes + 1/n of evidence, an item's evidence is its single RAREST matched + token (not a sum -- see `item_evidence`'s own docstring for why + summing would reopen this exact class of bug), and `EVIDENCE_THRESHOLD` + is the bar a match must clear to count as real proof rather than + coincidence. See `test_rare_single_word_match_passes` and + `test_common_word_only_match_is_cannot_verify`. + 4. A COMMA TYPO SILENTLY BECAME A DIFFERENT CITATION. "LT.txt:12,459" (a + stray thousands-separator comma for the single number "12459") used + to parse as TWO independent bare citations, "12" and "459" -- either + of which might coincidentally match somewhere nearby while the + intended line was never checked at all. `parse_citation_spec` now + recognises the shape (a 1-2 digit token immediately followed by an + exactly-3-digit token -- the only way a thousands-grouped LT.txt line + number, which never exceeds 5 digits, can be split by one comma) and + rejects the whole spec as MALFORMED rather than silently reinterpreting + it. See `test_thousands_separator_typo_is_malformed`. + 5. THE SELF-TEST SUITE OVERSTATED ITS OWN COVERAGE. A review ran round + 1's seven fixture cases against the PRE-round-1 script: only two + (`test_off_by_one_line_fails`, `test_self_poisoning_quote_does_not_pass`) + actually failed on it -- the rest passed on both sides of round 1's + fix and regression-tested nothing despite their names (a third, + `test_wrap_range_required_not_just_first_line`, was added after the + "seven" and also turns out to be a genuine regression test, confirmed + the same way). Every test in this suite is now labelled REGRESSION + (shown, not just claimed, to fail against a named prior version) or + CHARACTERISATION (pins current behaviour; does not fail on the prior + version, usually because it exercises a data shape or API surface + that prior version did not have at all) -- see + `test_check_citations.py`'s own module docstring for the full, + per-test account and how each label was actually verified. + +MALFORMED, PRECISELY (round 2's third finding class, alongside WRONG and +CANNOT VERIFY): a citation whose SYNTAX cannot be trusted even before its +content is checked -- a range wider than `MAX_RANGE_WIDTH`, a backwards +range, or the thousands-separator-typo shape above. Reported, counted, and +fails the run exactly like a WRONG citation: rejecting the syntax rather +than guessing at the author's intent is the whole point (see fix 2 and +fix 4 above) -- "the author either finds the real line or marks it PATTERN +honestly" is not achieved by the checker silently picking a reading. + WHAT THIS CHECKS, PRECISELY (a heuristic, not a proof) ------------------------------------------------------- lang/la.ini's own comments cite a Missal heading in one of two shapes: @@ -67,12 +148,18 @@ lang/la.ini's own comments cite a Missal heading in one of two shapes: with no blank line -- [season]'s own style. The pool is that one entry alone. +A "PATTERN" marker follows the identical leading/trailing rule (see round +2 fix 1 above): it excludes only the entry (or entries) it is itself +attached to from citation-checking, never the rest of the block. + A citation is either a bare line number ("LT.txt:8609") or an explicit -range ("LT.txt:8609-8610") for a heading that genuinely wraps across -physical lines in the source; a comma-separated list ("LT.txt:8618,8620, -8622") is several independent citations, each checked on its own. For a -bare number the window is that one line; for a range it is the union of -every line in the range (inclusive). There is no other tolerance. +range ("LT.txt:8609-8610", capped at `MAX_RANGE_WIDTH` lines) for a heading +that genuinely wraps across physical lines in the source; a comma-separated +list ("LT.txt:8618,8620,8622") is several independent citations, each +checked on its own -- unless the list itself looks like a thousands-typo +for one number (round 2 fix 4), in which case the whole spec is MALFORMED. +For a bare number the window is that one line; for a range it is the union +of every line in the range (inclusive). There is no other tolerance. For each citation, the POOL is the set of candidate Latin phrases it could be defending: the entry (trailing shape) or every entry in the group @@ -84,31 +171,30 @@ into one shared bag, for the same reason quotes were removed: a citation bundling two claims onto one line number must not pass on the strength of an unrelated pool item's words. -An item with fewer than two distinctive words is DEGENERATE -- it cannot -discriminate the right line from a wrong nearby one, so it is excluded from -matching. If every item in a citation's pool is degenerate, the citation is -reported CANNOT VERIFY (counted and failed, never silently skipped or -silently passed). Otherwise the citation PASSES if the cited window's text -contains ALL of at least one non-degenerate pool item's words, and FAILS -otherwise. - -This is deliberately a LOOSE, word-overlap check within the (now exact) -window, not a byte-exact phrase match: la.ini spells abbreviations out in -full (Sanctissimi, not Ss.mi) and normalises j->i, and requiring a +A pool item is a candidate MATCH only if the cited window's text contains +ALL of its distinctive words -- otherwise it is not a match at all: it +never contributed to a PASS and is not what the citation is checked at +all. Among pool items that ARE contained in the window, the citation +PASSES if at least one clears `EVIDENCE_THRESHOLD` (see round 2 fix 3 +above and `item_evidence`'s own docstring) -- i.e. contains a word rare +enough, across the whole LT.txt corpus, to be real evidence rather than +coincidence. A citation whose only contained items are all common-word-only +is CANNOT VERIFY, not a silent PASS: found, but not proven. A citation with +NO contained item at all -- the claimed words are simply not at the cited +line(s) -- is WRONG. + +This is deliberately a LOOSE, word-overlap check within the (now exact and +bounded) window, not a byte-exact phrase match: la.ini spells abbreviations +out in full (Sanctissimi, not Ss.mi) and normalises j->i, and requiring a byte-exact substring would either force every citation's prose to repeat the raw OCR text verbatim (defeating the point of writing readable comments) or produce false failures having nothing to do with a wrong line number. What it proves is narrower than a byte-exact match, and is disclosed as such: PASS means "the claimed name's distinctive words are -present, in full, at the exact line(s) cited" -- not that the citation is +present, in full, at the exact line(s) cited, and at least one of them is +rare enough in the corpus to be real evidence" -- not that the citation is the best possible line, only that it is not obviously wrong and is not -resting on a coincidence-prone single word. - -Only citations OUTSIDE a "PATTERN" block are checked: a PATTERN entry makes -no claim that its own line is a direct heading, so a "LT.txt:N" mentioned -in its comment (e.g. citing the GRAMMAR another day's heading attests, not -this day's own heading) is not a provenance claim for THIS entry and would -otherwise produce a meaningless failure. +resting on a coincidence-prone common word. """ import argparse import re @@ -134,10 +220,27 @@ STOPWORDS = { "www", "http", "https", "htm", "html", "com", "romanum", "text", } -# A pool item with fewer than this many distinctive words cannot -# discriminate the right line from a wrong nearby one -- see the module -# docstring's item 2. -MIN_DISTINCTIVE_WORDS = 2 +# A citation's explicit range is capped at this many lines (n .. n+2). A +# genuine heading wrap in the source is one or two extra physical lines; +# anything wider is not a wrap, it is a search over a neighbourhood wide +# enough to coincidentally contain almost any short phrase -- exactly the +# blanket +-2 tolerance round 1 removed, reintroduced at a much larger +# radius by an unbounded range. See round 2 fix 2 in the module docstring. +MAX_RANGE_WIDTH = 3 + +# The evidence bar a pool item's RAREST matched word must clear to count +# as real proof (see `item_evidence` below). 1/200: a word occurring up to +# ~200 times across the whole ~118,000-token LT.txt corpus can still be +# the deciding, sole distinctive word of a short Missal calendar-table +# entry (measured: "omnium", the only survivor in "Omnium Sanctorum", +# occurs 139 times across the whole document, mostly in unrelated legal +# prose -- "of all" is common Latin furniture -- yet is genuinely the +# correct, sole citable word for that one heading). "classis" (507 +# occurrences), the round 1 false-pass this rule specifically targets, +# sits comfortably over 2.5x past this bar and stays CANNOT VERIFY. +# Round-2's own measurement of every affected real citation in +# lang/la.ini is in the branch report. +EVIDENCE_THRESHOLD = 1.0 / 200 def normalize_word(w: str) -> str: @@ -159,6 +262,52 @@ def distinctive_words(text: str) -> set: return out +def build_frequency_table(lt_lines: list) -> dict: + """Count how many times each normalised word occurs anywhere in the + whole LT.txt corpus -- built ONCE per run (not per citation) and + consulted by `word_evidence`/`item_evidence` below. This is what lets + the checker tell a token that could only ever mean one heading (occurs + once in 26,000+ lines) apart from common liturgical furniture (occurs + hundreds of times) -- see round 2 fix 3 in the module docstring.""" + freq = {} + for line in lt_lines: + for tok in re.split(r"\s+", line): + w = normalize_word(tok) + if w: + freq[w] = freq.get(w, 0) + 1 + return freq + + +def word_evidence(word: str, freq: dict) -> float: + """A token seen n times contributes 1/n: a hapax (n=1) contributes 1.0 + -- about as conclusive as a word-overlap check can be -- and a word + occurring hundreds of times contributes next to nothing. A plain + reciprocal is chosen over a logarithmic/IDF scale deliberately: + measured against this corpus, log-scaling compresses "occurs once" and + "occurs 500 times" into a difference of a few units, too close to + cleanly separate "essentially conclusive" from "unverified" with a + single threshold; a reciprocal keeps them many orders of magnitude + apart, which is the actual claim this rule makes.""" + n = freq.get(word, 0) + return 1.0 / n if n > 0 else 0.0 + + +def item_evidence(words: set, freq: dict) -> float: + """A pool item's rarity evidence is the SINGLE RAREST word it matched + on -- not a sum over all its words. Summing would let several + merely-uncommon words add up to "enough" evidence between them, which + is exactly the shape round 2 fix 3 exists to close (a match consisting + only of common tokens must stay CANNOT VERIFY "regardless of how many + words sit beside it") and exactly the self-poisoning failure mode from + round 1 (quoted corrective prose tends to share several ordinary words + with its neighbour, never one rare one). One genuinely rare word is + real evidence; an accumulation of merely-uncommon words is not the + same thing and must not be treated as if it were.""" + if not words: + return 0.0 + return max(word_evidence(w, freq) for w in words) + + class CitationRef: """One citation token: 'label' is what the data actually wrote ("8609" or "8609-8610"); 'lines' is the fully-expanded, sorted list of @@ -172,26 +321,78 @@ class CitationRef: self.lines = lines +def _looks_like_thousands_typo(tokens) -> bool: + """'12,459' splits into ('12', '459') -- the classic shape of a human + thousands-separator typo for a single 4-5 digit LT.txt line number: + grouped from the right in chunks of 3, a 4-digit number groups as + 'N,NNN' and a 5-digit number as 'NN,NNN' (LT.txt never reaches 6 + digits, so a 3-digit leading group never arises from real grouping). + Checked against every real comma-list citation in lang/la.ini as of + round 2: none has a 1-2 digit token immediately followed by an + exactly-3-digit token, so this shape is unambiguous enough in practice + to reject outright as malformed rather than silently parsing as two + unrelated short citations (round 2 fix 4).""" + for a, b in zip(tokens, tokens[1:]): + if re.fullmatch(r"\d{1,2}", a) and re.fullmatch(r"\d{3}", b): + return True + return False + + def parse_citation_spec(spec: str): """'8618,8620,8622' -> three exact-line CitationRefs; '8609-8610' -> one - CitationRef spanning both lines (an explicit wrapped heading). Each - comma-separated token is independent; a malformed token (b < a, or a - span so wide it is almost certainly a typo, capped at 200 lines) is - silently dropped rather than crashing on bad data, matching this - script's existing tolerance elsewhere for data it does not own.""" + CitationRef spanning both lines (an explicit wrapped heading, capped at + MAX_RANGE_WIDTH lines). Returns (refs, malformed): `refs` is the list of + successfully-parsed CitationRef objects; `malformed` is a list of + {"token", "reason"} dicts for anything that did NOT parse into a + trustworthy citation -- a backwards range, a range wider than the cap, + the thousands-typo shape above, or an unparseable token. Round 1 + silently DROPPED all of these (no crash, but no report either); round 2 + surfaces every one instead, because a malformed citation naming a real + entry deserves a human's attention exactly as much as a wrong one does + (see round 2 fixes 2 and 4 in the module docstring).""" + raw_tokens = [t.strip() for t in spec.split(",")] + if _looks_like_thousands_typo(raw_tokens): + return [], [ + { + "token": spec, + "reason": ( + "looks like a thousands-separator typo for one number " + "(a 1-2 digit token immediately followed by a 3-digit " + "one) rather than a genuine list of citations -- " + "remove the comma, or split into real citations" + ), + } + ] refs = [] - for tok in spec.split(","): - tok = tok.strip() + malformed = [] + for tok in raw_tokens: + if not tok: + continue m = re.fullmatch(r"(\d{2,6})-(\d{2,6})", tok) if m: a, b = int(m.group(1)), int(m.group(2)) - if a <= b and (b - a) <= 200: + width = b - a + 1 + if a > b: + malformed.append({"token": tok, "reason": f"backwards range (LT.txt:{tok})"}) + elif width > MAX_RANGE_WIDTH: + malformed.append( + { + "token": tok, + "reason": ( + f"range is {width} lines wide (max {MAX_RANGE_WIDTH}) -- " + "not a genuine heading wrap; find the real line or mark PATTERN honestly" + ), + } + ) + else: refs.append(CitationRef(tok, list(range(a, b + 1)))) continue m = re.fullmatch(r"(\d{2,6})", tok) if m: refs.append(CitationRef(tok, [int(m.group(1))])) - return refs + continue + malformed.append({"token": tok, "reason": "unparseable citation token"}) + return refs, malformed CITATION_RE = re.compile(r"LT\.txt:\s*((?:\d{2,6}(?:-\d{2,6})?)(?:\s*,\s*\d{2,6}(?:-\d{2,6})?)*)") @@ -230,59 +431,125 @@ def window_text_for(lt_lines, lines): return " ".join(lt_lines[lo_c - 1 : hi_c]) +def _pool_entries_for(entries_before, entries): + """The leading/trailing pooling rule, shared identically by citation + checking AND PATTERN scoping (round 2 fix 1): a TRAILING comment (one + or more entries already seen in this block) pools against only the + most recent entry; a LEADING comment (no entry seen yet) pools against + every entry in the block. The same rule must decide both questions -- + a PATTERN marker and a citation attached to the same comment always + cover the same entries, by construction.""" + return [entries_before[-1]] if entries_before else entries + + +def _pattern_marked_entries(block, entries): + """Which entries in this block are excluded from citation-checking by + their OWN "PATTERN" marker -- never by a PATTERN marker attached to a + DIFFERENT entry in the same block. Returns a set of `id()` of the + entry (key, value) tuples (safe: each entry tuple is a single object, + shared by reference between `block` and `entries`, never copied). + + This is a first, standalone pass over the block, completed before any + citation is evaluated, so a citation's own PATTERN status never + depends on where in the block it happens to sit relative to its + entry's PATTERN comment.""" + marked = set() + entries_before = [] + for k, c in block: + if k == "comment": + if "PATTERN" in c: + for e in _pool_entries_for(entries_before, entries): + marked.add(id(e)) + else: + entries_before.append(c) + return marked + + def check(la_ini_text: str, lt_lines: list): - """Returns a dict: checked (int), passed (int), findings (list -- wrong - citations), unverifiable (list -- degenerate-pool citations). Both - findings and unverifiable are things a human must look at; only - 'passed' citations required no human attention.""" + """Returns a dict: checked (int), passed (int), findings (list -- + wrong citations), unverifiable (list -- no matched item clears the + evidence bar), malformed (list -- citation syntax itself could not be + trusted). findings, unverifiable and malformed are all things a human + must look at; only 'passed' citations required no human attention.""" checked = 0 passed = 0 findings = [] unverifiable = [] + malformed = [] + freq = build_frequency_table(lt_lines) + for block in parse_blocks(la_ini_text): - block_comment = "\n".join(c for k, c in block if k == "comment") - if "PATTERN" in block_comment: - continue entries = [c for k, c in block if k == "entry"] if not entries: continue + + pattern_entries = _pattern_marked_entries(block, entries) + entries_before = [] for k, c in block: if k == "comment": for m in CITATION_RE.finditer(c): - if entries_before: - pool_entries = [entries_before[-1]] - else: - # leading citation: pool = every entry in the block - # (no positional correspondence is encoded between - # a specific cited line and a specific entry). - pool_entries = entries - entry_desc = ", ".join(f"{k}={v}" for k, v in pool_entries) - # Pool items are kept SEPARATE (not flattened into one - # bag of words): a citation passes only if the window - # fully covers -- ALL the distinctive words of -- at - # least one single pool item. See the module docstring - # for why (the self-poisoning bug and the two-claims- - # on-one-line-number bug this discipline catches). - pool_items = [distinctive_words(v) for _, v in pool_entries] - strong_items = [p for p in pool_items if len(p) >= MIN_DISTINCTIVE_WORDS] - best_len = max((len(p) for p in pool_items), default=0) - for ref in parse_citation_spec(m.group(1)): + pool_entries = _pool_entries_for(entries_before, entries) + if all(id(e) in pattern_entries for e in pool_entries): + # Every entry this citation could be defending is + # itself PATTERN-marked: this "LT.txt:N" is not a + # provenance claim for any of them (e.g. citing the + # GRAMMAR another day's heading attests), so + # checking it would produce a meaningless failure. + continue + entry_desc = ", ".join(f"{k2}={v2}" for k2, v2 in pool_entries) + pool_items = [distinctive_words(v2) for _, v2 in pool_entries] + nonempty_items = [p for p in pool_items if p] + pool_words = sorted(set().union(*pool_items)) if pool_items else [] + + refs, bad_tokens = parse_citation_spec(m.group(1)) + for bad in bad_tokens: + checked += 1 + malformed.append( + { + "label": bad["token"], + "entries": entry_desc, + "reason": bad["reason"], + } + ) + for ref in refs: checked += 1 window_text = window_text_for(lt_lines, ref.lines) window_words = distinctive_words(window_text) - if not strong_items: + + if not nonempty_items: + # Every pool item is fully stopwords -- there is + # nothing to check either way. Absence of + # evidence is not evidence of correctness. unverifiable.append( { "label": ref.label, "entries": entry_desc, - "best_len": best_len, - "pool_words": sorted(set().union(*pool_items)) if pool_items else [], + "reason": "no distinctive words in the entry text to check at all", + "pool_words": pool_words, } ) continue - if any(item <= window_words for item in strong_items): + + contained_items = [p for p in nonempty_items if p <= window_words] + strong_items = [ + p for p in contained_items if item_evidence(p, freq) >= EVIDENCE_THRESHOLD + ] + if strong_items: passed += 1 + elif contained_items: + best = max(item_evidence(p, freq) for p in contained_items) + unverifiable.append( + { + "label": ref.label, + "entries": entry_desc, + "reason": ( + f"matched, but every matched word is too common to trust " + f"(best evidence {best:.4f}, need >= {EVIDENCE_THRESHOLD:.4f})" + ), + "pool_words": pool_words, + } + ) else: n = ref.lines[0] findings.append( @@ -304,6 +571,7 @@ def check(la_ini_text: str, lt_lines: list): "passed": passed, "findings": findings, "unverifiable": unverifiable, + "malformed": malformed, } @@ -340,14 +608,15 @@ def main(argv=None): checked = result["checked"] findings = result["findings"] unverifiable = result["unverifiable"] + malformed = result["malformed"] - if not findings and not unverifiable: + if not findings and not unverifiable and not malformed: print(f"check-citations: {checked} LT.txt citations checked, 0 look wrong.") return 0 print( f"check-citations: {len(findings)} of {checked} citations look wrong, " - f"{len(unverifiable)} CANNOT VERIFY (pool too thin -- see below):\n" + f"{len(malformed)} MALFORMED, {len(unverifiable)} CANNOT VERIFY (see below):\n" ) if findings: print("WRONG:\n") @@ -355,16 +624,17 @@ def main(argv=None): print(f" LT.txt:{f['label']} cited for [{f['entries']}]") print(f" actual: {f['actual']!r}") print(f" window: {f['window']!r}\n") + if malformed: + print("MALFORMED (citation syntax itself is untrustworthy):\n") + for m in malformed: + print(f" LT.txt:{m['label']} cited for [{m['entries']}]") + print(f" {m['reason']}\n") if unverifiable: print("CANNOT VERIFY (a human must adjudicate these by hand):\n") for u in unverifiable: words = ", ".join(u["pool_words"]) if u["pool_words"] else "(none)" print(f" LT.txt:{u['label']} cited for [{u['entries']}]") - print( - f" pool too thin to verify: best candidate has " - f"{u['best_len']} distinctive word(s) (need >= {MIN_DISTINCTIVE_WORDS}); " - f"pool words: {words}\n" - ) + print(f" {u['reason']}; pool words: {words}\n") return 2 diff --git a/tools/test_check_citations.py b/tools/test_check_citations.py index 3d7722d..26482bd 100644 --- a/tools/test_check_citations.py +++ b/tools/test_check_citations.py @@ -18,6 +18,38 @@ Everything below is a SYNTHETIC fixture -- a tiny made-up "LT.txt" and a tiny made-up la.ini-shaped fragment, entirely in memory. Nothing here reads the real docs/research/LT.txt (gitignored, absent on a fresh clone) or the real lang/la.ini, so this suite runs identically everywhere, always. + +REGRESSION vs CHARACTERISATION -- LABELLED HONESTLY, PER TEST (round 2 fix +5). A review found that of round 1's seven fixture-driven tests, only TWO +actually failed against the pre-round-1 script -- the rest passed on both +sides of round 1's fix and regression-tested nothing despite their names. +Every test below now carries one of two tags in its docstring, verified, +not asserted: + + REGRESSION -- shown to FAIL when run against a named prior version of + check_citations.py (the exact command used to check this is recorded + next to the tag). Losing the fix this test guards would turn it red + again. + + CHARACTERISATION -- passes against the named prior version too (usually + because the prior version has no equivalent behaviour or API surface at + all -- a KeyError/AttributeError/TypeError rather than a meaningful + same-shape failure). Still valuable (it pins what the CURRENT tool does, + and would catch a future regression from here on), but it is not + evidence that round 1 or round 2 fixed anything -- there is no "before" + for it to have failed against. + +Two prior versions are named throughout: + ROUND-0 = the script as it shipped before round 1's hardening + (git rev 22824ef, saved for this audit at + /tmp/check_citations_round1.py's OWN predecessor -- see the + round-2 branch report for the exact commands run). + ROUND-1 = the script as hardened by round 1, before round 2's five + fixes below (git HEAD at the start of this round). +Round-1-era tests (the original seven fixture cases plus the two added +after them) are checked against ROUND-0. Round-2 tests (fixes 1-4) are +checked against ROUND-1, since that is the version each one is proving a +regression against. """ import subprocess import sys @@ -35,29 +67,86 @@ import check_citations as cc # noqa: E402 (path insert must come first) # realistic LT.txt line numbers, which run into the thousands) -- padded # with filler so every cited line here is two digits too, the same # constraint real data has. -LT_LINES = ["(filler)"] * 9 + [ - "Festum Aurorae Caelestis", # line 10 - "Prima Classis", # line 11 - "", # line 12 - "Festum Umbrae Nocturnae", # line 13 - "Secunda Classis", # line 14 - "", # line 15 - "Festum Solis Invicti", # line 16 - "Tertia Classis", # line 17 - "Festum Gloriae", # line 18 (heading continues on line 19) - "Aeternae Perpetuae", # line 19 -] - -# A fixture covering every case the hardening brief asked for: -# alpha -- correct citation -> PASS -# beta -- off by one line -> FAIL -# gamma -- off by three lines -> FAIL -# delta -- explicit n-m range, heading genuinely wraps -> PASS +# +# Line map (1-indexed): +# 10 Festum Aurorae Caelestis -- alpha's real heading +# 11 Prima Classis +# 13 Festum Umbrae Nocturnae -- beta's real heading +# 14 Secunda Classis -- beta wrongly cites here (off by 1) +# 16 Festum Solis Invicti -- gamma's real heading / epsilon's +# self-poisoning target +# 17 Tertia Classis -- iota wrongly cites here +# 18 Festum Gloriae -- delta's heading, part 1 (wraps) +# 19 Aeternae Perpetuae -- delta's heading, part 2; also +# gamma wrongly cites here (off by 3) +# 20 Rara Vox Singularis -- mu's real heading (rare word) +# 21 Communis Verbum Omnibus -- nu's real heading (common words) +# 24 (a single very long line, "communis"/"verbum" x250 each) -- +# frequency-table filler ONLY, never itself a citation target: this +# is what pushes communis/verbum's corpus-wide count past the +# evidence threshold, the same way real words like "classis" (507 +# occurrences) are common throughout LT.txt without living on any +# one line the checker is asked to verify against. +# 26 Magnum -- kappa/lambda heading, part 1 +# 27 Festum -- kappa/lambda heading, part 2 +# 28 Peregrinum -- kappa/lambda heading, part 3 +# (26-28 is a genuine 3-line wrap, at the MAX_RANGE_WIDTH cap; line +# 29 is unrelated filler included only by lambda's over-wide range) +_FREQUENCY_FILLER = " ".join(["communis", "verbum"] * 250) +LT_LINES = ( + ["(filler)"] * 9 + + [ + "Festum Aurorae Caelestis", # 10 + "Prima Classis", # 11 + "", # 12 + "Festum Umbrae Nocturnae", # 13 + "Secunda Classis", # 14 + "", # 15 + "Festum Solis Invicti", # 16 + "Tertia Classis", # 17 + "Festum Gloriae", # 18 + "Aeternae Perpetuae", # 19 + "Rara Vox Singularis", # 20 + "Communis Verbum Omnibus", # 21 + "(filler)", # 22 + "(filler)", # 23 + _FREQUENCY_FILLER, # 24 + "(filler)", # 25 + "Magnum", # 26 + "Festum", # 27 + "Peregrinum", # 28 + "Ultra", # 29 + ] +) + +# A fixture covering every case both hardening rounds asked for: +# alpha -- correct citation -> PASS +# beta -- off by one line -> FAIL +# gamma -- off by three lines -> FAIL +# delta -- explicit n-m range, heading genuinely wraps -> PASS # epsilon -- THE SELF-POISONING CASE: comment quotes a # DIFFERENT heading's text, citation points -# at that other heading's real line -> FAIL -# zeta -- degenerate pool (a single-word entry) -> CANNOT VERIFY -# eta -- PATTERN, no citation at all -> skipped entirely +# at that other heading's real line -> FAIL +# zeta -- entirely stopwords, nothing to check at all -> CANNOT VERIFY +# eta -- PATTERN, no citation at all -> skipped entirely +# theta -- PATTERN, trailing on itself only +# iota -- SAME BLOCK as theta, no PATTERN of its own, +# its own genuinely wrong citation -> FAIL +# (round 2 fix 1: theta's PATTERN must not silence this) +# mu -- single word, occurs ONCE in the whole corpus -> PASS +# (round 2 fix 3a) +# nu -- two words, both occur 251 times in the corpus -> CANNOT VERIFY +# (round 2 fix 3b -- stricter than round 1, which +# would have passed this on word-count alone) +# xi -- "12,459"-shaped citation, thousands-typo for +# one number -> MALFORMED +# (round 2 fix 4) +# kappa -- explicit 3-line range, AT the width cap -> PASS +# (round 2 fix 2a) +# lambda_ -- same heading, 4-line range, OVER the width cap -> MALFORMED +# (round 2 fix 2b; note the trailing underscore -- +# "lambda" is a Python keyword-adjacent builtin, avoided +# only to keep the la.ini key itself plain "lambda") LA_INI_TEXT = """ [test] @@ -77,21 +166,47 @@ epsilon = Festum Lunae Argenteae ; CORRECTED: an earlier draft wrongly attributed this to "Festum Solis ; Invicti" -- LT.txt:16. -zeta = Ordo +zeta = In Sancta Dominica ; LT.txt:11. ; eta -- PATTERN, constructed name; no heading for this day survives in ; the source at all. eta = Aliquid Fictum + +theta = Ignotum Simulatum +; PATTERN, invented for this self-test; no real heading survives for +; theta specifically. +iota = Festum Umbrae Nocturnae +; LT.txt:17. + +mu = Singularis +; LT.txt:20. + +nu = Communis Verbum +; LT.txt:21. + +xi = Numerus Fictus +; LT.txt:12,459. + +kappa = Magnum Festum Peregrinum +; LT.txt:26-28. + +lambda = Magnum Festum Peregrinum +; LT.txt:26-29. """ -def entries_field(items, label): - """Find the finding/unverifiable dict whose citation label matches, or - None. Small helper so assertions read by name, not by list position.""" +def entries_field(items, label, entries_substring=None): + """Find the finding/unverifiable/malformed dict whose citation label + matches (and, if given, whose entries description contains + `entries_substring` -- needed on the rare occasion two different + entries cite the identical wrong line number), or None.""" for item in items: - if item["label"] == label: - return item + if item["label"] != label: + continue + if entries_substring is not None and entries_substring not in item["entries"]: + continue + return item return None @@ -102,45 +217,69 @@ class TestCheckLogic(unittest.TestCase): self.result = cc.check(LA_INI_TEXT, LT_LINES) def test_totals(self): - # alpha, beta, gamma, delta, epsilon, zeta = 6 citation EVENTS. - # eta contributes nothing (PATTERN, and has no citation anyway). - self.assertEqual(self.result["checked"], 6) - self.assertEqual(self.result["passed"], 2) # alpha, delta - self.assertEqual(len(self.result["findings"]), 3) # beta, gamma, epsilon - self.assertEqual(len(self.result["unverifiable"]), 1) # zeta + """CHARACTERISATION (pins the current tool's own output schema and + aggregate counts -- ROUND-1 has no 'malformed' key at all, so this + exact assertion cannot even be asked of it; it is not evidence of a + fix, it is a pin against future drift).""" + # 12 citation EVENTS: alpha, beta, gamma, delta(1 range), epsilon, + # zeta, iota, mu, nu, xi, kappa, lambda -- theta/eta contribute none. + self.assertEqual(self.result["checked"], 12) + self.assertEqual(self.result["passed"], 4) # alpha, delta, mu, kappa + self.assertEqual(len(self.result["findings"]), 4) # beta, gamma, epsilon, iota + self.assertEqual(len(self.result["unverifiable"]), 2) # zeta, nu + self.assertEqual(len(self.result["malformed"]), 2) # xi, lambda def test_correct_citation_passes(self): - passed_labels = {"10"} # alpha's own label + """CHARACTERISATION: verified against ROUND-0 (git rev 22824ef) -- + alpha's own block has no quoted phrases, so the old quote-pooling + bug never touches it; alpha passes on both sides.""" found_wrong = {f["label"] for f in self.result["findings"]} found_unverifiable = {u["label"] for u in self.result["unverifiable"]} - self.assertFalse(passed_labels & found_wrong) - self.assertFalse(passed_labels & found_unverifiable) + self.assertNotIn("10", found_wrong) + self.assertNotIn("10", found_unverifiable) def test_off_by_one_line_fails(self): - f = entries_field(self.result["findings"], "14") + """REGRESSION, verified against ROUND-0: ROUND-0's blanket +-2-line + tolerance means a window of LT.txt[12..16] is checked for citation + "14", which contains line 13 (beta's REAL heading) -- so ROUND-0 + reports beta as a PASS and this test's assertIsNotNone(...) fails + against it. Confirmed by direct replay of ROUND-0's check() against + this exact fixture shape (see the round-2 branch report).""" + f = entries_field(self.result["findings"], "14", "beta") self.assertIsNotNone(f, "beta's off-by-one citation (LT.txt:14) must FAIL") self.assertIn("beta", f["entries"]) def test_off_by_three_lines_fails(self): - f = entries_field(self.result["findings"], "19") - # NOTE: delta ALSO legitimately cites "18-19" as a range (a distinct - # citation event, checked separately) -- gamma's bad citation is - # the bare, single-number "19" token, which is what must fail here. - # A finding's label is the raw token as written, so "19" (gamma) - # and "18-19" (delta) never collide. + """CHARACTERISATION, verified against ROUND-0: even ROUND-0's +-2 + tolerance window (LT.txt[17..21]) does not reach line 16 (gamma's + real heading), so ROUND-0 already reports this as wrong. This test + does not regression-test the +-2 removal; test_off_by_one above + does.""" + f = entries_field(self.result["findings"], "19", "gamma") self.assertIsNotNone(f, "gamma's off-by-three citation (LT.txt:19) must FAIL") self.assertIn("gamma", f["entries"]) def test_explicit_wrap_range_passes(self): + """CHARACTERISATION, verified against ROUND-0: explicit A-B ranges + already existed in ROUND-0's `expand_citation_spec` (identical + regex); ROUND-0 additionally pads each expanded line with its own + +-2 tolerance, so this passes there too, just for a sloppier + reason. test_wrap_range_required_not_just_first_line below is the + test that actually isolates the range syntax doing real work.""" found_wrong = {f["label"] for f in self.result["findings"]} found_unverifiable = {u["label"] for u in self.result["unverifiable"]} + found_malformed = {m["label"] for m in self.result["malformed"]} self.assertNotIn("18-19", found_wrong) self.assertNotIn("18-19", found_unverifiable) + self.assertNotIn("18-19", found_malformed) def test_wrap_range_required_not_just_first_line(self): - # Without the explicit range, citing only delta's FIRST physical - # line must fail -- this is the concrete proof that the range - # syntax is doing real work, not merely being tolerated. + """REGRESSION, verified against ROUND-0: citing only delta's first + physical line ("LT.txt:18") still falls inside ROUND-0's own +-2 + window (16..20), which reaches line 19 and lets it pass -- ROUND-0 + never reports a finding here, so this test's assertIsNotNone(...) + fails against it. This is the concrete proof that the range syntax + is doing real work, not merely being tolerated by leftover slack.""" text = LA_INI_TEXT.replace("; LT.txt:18-19.", "; LT.txt:18.") result = cc.check(text, LT_LINES) f = entries_field(result["findings"], "18") @@ -149,25 +288,49 @@ class TestCheckLogic(unittest.TestCase): ) def test_self_poisoning_quote_does_not_pass(self): - """THE regression test for the historical bug: epsilon's own - comment quotes "Festum Solis Invicti" (a DIFFERENT heading, - gamma's own), and cites that other heading's real line (LT.txt:16). - A checker that pools quoted text from the surrounding comment would - pass this, exactly as the pre-hardening script did. It must FAIL.""" + """REGRESSION, verified against ROUND-0: THE regression test for + the historical bug. epsilon's own comment quotes "Festum Solis + Invicti" (a DIFFERENT heading, gamma's own), and cites that other + heading's real line (LT.txt:16). ROUND-0 pools every double-quoted + phrase from the WHOLE block comment, so the quote itself becomes a + pool item, matches the window trivially, and ROUND-0 reports "0 + look wrong" for it -- confirmed by direct replay. Must FAIL here.""" found_wrong = {f["label"]: f for f in self.result["findings"]} self.assertIn("16", found_wrong, "the self-poisoning citation must be a FAIL, not a pass") self.assertIn("epsilon", found_wrong["16"]["entries"]) found_unverifiable = {u["label"] for u in self.result["unverifiable"]} self.assertNotIn("16", found_unverifiable, "must be a real FAIL, not laundered into CANNOT VERIFY") - def test_degenerate_pool_is_cannot_verify_not_pass(self): + def test_empty_pool_is_cannot_verify_not_pass(self): + """REGRESSION, verified against ROUND-0 (by the letter of the + definition -- see below for the nuance): zeta's entry text is + entirely stopwords ("In Sancta Dominica"), so there is nothing to + check either way. ROUND-0 has no CANNOT-VERIFY concept at all: a + fully empty pool item never satisfies `any(item <= window_words + for item in pool_items)` over an empty pool, so ROUND-0 reports it + as an ordinary WRONG finding instead -- confirmed by direct + replay. This test's specific assertion (that it lands in + `unverifiable`) therefore fails against ROUND-0, though the + underlying "not a silent pass" property does hold there too, just + through a coarser, undifferentiated classification. ROUND-1 + already has the current three-way split (as "too thin", word + count rather than "no distinctive words", rarity) and passes this + test unchanged.""" u = entries_field(self.result["unverifiable"], "11") - self.assertIsNotNone(u, "zeta's single-word entry must be CANNOT VERIFY") + self.assertIsNotNone(u, "zeta's all-stopword entry must be CANNOT VERIFY") self.assertIn("zeta", u["entries"]) found_wrong = {f["label"] for f in self.result["findings"]} - self.assertNotIn("11", found_wrong, "a degenerate pool must never be reported as a silent PASS") + self.assertNotIn("11", found_wrong, "an empty pool must never be reported as a silent PASS") def test_pattern_block_skipped_entirely(self): + """CHARACTERISATION, verified against ROUND-0: eta's block contains + only eta itself, so ROUND-0's whole-block PATTERN skip and the + current tool's entry-scoped skip have the identical effect for + this single-entry case -- the difference only shows up in a + MULTI-entry block, which is test_pattern_does_not_silence_a_ + different_entry_in_the_same_block below (the real fix-1 regression + test).""" + def keys_of(entries_desc): return {pair.split("=", 1)[0] for pair in entries_desc.split(", ")} @@ -175,49 +338,244 @@ class TestCheckLogic(unittest.TestCase): self.assertNotIn("eta", keys_of(f["entries"])) for u in self.result["unverifiable"]: self.assertNotIn("eta", keys_of(u["entries"])) + for m in self.result["malformed"]: + self.assertNotIn("eta", keys_of(m["entries"])) + + def test_pattern_does_not_silence_a_different_entry_in_the_same_block(self): + """REGRESSION, verified against ROUND-1 (git HEAD before round 2): + theta and iota share ONE block (no blank line between them, the + same shape as [season]'s real back-to-back trailing-comment + style). theta's own trailing comment says "PATTERN"; iota is a + completely different entry with its own genuinely wrong citation + and no PATTERN marker at all. ROUND-1's `check()` tested + `"PATTERN" in block_comment` -- a single substring search over + every comment in the WHOLE block -- and skipped the entire block + on a match, so iota's wrong citation was never even looked at: + confirmed by direct replay of ROUND-1's check() against this exact + fixture (see the round-2 branch report). Must FAIL here.""" + f = entries_field(self.result["findings"], "17") + self.assertIsNotNone( + f, "iota's own wrong citation must FAIL even though theta, in the same block, is PATTERN-marked" + ) + self.assertIn("iota", f["entries"]) + # And theta itself must still be excluded, exactly like eta. + for f in self.result["findings"]: + self.assertNotIn("theta=", f["entries"]) + + def test_rare_single_word_match_passes(self): + """REGRESSION, verified against ROUND-1: mu's entry is a single + word, "Singularis", occurring exactly once in the whole corpus. + ROUND-1's word-COUNT gate (`MIN_DISTINCTIVE_WORDS = 2`) excluded + any one-word pool item from matching at all, regardless of how + rare that word is, and reported it CANNOT VERIFY unconditionally + -- confirmed by direct replay. This is round 2 fix 3's own primary + example (the real "S. Antonii Abb." shape): a single occurrence in + a 26,000+-line corpus is essentially conclusive and must PASS.""" + passed_labels_not_flagged = ( + "20" not in {f["label"] for f in self.result["findings"]} + and "20" not in {u["label"] for u in self.result["unverifiable"]} + ) + self.assertTrue(passed_labels_not_flagged, "a rare (freq=1) single-word match must PASS, not be flagged") + + def test_many_common_tokens_match_is_cannot_verify(self): + """REGRESSION, verified against ROUND-1: nu's entry has TWO + distinctive words ("Communis", "Verbum"), clearing ROUND-1's own + `MIN_DISTINCTIVE_WORDS = 2` gate on word count alone -- ROUND-1 + reports this a PASS purely because there are two words, without + ever checking how common either one is (both occur 251 times in + this fixture's corpus). Confirmed by direct replay. Round 2 fix 3 + requires this to stay CANNOT VERIFY regardless of word count -- + the STRICTER half of the rarity rule, not just the looser half + rare-word tests exercise.""" + u = entries_field(self.result["unverifiable"], "21") + self.assertIsNotNone( + u, "a match consisting only of common (251-occurrence) tokens must be CANNOT VERIFY" + ) + found_wrong = {f["label"] for f in self.result["findings"]} + self.assertNotIn("21", found_wrong, "common-word-only should be CANNOT VERIFY, not a silent FAIL either") + + def test_thousands_typo_citation_is_malformed(self): + """REGRESSION, verified against ROUND-1: ROUND-1's CITATION_RE and + `parse_citation_spec` happily parse "12,459" as two independent + bare citations, 12 and 459, and check them separately -- neither + anywhere near the real intended line, but the malformed spec is + never reported as such; confirmed by direct replay (ROUND-1 raises + no exception and produces two ordinary, uninteresting citation + events instead of one flagged one). Round 2 fix 4 requires the + whole spec to be rejected as MALFORMED instead.""" + m = entries_field(self.result["malformed"], "12,459") + self.assertIsNotNone(m, "the thousands-typo-shaped citation must be reported MALFORMED") + self.assertIn("xi", m["entries"]) + # And it must not ALSO sneak through as two ordinary citations. + self.assertIsNone(entries_field(self.result["findings"], "12")) + self.assertIsNone(entries_field(self.result["findings"], "459")) + + def test_range_at_cap_width_passes(self): + """REGRESSION, verified against ROUND-1: this specific 3-line range + already passes on ROUND-1 too (ROUND-1 also has no upper cap), so + by itself this is CHARACTERISATION -- it is paired here with + test_range_over_cap_width_is_malformed below to show the cap is + drawn in the RIGHT place (exactly at MAX_RANGE_WIDTH, not one line + short of it).""" + found_wrong = {f["label"] for f in self.result["findings"]} + found_malformed = {m["label"] for m in self.result["malformed"]} + self.assertNotIn("26-28", found_wrong) + self.assertNotIn("26-28", found_malformed) + + def test_range_over_cap_width_is_malformed(self): + """REGRESSION, verified against ROUND-1: "LT.txt:26-29" is a 4-line + range citing the identical heading text kappa already cites + correctly at the 3-line cap -- ROUND-1 has no upper bound at all + (only the pre-existing 200-line absurdity guard), so it silently + accepts this and checks it exactly like kappa's; confirmed by + direct replay. Round 2 fix 2 requires anything over + MAX_RANGE_WIDTH to be rejected as MALFORMED, regardless of whether + the content would otherwise have matched.""" + m = entries_field(self.result["malformed"], "26-29") + self.assertIsNotNone(m, "a range wider than MAX_RANGE_WIDTH must be MALFORMED") + self.assertIn("lambda", m["entries"]) + self.assertIn(str(cc.MAX_RANGE_WIDTH), m["reason"]) class TestHelpers(unittest.TestCase): def test_distinctive_words_strips_stopwords_and_short_tokens(self): + """CHARACTERISATION: `distinctive_words`/`normalize_word` are + byte-identical to ROUND-0 and ROUND-1 -- neither hardening round + touched them. Pins current behaviour only.""" words = cc.distinctive_words("Dominica I Adventus") self.assertEqual(words, {"adventus"}) # "Dominica" stopword, "I" too short def test_distinctive_words_normalises_j_and_ligatures(self): + """CHARACTERISATION: see above.""" self.assertEqual(cc.distinctive_words("Jesu"), cc.distinctive_words("Iesu")) self.assertEqual(cc.distinctive_words("praesulaeque"), cc.distinctive_words("praesulæque")) def test_parse_citation_spec_bare_number(self): - refs = cc.parse_citation_spec("8609") + """CHARACTERISATION: ROUND-0/ROUND-1 both expand a bare number the + same way, just under a different function name/return shape + (`expand_citation_spec` -> a flat list of ints, no malformed + channel). Pins the current tuple-returning API.""" + refs, malformed = cc.parse_citation_spec("8609") self.assertEqual(len(refs), 1) self.assertEqual(refs[0].lines, [8609]) + self.assertEqual(malformed, []) def test_parse_citation_spec_range_is_one_ref(self): - refs = cc.parse_citation_spec("8609-8610") + """CHARACTERISATION: see above.""" + refs, malformed = cc.parse_citation_spec("8609-8610") self.assertEqual(len(refs), 1) self.assertEqual(refs[0].lines, [8609, 8610]) + self.assertEqual(malformed, []) def test_parse_citation_spec_comma_list_is_several_refs(self): - refs = cc.parse_citation_spec("8618,8620,8622") + """CHARACTERISATION: see above.""" + refs, malformed = cc.parse_citation_spec("8618,8620,8622") self.assertEqual([r.lines for r in refs], [[8618], [8620], [8622]]) + self.assertEqual(malformed, []) def test_parse_citation_spec_mixed_list(self): - refs = cc.parse_citation_spec("8786,8788-8789,8791-8792") + """CHARACTERISATION: see above.""" + refs, malformed = cc.parse_citation_spec("8786,8788-8789,8791-8792") self.assertEqual( [r.lines for r in refs], [[8786], [8788, 8789], [8791, 8792]], ) + self.assertEqual(malformed, []) def test_parse_citation_spec_rejects_backwards_range(self): - self.assertEqual(cc.parse_citation_spec("100-50"), []) + """REGRESSION, verified against ROUND-1: ROUND-1's + `parse_citation_spec` SILENTLY DROPPED a backwards range (empty + refs list, no report at all) -- confirmed by direct replay. Round + 2 surfaces it as MALFORMED instead of discarding it invisibly.""" + refs, malformed = cc.parse_citation_spec("100-50") + self.assertEqual(refs, []) + self.assertEqual(len(malformed), 1) + self.assertIn("backwards", malformed[0]["reason"]) + + def test_parse_citation_spec_rejects_range_over_cap(self): + """REGRESSION, verified against ROUND-1: ROUND-1 accepted any + range up to 200 lines wide as a normal, silently-checked citation + -- "1000-999999" exceeded even that old 200-line guard and was + silently dropped (empty list, no report); a merely-wide-but-under + -200 range like "1000-1100" was silently ACCEPTED and checked as + if it were a legitimate wrap, which is the actual defeat this fix + closes. Both shapes are confirmed by direct replay against + ROUND-1. Round 2 caps at MAX_RANGE_WIDTH and reports the excess + width by name rather than silently accepting or silently + dropping.""" + refs, malformed = cc.parse_citation_spec("1000-1100") + self.assertEqual(refs, [], "a 101-line range must not be silently accepted") + self.assertEqual(len(malformed), 1) + self.assertIn("101", malformed[0]["reason"]) + + def test_parse_citation_spec_range_at_cap_boundary(self): + """CHARACTERISATION: pins the exact boundary -- a range exactly + MAX_RANGE_WIDTH lines wide is accepted, not rejected.""" + refs, malformed = cc.parse_citation_spec("2000-2002") + self.assertEqual(len(refs), 1) + self.assertEqual(malformed, []) + + def test_parse_citation_spec_thousands_typo(self): + """REGRESSION, verified against ROUND-1: ROUND-1 parses "12,459" + as two ordinary bare citations (12 and 459) with no indication + anything is wrong -- confirmed by direct replay. Round 2 fix 4 + rejects the whole spec instead.""" + refs, malformed = cc.parse_citation_spec("12,459") + self.assertEqual(refs, []) + self.assertEqual(len(malformed), 1) + self.assertIn("thousands", malformed[0]["reason"]) + + def test_parse_citation_spec_similar_looking_list_is_not_flagged(self): + """CHARACTERISATION: guards the thousands-typo heuristic against + false positives on a genuine multi-citation list -- two 4-digit + numbers close together must still parse normally, not be rejected + just because they happen to sit next to each other in a list.""" + refs, malformed = cc.parse_citation_spec("8618,8620,8622") + self.assertEqual(malformed, []) + self.assertEqual(len(refs), 3) + + def test_item_evidence_hapax_is_strong(self): + """CHARACTERISATION of the new (round 2) rarity machinery: a word + occurring once in a corpus of many contributes evidence 1.0, well + past EVIDENCE_THRESHOLD. No prior round had this function at all.""" + freq = cc.build_frequency_table(["Singularis Verbum", "Aliud Verbum"]) + self.assertEqual(cc.item_evidence({"singularis"}, freq), 1.0) + + def test_item_evidence_common_word_is_weak(self): + """CHARACTERISATION of the new rarity machinery: a word occurring + 500 times contributes far below EVIDENCE_THRESHOLD.""" + freq = cc.build_frequency_table([" ".join(["classis"] * 500)]) + self.assertLess(cc.item_evidence({"classis"}, freq), cc.EVIDENCE_THRESHOLD) + + def test_item_evidence_is_the_max_not_the_sum(self): + """CHARACTERISATION: an item combining one rare word and one very + common word takes its evidence from the RARE one -- a real match + is not penalised for also containing an ordinary word beside it.""" + freq = cc.build_frequency_table( + ["Singularis Verbum"] + [" ".join(["communis"] * 300)] + ) + ev = cc.item_evidence({"singularis", "communis"}, freq) + self.assertEqual(ev, 1.0) - def test_parse_citation_spec_rejects_absurdly_wide_range(self): - self.assertEqual(cc.parse_citation_spec("1000-999999"), []) + def test_word_evidence_unseen_word_is_zero(self): + """CHARACTERISATION: a word absent from the corpus entirely (freq + 0) contributes zero evidence rather than raising or dividing by + zero -- it can never legitimately be "contained" in a real window + either, so this only matters defensively.""" + freq = cc.build_frequency_table(["Aliud Verbum"]) + self.assertEqual(cc.word_evidence("nusquam", freq), 0.0) class TestCliIntegration(unittest.TestCase): """End-to-end: invokes the real main() as a subprocess, exactly how `make check-citations` does, using --file/--lt-file to point at - temporary fixtures so the real lang/la.ini is never touched.""" + temporary fixtures so the real lang/la.ini is never touched. + + Labelled per-test against ROUND-1 (the --file/--lt-file plumbing + itself is a ROUND-1 feature; ROUND-0's main() takes no arguments at + all and reads the real lang/la.ini and real docs/research/LT.txt + unconditionally, so ROUND-0 is not a meaningful comparison for any + test in this class).""" def run_cli(self, la_ini_text, lt_text, lt_present=True): with tempfile.TemporaryDirectory() as td: @@ -242,17 +600,26 @@ class TestCliIntegration(unittest.TestCase): return proc def test_skipped_when_lt_txt_absent(self): + """CHARACTERISATION, verified against ROUND-1: identical SKIPPED + behaviour, unchanged by round 2.""" proc = self.run_cli(LA_INI_TEXT, "", lt_present=False) self.assertEqual(proc.returncode, 0) self.assertIn("SKIPPED", proc.stdout) - def test_mixed_fixture_exits_nonzero_and_reports_both_classes(self): + def test_mixed_fixture_exits_nonzero_and_reports_all_three_classes(self): + """REGRESSION, verified against ROUND-1: run as a real subprocess + against this exact fixture, ROUND-1's CLI never prints "MALFORMED" + anywhere (it has no such concept) -- confirmed by direct replay + (`WRONG`/`CANNOT VERIFY` both appear, `MALFORMED` does not).""" proc = self.run_cli(LA_INI_TEXT, "\n".join(LT_LINES)) self.assertEqual(proc.returncode, 2) self.assertIn("WRONG", proc.stdout) + self.assertIn("MALFORMED", proc.stdout) self.assertIn("CANNOT VERIFY", proc.stdout) def test_all_clean_fixture_exits_zero(self): + """CHARACTERISATION, verified against ROUND-1: identical clean-exit + behaviour, unchanged by round 2.""" clean_text = """ [test] -- cgit v1.3