diff options
Diffstat (limited to 'tools')
| -rwxr-xr-x | tools/check_citations.py | 642 | ||||
| -rw-r--r-- | tools/dune | 19 | ||||
| -rw-r--r-- | tools/test_check_citations.py | 635 |
3 files changed, 1296 insertions, 0 deletions
diff --git a/tools/check_citations.py b/tools/check_citations.py new file mode 100755 index 0000000..688b00e --- /dev/null +++ b/tools/check_citations.py @@ -0,0 +1,642 @@ +#!/usr/bin/env python3 +"""check_citations.py -- verify every "LT.txt:<n>" citation in lang/la.ini +actually resolves to the Latin text it claims, in docs/research/LT.txt. + +Run via `make check-citations` (or directly: `python3 tools/check_citations.py +[--file LA_INI] [--lt-file LT_TXT]`). Exits 2 with a report if any citation +looks WRONG, is MALFORMED, or CANNOT BE VERIFIED; exits 0 (with a summary +line) only if every citation checked out cleanly; exits 0 with a loud +"SKIPPED" line if docs/research/LT.txt is not present (it is gitignored -- +see below). + +THIS SCRIPT WAS ITSELF FOUND TO BE SELF-POISONING (2026-08-19) AND HARDENED +------------------------------------------------------------------------ +An earlier version pooled distinctive words from the whole COMMENT BLOCK +around a citation, including any double-quoted phrase the comment happened +to mention -- and comments routinely quote a WRONG historical value while +explaining a past fix (e.g. "CORRECTED: previously cited LT.txt:12459, +which is 'Dominica ultima Octobris', not this heading"). That quote landed +in the pool, so re-introducing the exact bug being documented -- citing +LT.txt:12459 again -- matched the very quote correcting it, and the script +reported "0 look wrong". A verification tool whose own documentation of a +fix defeats the check for that fix is worse than no tool: it manufactures +false confidence. Proven with a reproduction: reintroducing that one wrong +citation into a real copy of lang/la.ini produced zero findings on the +pre-hardening script. Four changes closed this, in order of how directly +each one addresses the reproduction: + + 1. THE POOL IS SCOPED TO THE CITATION'S OWN ENTRY, never to the + surrounding comment's quoted text. A citation is verified against the + name it claims, not against anything quoted nearby -- see + `distinctive_words` / the per-citation `pool_items` construction + below. This alone closes the reproduced bug (see `test_check_citations.py`'s + own `test_self_poisoning_quote_does_not_pass`). + 2. A POOL TOO THIN TO VERIFY FAILS CLOSED (round 1's shape; round 2 + replaced the mechanism -- see below -- but kept the discipline: no + match is ever reported as a silent PASS just because nothing better + was checked). + 3. NO MORE BLANKET +-2-LINE TOLERANCE. A citation is checked at the EXACT + line it names. A heading that genuinely spans more than one physical + line in the source must say so explicitly, "LT.txt:8609-8610" -- the + allowance moves into the data, where a reader (and a future citation + added nearby) can see it, instead of silently forgiving ANY citation + within two lines of the truth. A bare "LT.txt:N" is checked at line N + only. + 4. THIS SCRIPT NOW HAS ITS OWN TEST SUITE (test_check_citations.py, + wired into `dune test` via tools/dune's own runtest rule) -- the + single most important change. The reproduced bug survived as long as + it did specifically BECAUSE nothing exercised this script's own + logic against a known-wrong citation. The self-poisoning case above + is now a permanent regression test. + +ROUND 2 (2026-08-19): A REVIEW DEFEATED ROUND 1 AGAIN -- FIVE MORE FIXES +------------------------------------------------------------------------ +Round 1's own hardening had three further holes, all found live against +this exact file, plus two smaller defects. Fixed in this order (matching +the order the defeats were found in, most dangerous first): + + 1. "; PATTERN" SILENCED A WHOLE BLOCK, NOT JUST ITS OWN ENTRY. `check()` + used to test `"PATTERN" in block_comment` -- one substring search over + every comment in the block -- and `continue` past the ENTIRE block if + it matched anywhere. In a block that mixes a PATTERN-marked entry with + an ordinary entry carrying its own, genuinely wrong citation (exactly + [season]'s own trailing-comment style: several entries back-to-back + with no blank line between them), the wrong citation was never even + looked at. Fixed by scoping PATTERN the same way citation pools are + already scoped: a PASS over the block first collects which entries a + PATTERN-bearing comment actually covers (the one preceding entry for a + trailing comment, every entry in the block for a leading one -- the + identical leading/trailing rule `check()` already uses for citation + pooling), and only THOSE entries are excluded from checking. See + `test_pattern_does_not_silence_a_different_entry_in_the_same_block`. + 2. EXPLICIT RANGES HAD NO UPPER BOUND. Round 1 replaced the old blanket + +-2-line tolerance with an explicit per-citation range + ("LT.txt:8609-8610") specifically so a wrap allowance would be + visible in the data instead of invisible in the checker -- but set no + cap on how WIDE that range could be. "LT.txt:8600-8650" passed if the + claimed text appeared anywhere across fifty lines: the same + "tolerance forgives a wrong line" defect the +-2 removal was meant to + close, reintroduced at 25x the radius, now with the data's own + apparent blessing. Fixed with `MAX_RANGE_WIDTH` (see below): a range + wider than a genuine heading wrap (1-2 extra physical lines) is + rejected as MALFORMED, naming the entry and the width, rather than + silently accepted. See `test_range_wider_than_cap_is_malformed`. + 3. THE "TOO THIN TO VERIFY" GATE COUNTED WORDS, NOT RARITY. Round 1 + required a pool item to have >=2 distinctive words before it could + even be checked, on the theory that a one-word match proves nothing. + Measured against the real data: this flagged 11 CORRECT citations + CANNOT VERIFY -- every one a short Latin hagionym whose Missal + heading genuinely has only one non-stopword ("S. Antonii Abb.": only + "antonii" survives the stopword filter) -- while a match on nothing + but "classis" (507 occurrences across LT.txt) passed the >=2 bar + freely whenever it shared a citation with three siblings ("I classis + / II classis / III classis / IV classis"). Word COUNT was never the + right proxy; word RARITY is. Replaced with `build_frequency_table` + + `item_evidence`: a token seen n times in the whole corpus contributes + 1/n of evidence, an item's evidence is its single RAREST matched + token (not a sum -- see `item_evidence`'s own docstring for why + summing would reopen this exact class of bug), and `EVIDENCE_THRESHOLD` + is the bar a match must clear to count as real proof rather than + coincidence. See `test_rare_single_word_match_passes` and + `test_common_word_only_match_is_cannot_verify`. + 4. A COMMA TYPO SILENTLY BECAME A DIFFERENT CITATION. "LT.txt:12,459" (a + stray thousands-separator comma for the single number "12459") used + to parse as TWO independent bare citations, "12" and "459" -- either + of which might coincidentally match somewhere nearby while the + intended line was never checked at all. `parse_citation_spec` now + recognises the shape (a 1-2 digit token immediately followed by an + exactly-3-digit token -- the only way a thousands-grouped LT.txt line + number, which never exceeds 5 digits, can be split by one comma) and + rejects the whole spec as MALFORMED rather than silently reinterpreting + it. See `test_thousands_separator_typo_is_malformed`. + 5. THE SELF-TEST SUITE OVERSTATED ITS OWN COVERAGE. A review ran round + 1's seven fixture cases against the PRE-round-1 script: only two + (`test_off_by_one_line_fails`, `test_self_poisoning_quote_does_not_pass`) + actually failed on it -- the rest passed on both sides of round 1's + fix and regression-tested nothing despite their names (a third, + `test_wrap_range_required_not_just_first_line`, was added after the + "seven" and also turns out to be a genuine regression test, confirmed + the same way). Every test in this suite is now labelled REGRESSION + (shown, not just claimed, to fail against a named prior version) or + CHARACTERISATION (pins current behaviour; does not fail on the prior + version, usually because it exercises a data shape or API surface + that prior version did not have at all) -- see + `test_check_citations.py`'s own module docstring for the full, + per-test account and how each label was actually verified. + +MALFORMED, PRECISELY (round 2's third finding class, alongside WRONG and +CANNOT VERIFY): a citation whose SYNTAX cannot be trusted even before its +content is checked -- a range wider than `MAX_RANGE_WIDTH`, a backwards +range, or the thousands-separator-typo shape above. Reported, counted, and +fails the run exactly like a WRONG citation: rejecting the syntax rather +than guessing at the author's intent is the whole point (see fix 2 and +fix 4 above) -- "the author either finds the real line or marks it PATTERN +honestly" is not achieved by the checker silently picking a reading. + +WHAT THIS CHECKS, PRECISELY (a heuristic, not a proof) +------------------------------------------------------- +lang/la.ini's own comments cite a Missal heading in one of two shapes: + + 1. A LEADING comment block, then a group of entries it covers, e.g. + "; Ash Wednesday and the three days after it -- LT.txt:8686-8689." + followed by four `key = value` lines. The pool for every citation + found in such a comment is EVERY entry in the group (there is no + positional correspondence encoded in the data between a particular + cited line and a particular entry in the list). + 2. A TRAILING comment immediately under the ONE entry it explains, e.g. + "advent = Tempus Adventus" then "; LT.txt:8609." on the next line, + with no blank line -- [season]'s own style. The pool is that one entry + alone. + +A "PATTERN" marker follows the identical leading/trailing rule (see round +2 fix 1 above): it excludes only the entry (or entries) it is itself +attached to from citation-checking, never the rest of the block. + +A citation is either a bare line number ("LT.txt:8609") or an explicit +range ("LT.txt:8609-8610", capped at `MAX_RANGE_WIDTH` lines) for a heading +that genuinely wraps across physical lines in the source; a comma-separated +list ("LT.txt:8618,8620,8622") is several independent citations, each +checked on its own -- unless the list itself looks like a thousands-typo +for one number (round 2 fix 4), in which case the whole spec is MALFORMED. +For a bare number the window is that one line; for a range it is the union +of every line in the range (inclusive). There is no other tolerance. + +For each citation, the POOL is the set of candidate Latin phrases it could +be defending: the entry (trailing shape) or every entry in the group +(leading shape) -- nothing pulled from quoted prose elsewhere in the +comment (see the self-poisoning account above). Each pool item keeps its +OWN distinctive-word set (>=4 letters, not on the small stopword list +below, j/i and ae/oe/diacritics normalised) -- items are never flattened +into one shared bag, for the same reason quotes were removed: a citation +bundling two claims onto one line number must not pass on the strength of +an unrelated pool item's words. + +A pool item is a candidate MATCH only if the cited window's text contains +ALL of its distinctive words -- otherwise it is not a match at all: it +never contributed to a PASS and is not what the citation is checked at +all. Among pool items that ARE contained in the window, the citation +PASSES if at least one clears `EVIDENCE_THRESHOLD` (see round 2 fix 3 +above and `item_evidence`'s own docstring) -- i.e. contains a word rare +enough, across the whole LT.txt corpus, to be real evidence rather than +coincidence. A citation whose only contained items are all common-word-only +is CANNOT VERIFY, not a silent PASS: found, but not proven. A citation with +NO contained item at all -- the claimed words are simply not at the cited +line(s) -- is WRONG. + +This is deliberately a LOOSE, word-overlap check within the (now exact and +bounded) window, not a byte-exact phrase match: la.ini spells abbreviations +out in full (Sanctissimi, not Ss.mi) and normalises j->i, and requiring a +byte-exact substring would either force every citation's prose to repeat +the raw OCR text verbatim (defeating the point of writing readable +comments) or produce false failures having nothing to do with a wrong line +number. What it proves is narrower than a byte-exact match, and is +disclosed as such: PASS means "the claimed name's distinctive words are +present, in full, at the exact line(s) cited, and at least one of them is +rare enough in the corpus to be real evidence" -- not that the citation is +the best possible line, only that it is not obviously wrong and is not +resting on a coincidence-prone common word. +""" +import argparse +import re +import sys +import unicodedata +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +DEFAULT_LA_INI = ROOT / "lang" / "la.ini" +DEFAULT_LT_TXT = ROOT / "docs" / "research" / "LT.txt" + +STOPWORDS = { + "in", "de", "et", "ad", "post", "ante", "cum", "per", "seu", "infra", + "vel", "si", "haec", "hoc", "hic", "qui", "quae", "quod", "quia", + "tempus", "dominica", "dominicam", "dominicae", "feria", "feriae", + "sabbato", "sabbatum", "die", "diebus", "eodem", "anno", "eius", + "sancti", "sancta", "sanctae", "sancto", "sanctorum", "sanctus", + "domini", "dominus", "octava", "octavam", "octavas", + "missae", "missa", "proprium", "gregorianus", "cantus", "pdf", + "forma", "longior", "brevior", "vide", "etiam", "dom", "prosper", + "sacro", "actio", "electronica", "formam", "novissimae", "variationes", + "copyright", "archivum", "liturgicum", "missale", "romanum", "index", + "www", "http", "https", "htm", "html", "com", "romanum", "text", +} + +# A citation's explicit range is capped at this many lines (n .. n+2). A +# genuine heading wrap in the source is one or two extra physical lines; +# anything wider is not a wrap, it is a search over a neighbourhood wide +# enough to coincidentally contain almost any short phrase -- exactly the +# blanket +-2 tolerance round 1 removed, reintroduced at a much larger +# radius by an unbounded range. See round 2 fix 2 in the module docstring. +MAX_RANGE_WIDTH = 3 + +# The evidence bar a pool item's RAREST matched word must clear to count +# as real proof (see `item_evidence` below). 1/200: a word occurring up to +# ~200 times across the whole ~118,000-token LT.txt corpus can still be +# the deciding, sole distinctive word of a short Missal calendar-table +# entry (measured: "omnium", the only survivor in "Omnium Sanctorum", +# occurs 139 times across the whole document, mostly in unrelated legal +# prose -- "of all" is common Latin furniture -- yet is genuinely the +# correct, sole citable word for that one heading). "classis" (507 +# occurrences), the round 1 false-pass this rule specifically targets, +# sits comfortably over 2.5x past this bar and stays CANNOT VERIFY. +# Round-2's own measurement of every affected real citation in +# lang/la.ini is in the branch report. +EVIDENCE_THRESHOLD = 1.0 / 200 + + +def normalize_word(w: str) -> str: + w = w.lower() + w = unicodedata.normalize("NFKD", w) + w = "".join(c for c in w if not unicodedata.combining(c)) + w = w.replace("æ", "ae").replace("œ", "oe") + w = re.sub(r"[^a-z]", "", w) + w = w.replace("j", "i") + return w + + +def distinctive_words(text: str) -> set: + out = set() + for tok in re.split(r"\s+", text): + w = normalize_word(tok) + if len(w) >= 4 and w not in STOPWORDS: + out.add(w) + return out + + +def build_frequency_table(lt_lines: list) -> dict: + """Count how many times each normalised word occurs anywhere in the + whole LT.txt corpus -- built ONCE per run (not per citation) and + consulted by `word_evidence`/`item_evidence` below. This is what lets + the checker tell a token that could only ever mean one heading (occurs + once in 26,000+ lines) apart from common liturgical furniture (occurs + hundreds of times) -- see round 2 fix 3 in the module docstring.""" + freq = {} + for line in lt_lines: + for tok in re.split(r"\s+", line): + w = normalize_word(tok) + if w: + freq[w] = freq.get(w, 0) + 1 + return freq + + +def word_evidence(word: str, freq: dict) -> float: + """A token seen n times contributes 1/n: a hapax (n=1) contributes 1.0 + -- about as conclusive as a word-overlap check can be -- and a word + occurring hundreds of times contributes next to nothing. A plain + reciprocal is chosen over a logarithmic/IDF scale deliberately: + measured against this corpus, log-scaling compresses "occurs once" and + "occurs 500 times" into a difference of a few units, too close to + cleanly separate "essentially conclusive" from "unverified" with a + single threshold; a reciprocal keeps them many orders of magnitude + apart, which is the actual claim this rule makes.""" + n = freq.get(word, 0) + return 1.0 / n if n > 0 else 0.0 + + +def item_evidence(words: set, freq: dict) -> float: + """A pool item's rarity evidence is the SINGLE RAREST word it matched + on -- not a sum over all its words. Summing would let several + merely-uncommon words add up to "enough" evidence between them, which + is exactly the shape round 2 fix 3 exists to close (a match consisting + only of common tokens must stay CANNOT VERIFY "regardless of how many + words sit beside it") and exactly the self-poisoning failure mode from + round 1 (quoted corrective prose tends to share several ordinary words + with its neighbour, never one rare one). One genuinely rare word is + real evidence; an accumulation of merely-uncommon words is not the + same thing and must not be treated as if it were.""" + if not words: + return 0.0 + return max(word_evidence(w, freq) for w in words) + + +class CitationRef: + """One citation token: 'label' is what the data actually wrote + ("8609" or "8609-8610"); 'lines' is the fully-expanded, sorted list of + line numbers the label names -- a single-element list for a bare + number, the whole inclusive range for an explicit wrap.""" + + __slots__ = ("label", "lines") + + def __init__(self, label, lines): + self.label = label + self.lines = lines + + +def _looks_like_thousands_typo(tokens) -> bool: + """'12,459' splits into ('12', '459') -- the classic shape of a human + thousands-separator typo for a single 4-5 digit LT.txt line number: + grouped from the right in chunks of 3, a 4-digit number groups as + 'N,NNN' and a 5-digit number as 'NN,NNN' (LT.txt never reaches 6 + digits, so a 3-digit leading group never arises from real grouping). + Checked against every real comma-list citation in lang/la.ini as of + round 2: none has a 1-2 digit token immediately followed by an + exactly-3-digit token, so this shape is unambiguous enough in practice + to reject outright as malformed rather than silently parsing as two + unrelated short citations (round 2 fix 4).""" + for a, b in zip(tokens, tokens[1:]): + if re.fullmatch(r"\d{1,2}", a) and re.fullmatch(r"\d{3}", b): + return True + return False + + +def parse_citation_spec(spec: str): + """'8618,8620,8622' -> three exact-line CitationRefs; '8609-8610' -> one + CitationRef spanning both lines (an explicit wrapped heading, capped at + MAX_RANGE_WIDTH lines). Returns (refs, malformed): `refs` is the list of + successfully-parsed CitationRef objects; `malformed` is a list of + {"token", "reason"} dicts for anything that did NOT parse into a + trustworthy citation -- a backwards range, a range wider than the cap, + the thousands-typo shape above, or an unparseable token. Round 1 + silently DROPPED all of these (no crash, but no report either); round 2 + surfaces every one instead, because a malformed citation naming a real + entry deserves a human's attention exactly as much as a wrong one does + (see round 2 fixes 2 and 4 in the module docstring).""" + raw_tokens = [t.strip() for t in spec.split(",")] + if _looks_like_thousands_typo(raw_tokens): + return [], [ + { + "token": spec, + "reason": ( + "looks like a thousands-separator typo for one number " + "(a 1-2 digit token immediately followed by a 3-digit " + "one) rather than a genuine list of citations -- " + "remove the comma, or split into real citations" + ), + } + ] + refs = [] + malformed = [] + for tok in raw_tokens: + if not tok: + continue + m = re.fullmatch(r"(\d{2,6})-(\d{2,6})", tok) + if m: + a, b = int(m.group(1)), int(m.group(2)) + width = b - a + 1 + if a > b: + malformed.append({"token": tok, "reason": f"backwards range (LT.txt:{tok})"}) + elif width > MAX_RANGE_WIDTH: + malformed.append( + { + "token": tok, + "reason": ( + f"range is {width} lines wide (max {MAX_RANGE_WIDTH}) -- " + "not a genuine heading wrap; find the real line or mark PATTERN honestly" + ), + } + ) + else: + refs.append(CitationRef(tok, list(range(a, b + 1)))) + continue + m = re.fullmatch(r"(\d{2,6})", tok) + if m: + refs.append(CitationRef(tok, [int(m.group(1))])) + continue + malformed.append({"token": tok, "reason": "unparseable citation token"}) + return refs, malformed + + +CITATION_RE = re.compile(r"LT\.txt:\s*((?:\d{2,6}(?:-\d{2,6})?)(?:\s*,\s*\d{2,6}(?:-\d{2,6})?)*)") + + +def parse_blocks(la_ini_text: str): + """Split la.ini into blocks on blank lines and [section] headers. Each + block is a list of (kind, content) where kind is 'comment' or 'entry', + content is the stripped comment text or (key, value).""" + blocks = [] + cur = [] + for raw in la_ini_text.split("\n"): + line = raw.rstrip("\n") + stripped = line.strip() + if stripped == "" or stripped.startswith("["): + if cur: + blocks.append(cur) + cur = [] + continue + if stripped.startswith(";"): + cur.append(("comment", stripped[1:].strip())) + elif "=" in stripped: + k, _, v = stripped.partition("=") + cur.append(("entry", (k.strip(), v.strip()))) + # anything else (shouldn't occur) is ignored + if cur: + blocks.append(cur) + return blocks + + +def window_text_for(lt_lines, lines): + lo, hi = lines[0], lines[-1] + lo_c, hi_c = max(1, lo), min(len(lt_lines), hi) + if lo_c > hi_c: + return "" + return " ".join(lt_lines[lo_c - 1 : hi_c]) + + +def _pool_entries_for(entries_before, entries): + """The leading/trailing pooling rule, shared identically by citation + checking AND PATTERN scoping (round 2 fix 1): a TRAILING comment (one + or more entries already seen in this block) pools against only the + most recent entry; a LEADING comment (no entry seen yet) pools against + every entry in the block. The same rule must decide both questions -- + a PATTERN marker and a citation attached to the same comment always + cover the same entries, by construction.""" + return [entries_before[-1]] if entries_before else entries + + +def _pattern_marked_entries(block, entries): + """Which entries in this block are excluded from citation-checking by + their OWN "PATTERN" marker -- never by a PATTERN marker attached to a + DIFFERENT entry in the same block. Returns a set of `id()` of the + entry (key, value) tuples (safe: each entry tuple is a single object, + shared by reference between `block` and `entries`, never copied). + + This is a first, standalone pass over the block, completed before any + citation is evaluated, so a citation's own PATTERN status never + depends on where in the block it happens to sit relative to its + entry's PATTERN comment.""" + marked = set() + entries_before = [] + for k, c in block: + if k == "comment": + if "PATTERN" in c: + for e in _pool_entries_for(entries_before, entries): + marked.add(id(e)) + else: + entries_before.append(c) + return marked + + +def check(la_ini_text: str, lt_lines: list): + """Returns a dict: checked (int), passed (int), findings (list -- + wrong citations), unverifiable (list -- no matched item clears the + evidence bar), malformed (list -- citation syntax itself could not be + trusted). findings, unverifiable and malformed are all things a human + must look at; only 'passed' citations required no human attention.""" + checked = 0 + passed = 0 + findings = [] + unverifiable = [] + malformed = [] + freq = build_frequency_table(lt_lines) + + for block in parse_blocks(la_ini_text): + entries = [c for k, c in block if k == "entry"] + if not entries: + continue + + pattern_entries = _pattern_marked_entries(block, entries) + + entries_before = [] + for k, c in block: + if k == "comment": + for m in CITATION_RE.finditer(c): + pool_entries = _pool_entries_for(entries_before, entries) + if all(id(e) in pattern_entries for e in pool_entries): + # Every entry this citation could be defending is + # itself PATTERN-marked: this "LT.txt:N" is not a + # provenance claim for any of them (e.g. citing the + # GRAMMAR another day's heading attests), so + # checking it would produce a meaningless failure. + continue + entry_desc = ", ".join(f"{k2}={v2}" for k2, v2 in pool_entries) + pool_items = [distinctive_words(v2) for _, v2 in pool_entries] + nonempty_items = [p for p in pool_items if p] + pool_words = sorted(set().union(*pool_items)) if pool_items else [] + + refs, bad_tokens = parse_citation_spec(m.group(1)) + for bad in bad_tokens: + checked += 1 + malformed.append( + { + "label": bad["token"], + "entries": entry_desc, + "reason": bad["reason"], + } + ) + for ref in refs: + checked += 1 + window_text = window_text_for(lt_lines, ref.lines) + window_words = distinctive_words(window_text) + + if not nonempty_items: + # Every pool item is fully stopwords -- there is + # nothing to check either way. Absence of + # evidence is not evidence of correctness. + unverifiable.append( + { + "label": ref.label, + "entries": entry_desc, + "reason": "no distinctive words in the entry text to check at all", + "pool_words": pool_words, + } + ) + continue + + contained_items = [p for p in nonempty_items if p <= window_words] + strong_items = [ + p for p in contained_items if item_evidence(p, freq) >= EVIDENCE_THRESHOLD + ] + if strong_items: + passed += 1 + elif contained_items: + best = max(item_evidence(p, freq) for p in contained_items) + unverifiable.append( + { + "label": ref.label, + "entries": entry_desc, + "reason": ( + f"matched, but every matched word is too common to trust " + f"(best evidence {best:.4f}, need >= {EVIDENCE_THRESHOLD:.4f})" + ), + "pool_words": pool_words, + } + ) + else: + n = ref.lines[0] + findings.append( + { + "label": ref.label, + "entries": entry_desc, + "actual": ( + lt_lines[n - 1].strip() + if 1 <= n <= len(lt_lines) + else "(out of range)" + ), + "window": window_text.strip()[:160], + } + ) + else: + entries_before.append(c) + return { + "checked": checked, + "passed": passed, + "findings": findings, + "unverifiable": unverifiable, + "malformed": malformed, + } + + +def build_arg_parser(): + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument( + "--file", + dest="la_ini", + type=Path, + default=DEFAULT_LA_INI, + help=f"the la.ini-shaped file to check (default: {DEFAULT_LA_INI})", + ) + p.add_argument( + "--lt-file", + dest="lt_txt", + type=Path, + default=DEFAULT_LT_TXT, + help=f"the LT.txt transcription to check against (default: {DEFAULT_LT_TXT})", + ) + return p + + +def main(argv=None): + args = build_arg_parser().parse_args(argv) + if not args.lt_txt.exists(): + print( + f"SKIPPED: {args.lt_txt} is absent (docs/ is gitignored -- " + "present locally only). Citations are NOT verified this run." + ) + return 0 + la_ini_text = args.la_ini.read_text(encoding="utf-8") + lt_lines = args.lt_txt.read_text(encoding="utf-8", errors="replace").split("\n") + result = check(la_ini_text, lt_lines) + checked = result["checked"] + findings = result["findings"] + unverifiable = result["unverifiable"] + malformed = result["malformed"] + + if not findings and not unverifiable and not malformed: + print(f"check-citations: {checked} LT.txt citations checked, 0 look wrong.") + return 0 + + print( + f"check-citations: {len(findings)} of {checked} citations look wrong, " + f"{len(malformed)} MALFORMED, {len(unverifiable)} CANNOT VERIFY (see below):\n" + ) + if findings: + print("WRONG:\n") + for f in findings: + print(f" LT.txt:{f['label']} cited for [{f['entries']}]") + print(f" actual: {f['actual']!r}") + print(f" window: {f['window']!r}\n") + if malformed: + print("MALFORMED (citation syntax itself is untrustworthy):\n") + for m in malformed: + print(f" LT.txt:{m['label']} cited for [{m['entries']}]") + print(f" {m['reason']}\n") + if unverifiable: + print("CANNOT VERIFY (a human must adjudicate these by hand):\n") + for u in unverifiable: + words = ", ".join(u["pool_words"]) if u["pool_words"] else "(none)" + print(f" LT.txt:{u['label']} cited for [{u['entries']}]") + print(f" {u['reason']}; pool words: {words}\n") + return 2 + + +if __name__ == "__main__": + sys.exit(main()) @@ -22,3 +22,22 @@ (executable (name bootstrap_lectionary) (libraries colitur_kernel rite_ef unix sexplib)) + +; check_citations.py's own self-test (test_check_citations.py). Python, not +; OCaml, so it cannot be a `(test ...)` stanza -- an alias rule invoking it +; directly is dune's own documented shape for a non-OCaml check. Wired into +; the `runtest` alias (what `dune test`/`make test`/`make check` all build) +; so this suite runs every time the rest of the project's tests do, not +; only when someone remembers to run it by hand -- the exact discipline +; missing when check_citations.py itself shipped with no tests and its own +; self-poisoning bug went uncaught. Depends on both .py files (the test +; imports check_citations as a plain module, found via its own directory, +; once dune copies both into the sandboxed build directory) and on nothing +; else -- the fixture is entirely synthetic, no docs/research/LT.txt or +; lang/la.ini involved, so this rule runs identically whether or not the +; gitignored research corpus is present locally. +(rule + (alias runtest) + (deps check_citations.py test_check_citations.py) + (action + (run python3 test_check_citations.py))) diff --git a/tools/test_check_citations.py b/tools/test_check_citations.py new file mode 100644 index 0000000..26482bd --- /dev/null +++ b/tools/test_check_citations.py @@ -0,0 +1,635 @@ +#!/usr/bin/env python3 +"""Self-test for check_citations.py. + +This is the regression suite the tool itself did not have when the +self-poisoning bug (see check_citations.py's own module docstring) shipped +undetected: the corrective comment documenting a past wrong citation quoted +the wrong historical value, and that quote sat in the same word pool the +checker verified against, so re-introducing the exact bug produced "0 look +wrong" instead of a failure. A verification tool with no tests of its own +is exactly how you get one that passes for the wrong reason -- this file is +the fix for that, not merely for the bug it happened to expose. + +Runs under `dune test` via tools/dune's own `(rule (alias runtest) ...)`, +not only as a standalone script or a `make` target, so it cannot rot +unnoticed. Also runnable directly: `python3 tools/test_check_citations.py`. + +Everything below is a SYNTHETIC fixture -- a tiny made-up "LT.txt" and a +tiny made-up la.ini-shaped fragment, entirely in memory. Nothing here reads +the real docs/research/LT.txt (gitignored, absent on a fresh clone) or the +real lang/la.ini, so this suite runs identically everywhere, always. + +REGRESSION vs CHARACTERISATION -- LABELLED HONESTLY, PER TEST (round 2 fix +5). A review found that of round 1's seven fixture-driven tests, only TWO +actually failed against the pre-round-1 script -- the rest passed on both +sides of round 1's fix and regression-tested nothing despite their names. +Every test below now carries one of two tags in its docstring, verified, +not asserted: + + REGRESSION -- shown to FAIL when run against a named prior version of + check_citations.py (the exact command used to check this is recorded + next to the tag). Losing the fix this test guards would turn it red + again. + + CHARACTERISATION -- passes against the named prior version too (usually + because the prior version has no equivalent behaviour or API surface at + all -- a KeyError/AttributeError/TypeError rather than a meaningful + same-shape failure). Still valuable (it pins what the CURRENT tool does, + and would catch a future regression from here on), but it is not + evidence that round 1 or round 2 fixed anything -- there is no "before" + for it to have failed against. + +Two prior versions are named throughout: + ROUND-0 = the script as it shipped before round 1's hardening + (git rev 22824ef, saved for this audit at + /tmp/check_citations_round1.py's OWN predecessor -- see the + round-2 branch report for the exact commands run). + ROUND-1 = the script as hardened by round 1, before round 2's five + fixes below (git HEAD at the start of this round). +Round-1-era tests (the original seven fixture cases plus the two added +after them) are checked against ROUND-0. Round-2 tests (fixes 1-4) are +checked against ROUND-1, since that is the version each one is proving a +regression against. +""" +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import check_citations as cc # noqa: E402 (path insert must come first) + +# --------------------------------------------------------------------------- +# Synthetic "LT.txt". 0-indexed list where element 0 IS line 1 (matching +# check_citations.py's own convention: lt_lines[n - 1] is line n). Real +# citations are always >= 2 digits (CITATION_RE requires \d{2,6}, matching +# realistic LT.txt line numbers, which run into the thousands) -- padded +# with filler so every cited line here is two digits too, the same +# constraint real data has. +# +# Line map (1-indexed): +# 10 Festum Aurorae Caelestis -- alpha's real heading +# 11 Prima Classis +# 13 Festum Umbrae Nocturnae -- beta's real heading +# 14 Secunda Classis -- beta wrongly cites here (off by 1) +# 16 Festum Solis Invicti -- gamma's real heading / epsilon's +# self-poisoning target +# 17 Tertia Classis -- iota wrongly cites here +# 18 Festum Gloriae -- delta's heading, part 1 (wraps) +# 19 Aeternae Perpetuae -- delta's heading, part 2; also +# gamma wrongly cites here (off by 3) +# 20 Rara Vox Singularis -- mu's real heading (rare word) +# 21 Communis Verbum Omnibus -- nu's real heading (common words) +# 24 (a single very long line, "communis"/"verbum" x250 each) -- +# frequency-table filler ONLY, never itself a citation target: this +# is what pushes communis/verbum's corpus-wide count past the +# evidence threshold, the same way real words like "classis" (507 +# occurrences) are common throughout LT.txt without living on any +# one line the checker is asked to verify against. +# 26 Magnum -- kappa/lambda heading, part 1 +# 27 Festum -- kappa/lambda heading, part 2 +# 28 Peregrinum -- kappa/lambda heading, part 3 +# (26-28 is a genuine 3-line wrap, at the MAX_RANGE_WIDTH cap; line +# 29 is unrelated filler included only by lambda's over-wide range) +_FREQUENCY_FILLER = " ".join(["communis", "verbum"] * 250) +LT_LINES = ( + ["(filler)"] * 9 + + [ + "Festum Aurorae Caelestis", # 10 + "Prima Classis", # 11 + "", # 12 + "Festum Umbrae Nocturnae", # 13 + "Secunda Classis", # 14 + "", # 15 + "Festum Solis Invicti", # 16 + "Tertia Classis", # 17 + "Festum Gloriae", # 18 + "Aeternae Perpetuae", # 19 + "Rara Vox Singularis", # 20 + "Communis Verbum Omnibus", # 21 + "(filler)", # 22 + "(filler)", # 23 + _FREQUENCY_FILLER, # 24 + "(filler)", # 25 + "Magnum", # 26 + "Festum", # 27 + "Peregrinum", # 28 + "Ultra", # 29 + ] +) + +# A fixture covering every case both hardening rounds asked for: +# alpha -- correct citation -> PASS +# beta -- off by one line -> FAIL +# gamma -- off by three lines -> FAIL +# delta -- explicit n-m range, heading genuinely wraps -> PASS +# epsilon -- THE SELF-POISONING CASE: comment quotes a +# DIFFERENT heading's text, citation points +# at that other heading's real line -> FAIL +# zeta -- entirely stopwords, nothing to check at all -> CANNOT VERIFY +# eta -- PATTERN, no citation at all -> skipped entirely +# theta -- PATTERN, trailing on itself only +# iota -- SAME BLOCK as theta, no PATTERN of its own, +# its own genuinely wrong citation -> FAIL +# (round 2 fix 1: theta's PATTERN must not silence this) +# mu -- single word, occurs ONCE in the whole corpus -> PASS +# (round 2 fix 3a) +# nu -- two words, both occur 251 times in the corpus -> CANNOT VERIFY +# (round 2 fix 3b -- stricter than round 1, which +# would have passed this on word-count alone) +# xi -- "12,459"-shaped citation, thousands-typo for +# one number -> MALFORMED +# (round 2 fix 4) +# kappa -- explicit 3-line range, AT the width cap -> PASS +# (round 2 fix 2a) +# lambda_ -- same heading, 4-line range, OVER the width cap -> MALFORMED +# (round 2 fix 2b; note the trailing underscore -- +# "lambda" is a Python keyword-adjacent builtin, avoided +# only to keep the la.ini key itself plain "lambda") +LA_INI_TEXT = """ +[test] + +alpha = Festum Aurorae Caelestis +; LT.txt:10. + +beta = Festum Umbrae Nocturnae +; LT.txt:14. + +gamma = Festum Solis Invicti +; LT.txt:19. + +delta = Festum Gloriae Aeternae +; LT.txt:18-19. + +epsilon = Festum Lunae Argenteae +; CORRECTED: an earlier draft wrongly attributed this to "Festum Solis +; Invicti" -- LT.txt:16. + +zeta = In Sancta Dominica +; LT.txt:11. + +; eta -- PATTERN, constructed name; no heading for this day survives in +; the source at all. +eta = Aliquid Fictum + +theta = Ignotum Simulatum +; PATTERN, invented for this self-test; no real heading survives for +; theta specifically. +iota = Festum Umbrae Nocturnae +; LT.txt:17. + +mu = Singularis +; LT.txt:20. + +nu = Communis Verbum +; LT.txt:21. + +xi = Numerus Fictus +; LT.txt:12,459. + +kappa = Magnum Festum Peregrinum +; LT.txt:26-28. + +lambda = Magnum Festum Peregrinum +; LT.txt:26-29. +""" + + +def entries_field(items, label, entries_substring=None): + """Find the finding/unverifiable/malformed dict whose citation label + matches (and, if given, whose entries description contains + `entries_substring` -- needed on the rare occasion two different + entries cite the identical wrong line number), or None.""" + for item in items: + if item["label"] != label: + continue + if entries_substring is not None and entries_substring not in item["entries"]: + continue + return item + return None + + +class TestCheckLogic(unittest.TestCase): + """Unit-level: exercises check() directly against the synthetic fixture.""" + + def setUp(self): + self.result = cc.check(LA_INI_TEXT, LT_LINES) + + def test_totals(self): + """CHARACTERISATION (pins the current tool's own output schema and + aggregate counts -- ROUND-1 has no 'malformed' key at all, so this + exact assertion cannot even be asked of it; it is not evidence of a + fix, it is a pin against future drift).""" + # 12 citation EVENTS: alpha, beta, gamma, delta(1 range), epsilon, + # zeta, iota, mu, nu, xi, kappa, lambda -- theta/eta contribute none. + self.assertEqual(self.result["checked"], 12) + self.assertEqual(self.result["passed"], 4) # alpha, delta, mu, kappa + self.assertEqual(len(self.result["findings"]), 4) # beta, gamma, epsilon, iota + self.assertEqual(len(self.result["unverifiable"]), 2) # zeta, nu + self.assertEqual(len(self.result["malformed"]), 2) # xi, lambda + + def test_correct_citation_passes(self): + """CHARACTERISATION: verified against ROUND-0 (git rev 22824ef) -- + alpha's own block has no quoted phrases, so the old quote-pooling + bug never touches it; alpha passes on both sides.""" + found_wrong = {f["label"] for f in self.result["findings"]} + found_unverifiable = {u["label"] for u in self.result["unverifiable"]} + self.assertNotIn("10", found_wrong) + self.assertNotIn("10", found_unverifiable) + + def test_off_by_one_line_fails(self): + """REGRESSION, verified against ROUND-0: ROUND-0's blanket +-2-line + tolerance means a window of LT.txt[12..16] is checked for citation + "14", which contains line 13 (beta's REAL heading) -- so ROUND-0 + reports beta as a PASS and this test's assertIsNotNone(...) fails + against it. Confirmed by direct replay of ROUND-0's check() against + this exact fixture shape (see the round-2 branch report).""" + f = entries_field(self.result["findings"], "14", "beta") + self.assertIsNotNone(f, "beta's off-by-one citation (LT.txt:14) must FAIL") + self.assertIn("beta", f["entries"]) + + def test_off_by_three_lines_fails(self): + """CHARACTERISATION, verified against ROUND-0: even ROUND-0's +-2 + tolerance window (LT.txt[17..21]) does not reach line 16 (gamma's + real heading), so ROUND-0 already reports this as wrong. This test + does not regression-test the +-2 removal; test_off_by_one above + does.""" + f = entries_field(self.result["findings"], "19", "gamma") + self.assertIsNotNone(f, "gamma's off-by-three citation (LT.txt:19) must FAIL") + self.assertIn("gamma", f["entries"]) + + def test_explicit_wrap_range_passes(self): + """CHARACTERISATION, verified against ROUND-0: explicit A-B ranges + already existed in ROUND-0's `expand_citation_spec` (identical + regex); ROUND-0 additionally pads each expanded line with its own + +-2 tolerance, so this passes there too, just for a sloppier + reason. test_wrap_range_required_not_just_first_line below is the + test that actually isolates the range syntax doing real work.""" + found_wrong = {f["label"] for f in self.result["findings"]} + found_unverifiable = {u["label"] for u in self.result["unverifiable"]} + found_malformed = {m["label"] for m in self.result["malformed"]} + self.assertNotIn("18-19", found_wrong) + self.assertNotIn("18-19", found_unverifiable) + self.assertNotIn("18-19", found_malformed) + + def test_wrap_range_required_not_just_first_line(self): + """REGRESSION, verified against ROUND-0: citing only delta's first + physical line ("LT.txt:18") still falls inside ROUND-0's own +-2 + window (16..20), which reaches line 19 and lets it pass -- ROUND-0 + never reports a finding here, so this test's assertIsNotNone(...) + fails against it. This is the concrete proof that the range syntax + is doing real work, not merely being tolerated by leftover slack.""" + text = LA_INI_TEXT.replace("; LT.txt:18-19.", "; LT.txt:18.") + result = cc.check(text, LT_LINES) + f = entries_field(result["findings"], "18") + self.assertIsNotNone( + f, "citing only the first physical line of a wrapped heading must FAIL" + ) + + def test_self_poisoning_quote_does_not_pass(self): + """REGRESSION, verified against ROUND-0: THE regression test for + the historical bug. epsilon's own comment quotes "Festum Solis + Invicti" (a DIFFERENT heading, gamma's own), and cites that other + heading's real line (LT.txt:16). ROUND-0 pools every double-quoted + phrase from the WHOLE block comment, so the quote itself becomes a + pool item, matches the window trivially, and ROUND-0 reports "0 + look wrong" for it -- confirmed by direct replay. Must FAIL here.""" + found_wrong = {f["label"]: f for f in self.result["findings"]} + self.assertIn("16", found_wrong, "the self-poisoning citation must be a FAIL, not a pass") + self.assertIn("epsilon", found_wrong["16"]["entries"]) + found_unverifiable = {u["label"] for u in self.result["unverifiable"]} + self.assertNotIn("16", found_unverifiable, "must be a real FAIL, not laundered into CANNOT VERIFY") + + def test_empty_pool_is_cannot_verify_not_pass(self): + """REGRESSION, verified against ROUND-0 (by the letter of the + definition -- see below for the nuance): zeta's entry text is + entirely stopwords ("In Sancta Dominica"), so there is nothing to + check either way. ROUND-0 has no CANNOT-VERIFY concept at all: a + fully empty pool item never satisfies `any(item <= window_words + for item in pool_items)` over an empty pool, so ROUND-0 reports it + as an ordinary WRONG finding instead -- confirmed by direct + replay. This test's specific assertion (that it lands in + `unverifiable`) therefore fails against ROUND-0, though the + underlying "not a silent pass" property does hold there too, just + through a coarser, undifferentiated classification. ROUND-1 + already has the current three-way split (as "too thin", word + count rather than "no distinctive words", rarity) and passes this + test unchanged.""" + u = entries_field(self.result["unverifiable"], "11") + self.assertIsNotNone(u, "zeta's all-stopword entry must be CANNOT VERIFY") + self.assertIn("zeta", u["entries"]) + found_wrong = {f["label"] for f in self.result["findings"]} + self.assertNotIn("11", found_wrong, "an empty pool must never be reported as a silent PASS") + + def test_pattern_block_skipped_entirely(self): + """CHARACTERISATION, verified against ROUND-0: eta's block contains + only eta itself, so ROUND-0's whole-block PATTERN skip and the + current tool's entry-scoped skip have the identical effect for + this single-entry case -- the difference only shows up in a + MULTI-entry block, which is test_pattern_does_not_silence_a_ + different_entry_in_the_same_block below (the real fix-1 regression + test).""" + + def keys_of(entries_desc): + return {pair.split("=", 1)[0] for pair in entries_desc.split(", ")} + + for f in self.result["findings"]: + self.assertNotIn("eta", keys_of(f["entries"])) + for u in self.result["unverifiable"]: + self.assertNotIn("eta", keys_of(u["entries"])) + for m in self.result["malformed"]: + self.assertNotIn("eta", keys_of(m["entries"])) + + def test_pattern_does_not_silence_a_different_entry_in_the_same_block(self): + """REGRESSION, verified against ROUND-1 (git HEAD before round 2): + theta and iota share ONE block (no blank line between them, the + same shape as [season]'s real back-to-back trailing-comment + style). theta's own trailing comment says "PATTERN"; iota is a + completely different entry with its own genuinely wrong citation + and no PATTERN marker at all. ROUND-1's `check()` tested + `"PATTERN" in block_comment` -- a single substring search over + every comment in the WHOLE block -- and skipped the entire block + on a match, so iota's wrong citation was never even looked at: + confirmed by direct replay of ROUND-1's check() against this exact + fixture (see the round-2 branch report). Must FAIL here.""" + f = entries_field(self.result["findings"], "17") + self.assertIsNotNone( + f, "iota's own wrong citation must FAIL even though theta, in the same block, is PATTERN-marked" + ) + self.assertIn("iota", f["entries"]) + # And theta itself must still be excluded, exactly like eta. + for f in self.result["findings"]: + self.assertNotIn("theta=", f["entries"]) + + def test_rare_single_word_match_passes(self): + """REGRESSION, verified against ROUND-1: mu's entry is a single + word, "Singularis", occurring exactly once in the whole corpus. + ROUND-1's word-COUNT gate (`MIN_DISTINCTIVE_WORDS = 2`) excluded + any one-word pool item from matching at all, regardless of how + rare that word is, and reported it CANNOT VERIFY unconditionally + -- confirmed by direct replay. This is round 2 fix 3's own primary + example (the real "S. Antonii Abb." shape): a single occurrence in + a 26,000+-line corpus is essentially conclusive and must PASS.""" + passed_labels_not_flagged = ( + "20" not in {f["label"] for f in self.result["findings"]} + and "20" not in {u["label"] for u in self.result["unverifiable"]} + ) + self.assertTrue(passed_labels_not_flagged, "a rare (freq=1) single-word match must PASS, not be flagged") + + def test_many_common_tokens_match_is_cannot_verify(self): + """REGRESSION, verified against ROUND-1: nu's entry has TWO + distinctive words ("Communis", "Verbum"), clearing ROUND-1's own + `MIN_DISTINCTIVE_WORDS = 2` gate on word count alone -- ROUND-1 + reports this a PASS purely because there are two words, without + ever checking how common either one is (both occur 251 times in + this fixture's corpus). Confirmed by direct replay. Round 2 fix 3 + requires this to stay CANNOT VERIFY regardless of word count -- + the STRICTER half of the rarity rule, not just the looser half + rare-word tests exercise.""" + u = entries_field(self.result["unverifiable"], "21") + self.assertIsNotNone( + u, "a match consisting only of common (251-occurrence) tokens must be CANNOT VERIFY" + ) + found_wrong = {f["label"] for f in self.result["findings"]} + self.assertNotIn("21", found_wrong, "common-word-only should be CANNOT VERIFY, not a silent FAIL either") + + def test_thousands_typo_citation_is_malformed(self): + """REGRESSION, verified against ROUND-1: ROUND-1's CITATION_RE and + `parse_citation_spec` happily parse "12,459" as two independent + bare citations, 12 and 459, and check them separately -- neither + anywhere near the real intended line, but the malformed spec is + never reported as such; confirmed by direct replay (ROUND-1 raises + no exception and produces two ordinary, uninteresting citation + events instead of one flagged one). Round 2 fix 4 requires the + whole spec to be rejected as MALFORMED instead.""" + m = entries_field(self.result["malformed"], "12,459") + self.assertIsNotNone(m, "the thousands-typo-shaped citation must be reported MALFORMED") + self.assertIn("xi", m["entries"]) + # And it must not ALSO sneak through as two ordinary citations. + self.assertIsNone(entries_field(self.result["findings"], "12")) + self.assertIsNone(entries_field(self.result["findings"], "459")) + + def test_range_at_cap_width_passes(self): + """REGRESSION, verified against ROUND-1: this specific 3-line range + already passes on ROUND-1 too (ROUND-1 also has no upper cap), so + by itself this is CHARACTERISATION -- it is paired here with + test_range_over_cap_width_is_malformed below to show the cap is + drawn in the RIGHT place (exactly at MAX_RANGE_WIDTH, not one line + short of it).""" + found_wrong = {f["label"] for f in self.result["findings"]} + found_malformed = {m["label"] for m in self.result["malformed"]} + self.assertNotIn("26-28", found_wrong) + self.assertNotIn("26-28", found_malformed) + + def test_range_over_cap_width_is_malformed(self): + """REGRESSION, verified against ROUND-1: "LT.txt:26-29" is a 4-line + range citing the identical heading text kappa already cites + correctly at the 3-line cap -- ROUND-1 has no upper bound at all + (only the pre-existing 200-line absurdity guard), so it silently + accepts this and checks it exactly like kappa's; confirmed by + direct replay. Round 2 fix 2 requires anything over + MAX_RANGE_WIDTH to be rejected as MALFORMED, regardless of whether + the content would otherwise have matched.""" + m = entries_field(self.result["malformed"], "26-29") + self.assertIsNotNone(m, "a range wider than MAX_RANGE_WIDTH must be MALFORMED") + self.assertIn("lambda", m["entries"]) + self.assertIn(str(cc.MAX_RANGE_WIDTH), m["reason"]) + + +class TestHelpers(unittest.TestCase): + def test_distinctive_words_strips_stopwords_and_short_tokens(self): + """CHARACTERISATION: `distinctive_words`/`normalize_word` are + byte-identical to ROUND-0 and ROUND-1 -- neither hardening round + touched them. Pins current behaviour only.""" + words = cc.distinctive_words("Dominica I Adventus") + self.assertEqual(words, {"adventus"}) # "Dominica" stopword, "I" too short + + def test_distinctive_words_normalises_j_and_ligatures(self): + """CHARACTERISATION: see above.""" + self.assertEqual(cc.distinctive_words("Jesu"), cc.distinctive_words("Iesu")) + self.assertEqual(cc.distinctive_words("praesulaeque"), cc.distinctive_words("praesulæque")) + + def test_parse_citation_spec_bare_number(self): + """CHARACTERISATION: ROUND-0/ROUND-1 both expand a bare number the + same way, just under a different function name/return shape + (`expand_citation_spec` -> a flat list of ints, no malformed + channel). Pins the current tuple-returning API.""" + refs, malformed = cc.parse_citation_spec("8609") + self.assertEqual(len(refs), 1) + self.assertEqual(refs[0].lines, [8609]) + self.assertEqual(malformed, []) + + def test_parse_citation_spec_range_is_one_ref(self): + """CHARACTERISATION: see above.""" + refs, malformed = cc.parse_citation_spec("8609-8610") + self.assertEqual(len(refs), 1) + self.assertEqual(refs[0].lines, [8609, 8610]) + self.assertEqual(malformed, []) + + def test_parse_citation_spec_comma_list_is_several_refs(self): + """CHARACTERISATION: see above.""" + refs, malformed = cc.parse_citation_spec("8618,8620,8622") + self.assertEqual([r.lines for r in refs], [[8618], [8620], [8622]]) + self.assertEqual(malformed, []) + + def test_parse_citation_spec_mixed_list(self): + """CHARACTERISATION: see above.""" + refs, malformed = cc.parse_citation_spec("8786,8788-8789,8791-8792") + self.assertEqual( + [r.lines for r in refs], + [[8786], [8788, 8789], [8791, 8792]], + ) + self.assertEqual(malformed, []) + + def test_parse_citation_spec_rejects_backwards_range(self): + """REGRESSION, verified against ROUND-1: ROUND-1's + `parse_citation_spec` SILENTLY DROPPED a backwards range (empty + refs list, no report at all) -- confirmed by direct replay. Round + 2 surfaces it as MALFORMED instead of discarding it invisibly.""" + refs, malformed = cc.parse_citation_spec("100-50") + self.assertEqual(refs, []) + self.assertEqual(len(malformed), 1) + self.assertIn("backwards", malformed[0]["reason"]) + + def test_parse_citation_spec_rejects_range_over_cap(self): + """REGRESSION, verified against ROUND-1: ROUND-1 accepted any + range up to 200 lines wide as a normal, silently-checked citation + -- "1000-999999" exceeded even that old 200-line guard and was + silently dropped (empty list, no report); a merely-wide-but-under + -200 range like "1000-1100" was silently ACCEPTED and checked as + if it were a legitimate wrap, which is the actual defeat this fix + closes. Both shapes are confirmed by direct replay against + ROUND-1. Round 2 caps at MAX_RANGE_WIDTH and reports the excess + width by name rather than silently accepting or silently + dropping.""" + refs, malformed = cc.parse_citation_spec("1000-1100") + self.assertEqual(refs, [], "a 101-line range must not be silently accepted") + self.assertEqual(len(malformed), 1) + self.assertIn("101", malformed[0]["reason"]) + + def test_parse_citation_spec_range_at_cap_boundary(self): + """CHARACTERISATION: pins the exact boundary -- a range exactly + MAX_RANGE_WIDTH lines wide is accepted, not rejected.""" + refs, malformed = cc.parse_citation_spec("2000-2002") + self.assertEqual(len(refs), 1) + self.assertEqual(malformed, []) + + def test_parse_citation_spec_thousands_typo(self): + """REGRESSION, verified against ROUND-1: ROUND-1 parses "12,459" + as two ordinary bare citations (12 and 459) with no indication + anything is wrong -- confirmed by direct replay. Round 2 fix 4 + rejects the whole spec instead.""" + refs, malformed = cc.parse_citation_spec("12,459") + self.assertEqual(refs, []) + self.assertEqual(len(malformed), 1) + self.assertIn("thousands", malformed[0]["reason"]) + + def test_parse_citation_spec_similar_looking_list_is_not_flagged(self): + """CHARACTERISATION: guards the thousands-typo heuristic against + false positives on a genuine multi-citation list -- two 4-digit + numbers close together must still parse normally, not be rejected + just because they happen to sit next to each other in a list.""" + refs, malformed = cc.parse_citation_spec("8618,8620,8622") + self.assertEqual(malformed, []) + self.assertEqual(len(refs), 3) + + def test_item_evidence_hapax_is_strong(self): + """CHARACTERISATION of the new (round 2) rarity machinery: a word + occurring once in a corpus of many contributes evidence 1.0, well + past EVIDENCE_THRESHOLD. No prior round had this function at all.""" + freq = cc.build_frequency_table(["Singularis Verbum", "Aliud Verbum"]) + self.assertEqual(cc.item_evidence({"singularis"}, freq), 1.0) + + def test_item_evidence_common_word_is_weak(self): + """CHARACTERISATION of the new rarity machinery: a word occurring + 500 times contributes far below EVIDENCE_THRESHOLD.""" + freq = cc.build_frequency_table([" ".join(["classis"] * 500)]) + self.assertLess(cc.item_evidence({"classis"}, freq), cc.EVIDENCE_THRESHOLD) + + def test_item_evidence_is_the_max_not_the_sum(self): + """CHARACTERISATION: an item combining one rare word and one very + common word takes its evidence from the RARE one -- a real match + is not penalised for also containing an ordinary word beside it.""" + freq = cc.build_frequency_table( + ["Singularis Verbum"] + [" ".join(["communis"] * 300)] + ) + ev = cc.item_evidence({"singularis", "communis"}, freq) + self.assertEqual(ev, 1.0) + + def test_word_evidence_unseen_word_is_zero(self): + """CHARACTERISATION: a word absent from the corpus entirely (freq + 0) contributes zero evidence rather than raising or dividing by + zero -- it can never legitimately be "contained" in a real window + either, so this only matters defensively.""" + freq = cc.build_frequency_table(["Aliud Verbum"]) + self.assertEqual(cc.word_evidence("nusquam", freq), 0.0) + + +class TestCliIntegration(unittest.TestCase): + """End-to-end: invokes the real main() as a subprocess, exactly how + `make check-citations` does, using --file/--lt-file to point at + temporary fixtures so the real lang/la.ini is never touched. + + Labelled per-test against ROUND-1 (the --file/--lt-file plumbing + itself is a ROUND-1 feature; ROUND-0's main() takes no arguments at + all and reads the real lang/la.ini and real docs/research/LT.txt + unconditionally, so ROUND-0 is not a meaningful comparison for any + test in this class).""" + + def run_cli(self, la_ini_text, lt_text, lt_present=True): + with tempfile.TemporaryDirectory() as td: + tdp = Path(td) + la_ini_path = tdp / "la.ini" + la_ini_path.write_text(la_ini_text, encoding="utf-8") + lt_path = tdp / "LT.txt" + if lt_present: + lt_path.write_text(lt_text, encoding="utf-8") + proc = subprocess.run( + [ + sys.executable, + str(Path(__file__).resolve().parent / "check_citations.py"), + "--file", + str(la_ini_path), + "--lt-file", + str(lt_path), + ], + capture_output=True, + text=True, + ) + return proc + + def test_skipped_when_lt_txt_absent(self): + """CHARACTERISATION, verified against ROUND-1: identical SKIPPED + behaviour, unchanged by round 2.""" + proc = self.run_cli(LA_INI_TEXT, "", lt_present=False) + self.assertEqual(proc.returncode, 0) + self.assertIn("SKIPPED", proc.stdout) + + def test_mixed_fixture_exits_nonzero_and_reports_all_three_classes(self): + """REGRESSION, verified against ROUND-1: run as a real subprocess + against this exact fixture, ROUND-1's CLI never prints "MALFORMED" + anywhere (it has no such concept) -- confirmed by direct replay + (`WRONG`/`CANNOT VERIFY` both appear, `MALFORMED` does not).""" + proc = self.run_cli(LA_INI_TEXT, "\n".join(LT_LINES)) + self.assertEqual(proc.returncode, 2) + self.assertIn("WRONG", proc.stdout) + self.assertIn("MALFORMED", proc.stdout) + self.assertIn("CANNOT VERIFY", proc.stdout) + + def test_all_clean_fixture_exits_zero(self): + """CHARACTERISATION, verified against ROUND-1: identical clean-exit + behaviour, unchanged by round 2.""" + clean_text = """ +[test] + +alpha = Festum Aurorae Caelestis +; LT.txt:10. +""" + proc = self.run_cli(clean_text, "\n".join(LT_LINES)) + self.assertEqual(proc.returncode, 0) + self.assertIn("0 look wrong", proc.stdout) + + +if __name__ == "__main__": + unittest.main() |