diff options
| author | Lukasz Kasprzak <lukas@labunix.xyz> | 2026-08-19 23:48:35 +0200 |
|---|---|---|
| committer | Lukasz Kasprzak <lukas@labunix.xyz> | 2026-08-19 23:48:35 +0200 |
| commit | ecadd969e1d96918820a1fdac0cf0d520d96ba06 (patch) | |
| tree | a06ed1160bfd28ae4578d9445242e48ac9ac3ea4 /tools/check_citations.py | |
| parent | 6762ce46af3cb12bc6ae37cda762c5d95add7903 (diff) | |
| parent | 329d07b49e397cb65ab52e7ca019b47313027136 (diff) | |
| download | colitur-ecadd969e1d96918820a1fdac0cf0d520d96ba06.tar.gz colitur-ecadd969e1d96918820a1fdac0cf0d520d96ba06.zip | |
feat: naming, localisation and the rebuilt printed output
colitur computed the calendar correctly and could not say what it had
computed. A printed ordo read ef-septuagesima-sunday-2 where a reader
expects Dominica in Sexagesima, and the wall calendar showed slugs in
every cell.
lib/naming a language table and a config file, both pure and total,
parsing the INI reader Overlay_ini already had
lang/ la.ini and en.ini -- 725 names, every one transcribed
from the 1962 Missal and citing the line it came from
templates the ordo rebuilt as an A5 booklet: one week per page, a
table of contents, framed days, a colour swatch; the wall
calendar now fills its sheet instead of a quarter of it
tools check_citations.py verifies all 400 citations resolve,
with 33 self-tests of its own
Names resolve through lang -> declared fallback -> the slug, so a partial
translation is usable from its first line and the fully degraded case is
the old output rather than a blank page.
colitur day and colitur readings are BYTE-IDENTICAL to before, verified
against main rather than asserted; --lang, --raw and the lang/config
subcommands are still to come, and man/colitur-templates.5 still
documents the pre-naming view, so writing a custom template needs the
source until that lands.
Diffstat (limited to 'tools/check_citations.py')
| -rwxr-xr-x | tools/check_citations.py | 642 |
1 files changed, 642 insertions, 0 deletions
diff --git a/tools/check_citations.py b/tools/check_citations.py new file mode 100755 index 0000000..688b00e --- /dev/null +++ b/tools/check_citations.py @@ -0,0 +1,642 @@ +#!/usr/bin/env python3 +"""check_citations.py -- verify every "LT.txt:<n>" citation in lang/la.ini +actually resolves to the Latin text it claims, in docs/research/LT.txt. + +Run via `make check-citations` (or directly: `python3 tools/check_citations.py +[--file LA_INI] [--lt-file LT_TXT]`). Exits 2 with a report if any citation +looks WRONG, is MALFORMED, or CANNOT BE VERIFIED; exits 0 (with a summary +line) only if every citation checked out cleanly; exits 0 with a loud +"SKIPPED" line if docs/research/LT.txt is not present (it is gitignored -- +see below). + +THIS SCRIPT WAS ITSELF FOUND TO BE SELF-POISONING (2026-08-19) AND HARDENED +------------------------------------------------------------------------ +An earlier version pooled distinctive words from the whole COMMENT BLOCK +around a citation, including any double-quoted phrase the comment happened +to mention -- and comments routinely quote a WRONG historical value while +explaining a past fix (e.g. "CORRECTED: previously cited LT.txt:12459, +which is 'Dominica ultima Octobris', not this heading"). That quote landed +in the pool, so re-introducing the exact bug being documented -- citing +LT.txt:12459 again -- matched the very quote correcting it, and the script +reported "0 look wrong". A verification tool whose own documentation of a +fix defeats the check for that fix is worse than no tool: it manufactures +false confidence. Proven with a reproduction: reintroducing that one wrong +citation into a real copy of lang/la.ini produced zero findings on the +pre-hardening script. Four changes closed this, in order of how directly +each one addresses the reproduction: + + 1. THE POOL IS SCOPED TO THE CITATION'S OWN ENTRY, never to the + surrounding comment's quoted text. A citation is verified against the + name it claims, not against anything quoted nearby -- see + `distinctive_words` / the per-citation `pool_items` construction + below. This alone closes the reproduced bug (see `test_check_citations.py`'s + own `test_self_poisoning_quote_does_not_pass`). + 2. A POOL TOO THIN TO VERIFY FAILS CLOSED (round 1's shape; round 2 + replaced the mechanism -- see below -- but kept the discipline: no + match is ever reported as a silent PASS just because nothing better + was checked). + 3. NO MORE BLANKET +-2-LINE TOLERANCE. A citation is checked at the EXACT + line it names. A heading that genuinely spans more than one physical + line in the source must say so explicitly, "LT.txt:8609-8610" -- the + allowance moves into the data, where a reader (and a future citation + added nearby) can see it, instead of silently forgiving ANY citation + within two lines of the truth. A bare "LT.txt:N" is checked at line N + only. + 4. THIS SCRIPT NOW HAS ITS OWN TEST SUITE (test_check_citations.py, + wired into `dune test` via tools/dune's own runtest rule) -- the + single most important change. The reproduced bug survived as long as + it did specifically BECAUSE nothing exercised this script's own + logic against a known-wrong citation. The self-poisoning case above + is now a permanent regression test. + +ROUND 2 (2026-08-19): A REVIEW DEFEATED ROUND 1 AGAIN -- FIVE MORE FIXES +------------------------------------------------------------------------ +Round 1's own hardening had three further holes, all found live against +this exact file, plus two smaller defects. Fixed in this order (matching +the order the defeats were found in, most dangerous first): + + 1. "; PATTERN" SILENCED A WHOLE BLOCK, NOT JUST ITS OWN ENTRY. `check()` + used to test `"PATTERN" in block_comment` -- one substring search over + every comment in the block -- and `continue` past the ENTIRE block if + it matched anywhere. In a block that mixes a PATTERN-marked entry with + an ordinary entry carrying its own, genuinely wrong citation (exactly + [season]'s own trailing-comment style: several entries back-to-back + with no blank line between them), the wrong citation was never even + looked at. Fixed by scoping PATTERN the same way citation pools are + already scoped: a PASS over the block first collects which entries a + PATTERN-bearing comment actually covers (the one preceding entry for a + trailing comment, every entry in the block for a leading one -- the + identical leading/trailing rule `check()` already uses for citation + pooling), and only THOSE entries are excluded from checking. See + `test_pattern_does_not_silence_a_different_entry_in_the_same_block`. + 2. EXPLICIT RANGES HAD NO UPPER BOUND. Round 1 replaced the old blanket + +-2-line tolerance with an explicit per-citation range + ("LT.txt:8609-8610") specifically so a wrap allowance would be + visible in the data instead of invisible in the checker -- but set no + cap on how WIDE that range could be. "LT.txt:8600-8650" passed if the + claimed text appeared anywhere across fifty lines: the same + "tolerance forgives a wrong line" defect the +-2 removal was meant to + close, reintroduced at 25x the radius, now with the data's own + apparent blessing. Fixed with `MAX_RANGE_WIDTH` (see below): a range + wider than a genuine heading wrap (1-2 extra physical lines) is + rejected as MALFORMED, naming the entry and the width, rather than + silently accepted. See `test_range_wider_than_cap_is_malformed`. + 3. THE "TOO THIN TO VERIFY" GATE COUNTED WORDS, NOT RARITY. Round 1 + required a pool item to have >=2 distinctive words before it could + even be checked, on the theory that a one-word match proves nothing. + Measured against the real data: this flagged 11 CORRECT citations + CANNOT VERIFY -- every one a short Latin hagionym whose Missal + heading genuinely has only one non-stopword ("S. Antonii Abb.": only + "antonii" survives the stopword filter) -- while a match on nothing + but "classis" (507 occurrences across LT.txt) passed the >=2 bar + freely whenever it shared a citation with three siblings ("I classis + / II classis / III classis / IV classis"). Word COUNT was never the + right proxy; word RARITY is. Replaced with `build_frequency_table` + + `item_evidence`: a token seen n times in the whole corpus contributes + 1/n of evidence, an item's evidence is its single RAREST matched + token (not a sum -- see `item_evidence`'s own docstring for why + summing would reopen this exact class of bug), and `EVIDENCE_THRESHOLD` + is the bar a match must clear to count as real proof rather than + coincidence. See `test_rare_single_word_match_passes` and + `test_common_word_only_match_is_cannot_verify`. + 4. A COMMA TYPO SILENTLY BECAME A DIFFERENT CITATION. "LT.txt:12,459" (a + stray thousands-separator comma for the single number "12459") used + to parse as TWO independent bare citations, "12" and "459" -- either + of which might coincidentally match somewhere nearby while the + intended line was never checked at all. `parse_citation_spec` now + recognises the shape (a 1-2 digit token immediately followed by an + exactly-3-digit token -- the only way a thousands-grouped LT.txt line + number, which never exceeds 5 digits, can be split by one comma) and + rejects the whole spec as MALFORMED rather than silently reinterpreting + it. See `test_thousands_separator_typo_is_malformed`. + 5. THE SELF-TEST SUITE OVERSTATED ITS OWN COVERAGE. A review ran round + 1's seven fixture cases against the PRE-round-1 script: only two + (`test_off_by_one_line_fails`, `test_self_poisoning_quote_does_not_pass`) + actually failed on it -- the rest passed on both sides of round 1's + fix and regression-tested nothing despite their names (a third, + `test_wrap_range_required_not_just_first_line`, was added after the + "seven" and also turns out to be a genuine regression test, confirmed + the same way). Every test in this suite is now labelled REGRESSION + (shown, not just claimed, to fail against a named prior version) or + CHARACTERISATION (pins current behaviour; does not fail on the prior + version, usually because it exercises a data shape or API surface + that prior version did not have at all) -- see + `test_check_citations.py`'s own module docstring for the full, + per-test account and how each label was actually verified. + +MALFORMED, PRECISELY (round 2's third finding class, alongside WRONG and +CANNOT VERIFY): a citation whose SYNTAX cannot be trusted even before its +content is checked -- a range wider than `MAX_RANGE_WIDTH`, a backwards +range, or the thousands-separator-typo shape above. Reported, counted, and +fails the run exactly like a WRONG citation: rejecting the syntax rather +than guessing at the author's intent is the whole point (see fix 2 and +fix 4 above) -- "the author either finds the real line or marks it PATTERN +honestly" is not achieved by the checker silently picking a reading. + +WHAT THIS CHECKS, PRECISELY (a heuristic, not a proof) +------------------------------------------------------- +lang/la.ini's own comments cite a Missal heading in one of two shapes: + + 1. A LEADING comment block, then a group of entries it covers, e.g. + "; Ash Wednesday and the three days after it -- LT.txt:8686-8689." + followed by four `key = value` lines. The pool for every citation + found in such a comment is EVERY entry in the group (there is no + positional correspondence encoded in the data between a particular + cited line and a particular entry in the list). + 2. A TRAILING comment immediately under the ONE entry it explains, e.g. + "advent = Tempus Adventus" then "; LT.txt:8609." on the next line, + with no blank line -- [season]'s own style. The pool is that one entry + alone. + +A "PATTERN" marker follows the identical leading/trailing rule (see round +2 fix 1 above): it excludes only the entry (or entries) it is itself +attached to from citation-checking, never the rest of the block. + +A citation is either a bare line number ("LT.txt:8609") or an explicit +range ("LT.txt:8609-8610", capped at `MAX_RANGE_WIDTH` lines) for a heading +that genuinely wraps across physical lines in the source; a comma-separated +list ("LT.txt:8618,8620,8622") is several independent citations, each +checked on its own -- unless the list itself looks like a thousands-typo +for one number (round 2 fix 4), in which case the whole spec is MALFORMED. +For a bare number the window is that one line; for a range it is the union +of every line in the range (inclusive). There is no other tolerance. + +For each citation, the POOL is the set of candidate Latin phrases it could +be defending: the entry (trailing shape) or every entry in the group +(leading shape) -- nothing pulled from quoted prose elsewhere in the +comment (see the self-poisoning account above). Each pool item keeps its +OWN distinctive-word set (>=4 letters, not on the small stopword list +below, j/i and ae/oe/diacritics normalised) -- items are never flattened +into one shared bag, for the same reason quotes were removed: a citation +bundling two claims onto one line number must not pass on the strength of +an unrelated pool item's words. + +A pool item is a candidate MATCH only if the cited window's text contains +ALL of its distinctive words -- otherwise it is not a match at all: it +never contributed to a PASS and is not what the citation is checked at +all. Among pool items that ARE contained in the window, the citation +PASSES if at least one clears `EVIDENCE_THRESHOLD` (see round 2 fix 3 +above and `item_evidence`'s own docstring) -- i.e. contains a word rare +enough, across the whole LT.txt corpus, to be real evidence rather than +coincidence. A citation whose only contained items are all common-word-only +is CANNOT VERIFY, not a silent PASS: found, but not proven. A citation with +NO contained item at all -- the claimed words are simply not at the cited +line(s) -- is WRONG. + +This is deliberately a LOOSE, word-overlap check within the (now exact and +bounded) window, not a byte-exact phrase match: la.ini spells abbreviations +out in full (Sanctissimi, not Ss.mi) and normalises j->i, and requiring a +byte-exact substring would either force every citation's prose to repeat +the raw OCR text verbatim (defeating the point of writing readable +comments) or produce false failures having nothing to do with a wrong line +number. What it proves is narrower than a byte-exact match, and is +disclosed as such: PASS means "the claimed name's distinctive words are +present, in full, at the exact line(s) cited, and at least one of them is +rare enough in the corpus to be real evidence" -- not that the citation is +the best possible line, only that it is not obviously wrong and is not +resting on a coincidence-prone common word. +""" +import argparse +import re +import sys +import unicodedata +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +DEFAULT_LA_INI = ROOT / "lang" / "la.ini" +DEFAULT_LT_TXT = ROOT / "docs" / "research" / "LT.txt" + +STOPWORDS = { + "in", "de", "et", "ad", "post", "ante", "cum", "per", "seu", "infra", + "vel", "si", "haec", "hoc", "hic", "qui", "quae", "quod", "quia", + "tempus", "dominica", "dominicam", "dominicae", "feria", "feriae", + "sabbato", "sabbatum", "die", "diebus", "eodem", "anno", "eius", + "sancti", "sancta", "sanctae", "sancto", "sanctorum", "sanctus", + "domini", "dominus", "octava", "octavam", "octavas", + "missae", "missa", "proprium", "gregorianus", "cantus", "pdf", + "forma", "longior", "brevior", "vide", "etiam", "dom", "prosper", + "sacro", "actio", "electronica", "formam", "novissimae", "variationes", + "copyright", "archivum", "liturgicum", "missale", "romanum", "index", + "www", "http", "https", "htm", "html", "com", "romanum", "text", +} + +# A citation's explicit range is capped at this many lines (n .. n+2). A +# genuine heading wrap in the source is one or two extra physical lines; +# anything wider is not a wrap, it is a search over a neighbourhood wide +# enough to coincidentally contain almost any short phrase -- exactly the +# blanket +-2 tolerance round 1 removed, reintroduced at a much larger +# radius by an unbounded range. See round 2 fix 2 in the module docstring. +MAX_RANGE_WIDTH = 3 + +# The evidence bar a pool item's RAREST matched word must clear to count +# as real proof (see `item_evidence` below). 1/200: a word occurring up to +# ~200 times across the whole ~118,000-token LT.txt corpus can still be +# the deciding, sole distinctive word of a short Missal calendar-table +# entry (measured: "omnium", the only survivor in "Omnium Sanctorum", +# occurs 139 times across the whole document, mostly in unrelated legal +# prose -- "of all" is common Latin furniture -- yet is genuinely the +# correct, sole citable word for that one heading). "classis" (507 +# occurrences), the round 1 false-pass this rule specifically targets, +# sits comfortably over 2.5x past this bar and stays CANNOT VERIFY. +# Round-2's own measurement of every affected real citation in +# lang/la.ini is in the branch report. +EVIDENCE_THRESHOLD = 1.0 / 200 + + +def normalize_word(w: str) -> str: + w = w.lower() + w = unicodedata.normalize("NFKD", w) + w = "".join(c for c in w if not unicodedata.combining(c)) + w = w.replace("æ", "ae").replace("œ", "oe") + w = re.sub(r"[^a-z]", "", w) + w = w.replace("j", "i") + return w + + +def distinctive_words(text: str) -> set: + out = set() + for tok in re.split(r"\s+", text): + w = normalize_word(tok) + if len(w) >= 4 and w not in STOPWORDS: + out.add(w) + return out + + +def build_frequency_table(lt_lines: list) -> dict: + """Count how many times each normalised word occurs anywhere in the + whole LT.txt corpus -- built ONCE per run (not per citation) and + consulted by `word_evidence`/`item_evidence` below. This is what lets + the checker tell a token that could only ever mean one heading (occurs + once in 26,000+ lines) apart from common liturgical furniture (occurs + hundreds of times) -- see round 2 fix 3 in the module docstring.""" + freq = {} + for line in lt_lines: + for tok in re.split(r"\s+", line): + w = normalize_word(tok) + if w: + freq[w] = freq.get(w, 0) + 1 + return freq + + +def word_evidence(word: str, freq: dict) -> float: + """A token seen n times contributes 1/n: a hapax (n=1) contributes 1.0 + -- about as conclusive as a word-overlap check can be -- and a word + occurring hundreds of times contributes next to nothing. A plain + reciprocal is chosen over a logarithmic/IDF scale deliberately: + measured against this corpus, log-scaling compresses "occurs once" and + "occurs 500 times" into a difference of a few units, too close to + cleanly separate "essentially conclusive" from "unverified" with a + single threshold; a reciprocal keeps them many orders of magnitude + apart, which is the actual claim this rule makes.""" + n = freq.get(word, 0) + return 1.0 / n if n > 0 else 0.0 + + +def item_evidence(words: set, freq: dict) -> float: + """A pool item's rarity evidence is the SINGLE RAREST word it matched + on -- not a sum over all its words. Summing would let several + merely-uncommon words add up to "enough" evidence between them, which + is exactly the shape round 2 fix 3 exists to close (a match consisting + only of common tokens must stay CANNOT VERIFY "regardless of how many + words sit beside it") and exactly the self-poisoning failure mode from + round 1 (quoted corrective prose tends to share several ordinary words + with its neighbour, never one rare one). One genuinely rare word is + real evidence; an accumulation of merely-uncommon words is not the + same thing and must not be treated as if it were.""" + if not words: + return 0.0 + return max(word_evidence(w, freq) for w in words) + + +class CitationRef: + """One citation token: 'label' is what the data actually wrote + ("8609" or "8609-8610"); 'lines' is the fully-expanded, sorted list of + line numbers the label names -- a single-element list for a bare + number, the whole inclusive range for an explicit wrap.""" + + __slots__ = ("label", "lines") + + def __init__(self, label, lines): + self.label = label + self.lines = lines + + +def _looks_like_thousands_typo(tokens) -> bool: + """'12,459' splits into ('12', '459') -- the classic shape of a human + thousands-separator typo for a single 4-5 digit LT.txt line number: + grouped from the right in chunks of 3, a 4-digit number groups as + 'N,NNN' and a 5-digit number as 'NN,NNN' (LT.txt never reaches 6 + digits, so a 3-digit leading group never arises from real grouping). + Checked against every real comma-list citation in lang/la.ini as of + round 2: none has a 1-2 digit token immediately followed by an + exactly-3-digit token, so this shape is unambiguous enough in practice + to reject outright as malformed rather than silently parsing as two + unrelated short citations (round 2 fix 4).""" + for a, b in zip(tokens, tokens[1:]): + if re.fullmatch(r"\d{1,2}", a) and re.fullmatch(r"\d{3}", b): + return True + return False + + +def parse_citation_spec(spec: str): + """'8618,8620,8622' -> three exact-line CitationRefs; '8609-8610' -> one + CitationRef spanning both lines (an explicit wrapped heading, capped at + MAX_RANGE_WIDTH lines). Returns (refs, malformed): `refs` is the list of + successfully-parsed CitationRef objects; `malformed` is a list of + {"token", "reason"} dicts for anything that did NOT parse into a + trustworthy citation -- a backwards range, a range wider than the cap, + the thousands-typo shape above, or an unparseable token. Round 1 + silently DROPPED all of these (no crash, but no report either); round 2 + surfaces every one instead, because a malformed citation naming a real + entry deserves a human's attention exactly as much as a wrong one does + (see round 2 fixes 2 and 4 in the module docstring).""" + raw_tokens = [t.strip() for t in spec.split(",")] + if _looks_like_thousands_typo(raw_tokens): + return [], [ + { + "token": spec, + "reason": ( + "looks like a thousands-separator typo for one number " + "(a 1-2 digit token immediately followed by a 3-digit " + "one) rather than a genuine list of citations -- " + "remove the comma, or split into real citations" + ), + } + ] + refs = [] + malformed = [] + for tok in raw_tokens: + if not tok: + continue + m = re.fullmatch(r"(\d{2,6})-(\d{2,6})", tok) + if m: + a, b = int(m.group(1)), int(m.group(2)) + width = b - a + 1 + if a > b: + malformed.append({"token": tok, "reason": f"backwards range (LT.txt:{tok})"}) + elif width > MAX_RANGE_WIDTH: + malformed.append( + { + "token": tok, + "reason": ( + f"range is {width} lines wide (max {MAX_RANGE_WIDTH}) -- " + "not a genuine heading wrap; find the real line or mark PATTERN honestly" + ), + } + ) + else: + refs.append(CitationRef(tok, list(range(a, b + 1)))) + continue + m = re.fullmatch(r"(\d{2,6})", tok) + if m: + refs.append(CitationRef(tok, [int(m.group(1))])) + continue + malformed.append({"token": tok, "reason": "unparseable citation token"}) + return refs, malformed + + +CITATION_RE = re.compile(r"LT\.txt:\s*((?:\d{2,6}(?:-\d{2,6})?)(?:\s*,\s*\d{2,6}(?:-\d{2,6})?)*)") + + +def parse_blocks(la_ini_text: str): + """Split la.ini into blocks on blank lines and [section] headers. Each + block is a list of (kind, content) where kind is 'comment' or 'entry', + content is the stripped comment text or (key, value).""" + blocks = [] + cur = [] + for raw in la_ini_text.split("\n"): + line = raw.rstrip("\n") + stripped = line.strip() + if stripped == "" or stripped.startswith("["): + if cur: + blocks.append(cur) + cur = [] + continue + if stripped.startswith(";"): + cur.append(("comment", stripped[1:].strip())) + elif "=" in stripped: + k, _, v = stripped.partition("=") + cur.append(("entry", (k.strip(), v.strip()))) + # anything else (shouldn't occur) is ignored + if cur: + blocks.append(cur) + return blocks + + +def window_text_for(lt_lines, lines): + lo, hi = lines[0], lines[-1] + lo_c, hi_c = max(1, lo), min(len(lt_lines), hi) + if lo_c > hi_c: + return "" + return " ".join(lt_lines[lo_c - 1 : hi_c]) + + +def _pool_entries_for(entries_before, entries): + """The leading/trailing pooling rule, shared identically by citation + checking AND PATTERN scoping (round 2 fix 1): a TRAILING comment (one + or more entries already seen in this block) pools against only the + most recent entry; a LEADING comment (no entry seen yet) pools against + every entry in the block. The same rule must decide both questions -- + a PATTERN marker and a citation attached to the same comment always + cover the same entries, by construction.""" + return [entries_before[-1]] if entries_before else entries + + +def _pattern_marked_entries(block, entries): + """Which entries in this block are excluded from citation-checking by + their OWN "PATTERN" marker -- never by a PATTERN marker attached to a + DIFFERENT entry in the same block. Returns a set of `id()` of the + entry (key, value) tuples (safe: each entry tuple is a single object, + shared by reference between `block` and `entries`, never copied). + + This is a first, standalone pass over the block, completed before any + citation is evaluated, so a citation's own PATTERN status never + depends on where in the block it happens to sit relative to its + entry's PATTERN comment.""" + marked = set() + entries_before = [] + for k, c in block: + if k == "comment": + if "PATTERN" in c: + for e in _pool_entries_for(entries_before, entries): + marked.add(id(e)) + else: + entries_before.append(c) + return marked + + +def check(la_ini_text: str, lt_lines: list): + """Returns a dict: checked (int), passed (int), findings (list -- + wrong citations), unverifiable (list -- no matched item clears the + evidence bar), malformed (list -- citation syntax itself could not be + trusted). findings, unverifiable and malformed are all things a human + must look at; only 'passed' citations required no human attention.""" + checked = 0 + passed = 0 + findings = [] + unverifiable = [] + malformed = [] + freq = build_frequency_table(lt_lines) + + for block in parse_blocks(la_ini_text): + entries = [c for k, c in block if k == "entry"] + if not entries: + continue + + pattern_entries = _pattern_marked_entries(block, entries) + + entries_before = [] + for k, c in block: + if k == "comment": + for m in CITATION_RE.finditer(c): + pool_entries = _pool_entries_for(entries_before, entries) + if all(id(e) in pattern_entries for e in pool_entries): + # Every entry this citation could be defending is + # itself PATTERN-marked: this "LT.txt:N" is not a + # provenance claim for any of them (e.g. citing the + # GRAMMAR another day's heading attests), so + # checking it would produce a meaningless failure. + continue + entry_desc = ", ".join(f"{k2}={v2}" for k2, v2 in pool_entries) + pool_items = [distinctive_words(v2) for _, v2 in pool_entries] + nonempty_items = [p for p in pool_items if p] + pool_words = sorted(set().union(*pool_items)) if pool_items else [] + + refs, bad_tokens = parse_citation_spec(m.group(1)) + for bad in bad_tokens: + checked += 1 + malformed.append( + { + "label": bad["token"], + "entries": entry_desc, + "reason": bad["reason"], + } + ) + for ref in refs: + checked += 1 + window_text = window_text_for(lt_lines, ref.lines) + window_words = distinctive_words(window_text) + + if not nonempty_items: + # Every pool item is fully stopwords -- there is + # nothing to check either way. Absence of + # evidence is not evidence of correctness. + unverifiable.append( + { + "label": ref.label, + "entries": entry_desc, + "reason": "no distinctive words in the entry text to check at all", + "pool_words": pool_words, + } + ) + continue + + contained_items = [p for p in nonempty_items if p <= window_words] + strong_items = [ + p for p in contained_items if item_evidence(p, freq) >= EVIDENCE_THRESHOLD + ] + if strong_items: + passed += 1 + elif contained_items: + best = max(item_evidence(p, freq) for p in contained_items) + unverifiable.append( + { + "label": ref.label, + "entries": entry_desc, + "reason": ( + f"matched, but every matched word is too common to trust " + f"(best evidence {best:.4f}, need >= {EVIDENCE_THRESHOLD:.4f})" + ), + "pool_words": pool_words, + } + ) + else: + n = ref.lines[0] + findings.append( + { + "label": ref.label, + "entries": entry_desc, + "actual": ( + lt_lines[n - 1].strip() + if 1 <= n <= len(lt_lines) + else "(out of range)" + ), + "window": window_text.strip()[:160], + } + ) + else: + entries_before.append(c) + return { + "checked": checked, + "passed": passed, + "findings": findings, + "unverifiable": unverifiable, + "malformed": malformed, + } + + +def build_arg_parser(): + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument( + "--file", + dest="la_ini", + type=Path, + default=DEFAULT_LA_INI, + help=f"the la.ini-shaped file to check (default: {DEFAULT_LA_INI})", + ) + p.add_argument( + "--lt-file", + dest="lt_txt", + type=Path, + default=DEFAULT_LT_TXT, + help=f"the LT.txt transcription to check against (default: {DEFAULT_LT_TXT})", + ) + return p + + +def main(argv=None): + args = build_arg_parser().parse_args(argv) + if not args.lt_txt.exists(): + print( + f"SKIPPED: {args.lt_txt} is absent (docs/ is gitignored -- " + "present locally only). Citations are NOT verified this run." + ) + return 0 + la_ini_text = args.la_ini.read_text(encoding="utf-8") + lt_lines = args.lt_txt.read_text(encoding="utf-8", errors="replace").split("\n") + result = check(la_ini_text, lt_lines) + checked = result["checked"] + findings = result["findings"] + unverifiable = result["unverifiable"] + malformed = result["malformed"] + + if not findings and not unverifiable and not malformed: + print(f"check-citations: {checked} LT.txt citations checked, 0 look wrong.") + return 0 + + print( + f"check-citations: {len(findings)} of {checked} citations look wrong, " + f"{len(malformed)} MALFORMED, {len(unverifiable)} CANNOT VERIFY (see below):\n" + ) + if findings: + print("WRONG:\n") + for f in findings: + print(f" LT.txt:{f['label']} cited for [{f['entries']}]") + print(f" actual: {f['actual']!r}") + print(f" window: {f['window']!r}\n") + if malformed: + print("MALFORMED (citation syntax itself is untrustworthy):\n") + for m in malformed: + print(f" LT.txt:{m['label']} cited for [{m['entries']}]") + print(f" {m['reason']}\n") + if unverifiable: + print("CANNOT VERIFY (a human must adjudicate these by hand):\n") + for u in unverifiable: + words = ", ".join(u["pool_words"]) if u["pool_words"] else "(none)" + print(f" LT.txt:{u['label']} cited for [{u['entries']}]") + print(f" {u['reason']}; pool words: {words}\n") + return 2 + + +if __name__ == "__main__": + sys.exit(main()) |
