fix(tools): harden check_citations.py against its own self-poisoning bug

Reproduced the defect: reintroducing the exact historical citation bug
(pointing class-1's citation back at LT.txt:12459, the value a prior
fix round corrected away from) made the tool report "147 citations
checked, 0 look wrong". The mechanism was that the corrective comment
documenting the old bug quotes the wrong historical value, and the
checker pooled every quoted phrase from the whole surrounding comment
block, so citing the wrong line matched the comment explaining why it
was wrong.

Four changes:

1. The word pool for a citation is now scoped to the entry(ies) it is
   attached to only -- never to quoted text elsewhere in the comment.
   This is the direct fix for the self-poisoning bug.
2. A citation whose pool has fewer than two distinctive words (Latin
   liturgical headings are short and stopword-heavy) cannot
   discriminate the right line from a wrong nearby one. Such a
   citation is now reported CANNOT VERIFY and fails the target,
   instead of silently passing.
3. The blanket +-2-line tolerance is gone. A bare "LT.txt:N" is
   checked at line N only; a heading that genuinely wraps must say so
   explicitly as "LT.txt:N-M". The allowance moves into the data,
   where it is visible.
4. The tool gets its own test suite, tools/test_check_citations.py,
   with a synthetic fixture covering: a correct citation, off-by-one
   and off-by-three mismatches, an explicit wrap range, a degenerate
   pool, a PATTERN-marked entry with no citation, and a dedicated
   regression test for the self-poisoning case itself. Wired into
   `dune test` via a new (rule (alias runtest) ...) in tools/dune (a
   plain (test ...) stanza cannot run a Python script), so it runs
   with the rest of the suite, not only as a `make` target.

Added a --file/--lt-file override to check_citations.py so the tool
(and its own tests) can point at a fixture without touching the real
lang/la.ini or docs/research/LT.txt. Confirmed the "SKIPPED, exit 0"
behaviour for a missing docs/research/LT.txt is unchanged.

tools/__pycache__/ (a stray artefact of this script, previously
untracked and ungitignored) is now in .gitignore.

Measured against the current lang/la.ini (another task is still
landing its sanctoral entries on this branch): 15 of 275 citations now
look wrong and 42 more cannot be verified, both far above the 0 the
unhardened tool reported. Not fixed here -- the data pass is separate,
once the sanctoral entries land.

1 files changed, 268 insertions, 0 deletions

diff --git a/tools/test_check_citations.py b/tools/test_check_citations.py
new file mode 100644
index 0000000..3d7722d
--- /dev/null
+++ b/tools/test_check_citations.py
@@ -0,0 +1,268 @@
+#!/usr/bin/env python3
+"""Self-test for check_citations.py.
+
+This is the regression suite the tool itself did not have when the
+self-poisoning bug (see check_citations.py's own module docstring) shipped
+undetected: the corrective comment documenting a past wrong citation quoted
+the wrong historical value, and that quote sat in the same word pool the
+checker verified against, so re-introducing the exact bug produced "0 look
+wrong" instead of a failure. A verification tool with no tests of its own
+is exactly how you get one that passes for the wrong reason -- this file is
+the fix for that, not merely for the bug it happened to expose.
+
+Runs under `dune test` via tools/dune's own `(rule (alias runtest) ...)`,
+not only as a standalone script or a `make` target, so it cannot rot
+unnoticed. Also runnable directly: `python3 tools/test_check_citations.py`.
+
+Everything below is a SYNTHETIC fixture -- a tiny made-up "LT.txt" and a
+tiny made-up la.ini-shaped fragment, entirely in memory. Nothing here reads
+the real docs/research/LT.txt (gitignored, absent on a fresh clone) or the
+real lang/la.ini, so this suite runs identically everywhere, always.
+"""
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import check_citations as cc  # noqa: E402  (path insert must come first)
+
+# ---------------------------------------------------------------------------
+# Synthetic "LT.txt". 0-indexed list where element 0 IS line 1 (matching
+# check_citations.py's own convention: lt_lines[n - 1] is line n). Real
+# citations are always >= 2 digits (CITATION_RE requires \d{2,6}, matching
+# realistic LT.txt line numbers, which run into the thousands) -- padded
+# with filler so every cited line here is two digits too, the same
+# constraint real data has.
+LT_LINES = ["(filler)"] * 9 + [
+    "Festum Aurorae Caelestis",  # line 10
+    "Prima Classis",             # line 11
+    "",                          # line 12
+    "Festum Umbrae Nocturnae",   # line 13
+    "Secunda Classis",           # line 14
+    "",                          # line 15
+    "Festum Solis Invicti",      # line 16
+    "Tertia Classis",            # line 17
+    "Festum Gloriae",            # line 18  (heading continues on line 19)
+    "Aeternae Perpetuae",        # line 19
+]
+
+# A fixture covering every case the hardening brief asked for:
+#   alpha   -- correct citation                          -> PASS
+#   beta    -- off by one line                            -> FAIL
+#   gamma   -- off by three lines                         -> FAIL
+#   delta   -- explicit n-m range, heading genuinely wraps -> PASS
+#   epsilon -- THE SELF-POISONING CASE: comment quotes a
+#              DIFFERENT heading's text, citation points
+#              at that other heading's real line           -> FAIL
+#   zeta    -- degenerate pool (a single-word entry)       -> CANNOT VERIFY
+#   eta     -- PATTERN, no citation at all                 -> skipped entirely
+LA_INI_TEXT = """
+[test]
+
+alpha = Festum Aurorae Caelestis
+; LT.txt:10.
+
+beta = Festum Umbrae Nocturnae
+; LT.txt:14.
+
+gamma = Festum Solis Invicti
+; LT.txt:19.
+
+delta = Festum Gloriae Aeternae
+; LT.txt:18-19.
+
+epsilon = Festum Lunae Argenteae
+; CORRECTED: an earlier draft wrongly attributed this to "Festum Solis
+; Invicti" -- LT.txt:16.
+
+zeta = Ordo
+; LT.txt:11.
+
+; eta -- PATTERN, constructed name; no heading for this day survives in
+; the source at all.
+eta = Aliquid Fictum
+"""
+
+
+def entries_field(items, label):
+    """Find the finding/unverifiable dict whose citation label matches, or
+    None. Small helper so assertions read by name, not by list position."""
+    for item in items:
+        if item["label"] == label:
+            return item
+    return None
+
+
+class TestCheckLogic(unittest.TestCase):
+    """Unit-level: exercises check() directly against the synthetic fixture."""
+
+    def setUp(self):
+        self.result = cc.check(LA_INI_TEXT, LT_LINES)
+
+    def test_totals(self):
+        # alpha, beta, gamma, delta, epsilon, zeta = 6 citation EVENTS.
+        # eta contributes nothing (PATTERN, and has no citation anyway).
+        self.assertEqual(self.result["checked"], 6)
+        self.assertEqual(self.result["passed"], 2)  # alpha, delta
+        self.assertEqual(len(self.result["findings"]), 3)  # beta, gamma, epsilon
+        self.assertEqual(len(self.result["unverifiable"]), 1)  # zeta
+
+    def test_correct_citation_passes(self):
+        passed_labels = {"10"}  # alpha's own label
+        found_wrong = {f["label"] for f in self.result["findings"]}
+        found_unverifiable = {u["label"] for u in self.result["unverifiable"]}
+        self.assertFalse(passed_labels & found_wrong)
+        self.assertFalse(passed_labels & found_unverifiable)
+
+    def test_off_by_one_line_fails(self):
+        f = entries_field(self.result["findings"], "14")
+        self.assertIsNotNone(f, "beta's off-by-one citation (LT.txt:14) must FAIL")
+        self.assertIn("beta", f["entries"])
+
+    def test_off_by_three_lines_fails(self):
+        f = entries_field(self.result["findings"], "19")
+        # NOTE: delta ALSO legitimately cites "18-19" as a range (a distinct
+        # citation event, checked separately) -- gamma's bad citation is
+        # the bare, single-number "19" token, which is what must fail here.
+        # A finding's label is the raw token as written, so "19" (gamma)
+        # and "18-19" (delta) never collide.
+        self.assertIsNotNone(f, "gamma's off-by-three citation (LT.txt:19) must FAIL")
+        self.assertIn("gamma", f["entries"])
+
+    def test_explicit_wrap_range_passes(self):
+        found_wrong = {f["label"] for f in self.result["findings"]}
+        found_unverifiable = {u["label"] for u in self.result["unverifiable"]}
+        self.assertNotIn("18-19", found_wrong)
+        self.assertNotIn("18-19", found_unverifiable)
+
+    def test_wrap_range_required_not_just_first_line(self):
+        # Without the explicit range, citing only delta's FIRST physical
+        # line must fail -- this is the concrete proof that the range
+        # syntax is doing real work, not merely being tolerated.
+        text = LA_INI_TEXT.replace("; LT.txt:18-19.", "; LT.txt:18.")
+        result = cc.check(text, LT_LINES)
+        f = entries_field(result["findings"], "18")
+        self.assertIsNotNone(
+            f, "citing only the first physical line of a wrapped heading must FAIL"
+        )
+
+    def test_self_poisoning_quote_does_not_pass(self):
+        """THE regression test for the historical bug: epsilon's own
+        comment quotes "Festum Solis Invicti" (a DIFFERENT heading,
+        gamma's own), and cites that other heading's real line (LT.txt:16).
+        A checker that pools quoted text from the surrounding comment would
+        pass this, exactly as the pre-hardening script did. It must FAIL."""
+        found_wrong = {f["label"]: f for f in self.result["findings"]}
+        self.assertIn("16", found_wrong, "the self-poisoning citation must be a FAIL, not a pass")
+        self.assertIn("epsilon", found_wrong["16"]["entries"])
+        found_unverifiable = {u["label"] for u in self.result["unverifiable"]}
+        self.assertNotIn("16", found_unverifiable, "must be a real FAIL, not laundered into CANNOT VERIFY")
+
+    def test_degenerate_pool_is_cannot_verify_not_pass(self):
+        u = entries_field(self.result["unverifiable"], "11")
+        self.assertIsNotNone(u, "zeta's single-word entry must be CANNOT VERIFY")
+        self.assertIn("zeta", u["entries"])
+        found_wrong = {f["label"] for f in self.result["findings"]}
+        self.assertNotIn("11", found_wrong, "a degenerate pool must never be reported as a silent PASS")
+
+    def test_pattern_block_skipped_entirely(self):
+        def keys_of(entries_desc):
+            return {pair.split("=", 1)[0] for pair in entries_desc.split(", ")}
+
+        for f in self.result["findings"]:
+            self.assertNotIn("eta", keys_of(f["entries"]))
+        for u in self.result["unverifiable"]:
+            self.assertNotIn("eta", keys_of(u["entries"]))
+
+
+class TestHelpers(unittest.TestCase):
+    def test_distinctive_words_strips_stopwords_and_short_tokens(self):
+        words = cc.distinctive_words("Dominica I Adventus")
+        self.assertEqual(words, {"adventus"})  # "Dominica" stopword, "I" too short
+
+    def test_distinctive_words_normalises_j_and_ligatures(self):
+        self.assertEqual(cc.distinctive_words("Jesu"), cc.distinctive_words("Iesu"))
+        self.assertEqual(cc.distinctive_words("praesulaeque"), cc.distinctive_words("praesulæque"))
+
+    def test_parse_citation_spec_bare_number(self):
+        refs = cc.parse_citation_spec("8609")
+        self.assertEqual(len(refs), 1)
+        self.assertEqual(refs[0].lines, [8609])
+
+    def test_parse_citation_spec_range_is_one_ref(self):
+        refs = cc.parse_citation_spec("8609-8610")
+        self.assertEqual(len(refs), 1)
+        self.assertEqual(refs[0].lines, [8609, 8610])
+
+    def test_parse_citation_spec_comma_list_is_several_refs(self):
+        refs = cc.parse_citation_spec("8618,8620,8622")
+        self.assertEqual([r.lines for r in refs], [[8618], [8620], [8622]])
+
+    def test_parse_citation_spec_mixed_list(self):
+        refs = cc.parse_citation_spec("8786,8788-8789,8791-8792")
+        self.assertEqual(
+            [r.lines for r in refs],
+            [[8786], [8788, 8789], [8791, 8792]],
+        )
+
+    def test_parse_citation_spec_rejects_backwards_range(self):
+        self.assertEqual(cc.parse_citation_spec("100-50"), [])
+
+    def test_parse_citation_spec_rejects_absurdly_wide_range(self):
+        self.assertEqual(cc.parse_citation_spec("1000-999999"), [])
+
+
+class TestCliIntegration(unittest.TestCase):
+    """End-to-end: invokes the real main() as a subprocess, exactly how
+    `make check-citations` does, using --file/--lt-file to point at
+    temporary fixtures so the real lang/la.ini is never touched."""
+
+    def run_cli(self, la_ini_text, lt_text, lt_present=True):
+        with tempfile.TemporaryDirectory() as td:
+            tdp = Path(td)
+            la_ini_path = tdp / "la.ini"
+            la_ini_path.write_text(la_ini_text, encoding="utf-8")
+            lt_path = tdp / "LT.txt"
+            if lt_present:
+                lt_path.write_text(lt_text, encoding="utf-8")
+            proc = subprocess.run(
+                [
+                    sys.executable,
+                    str(Path(__file__).resolve().parent / "check_citations.py"),
+                    "--file",
+                    str(la_ini_path),
+                    "--lt-file",
+                    str(lt_path),
+                ],
+                capture_output=True,
+                text=True,
+            )
+            return proc
+
+    def test_skipped_when_lt_txt_absent(self):
+        proc = self.run_cli(LA_INI_TEXT, "", lt_present=False)
+        self.assertEqual(proc.returncode, 0)
+        self.assertIn("SKIPPED", proc.stdout)
+
+    def test_mixed_fixture_exits_nonzero_and_reports_both_classes(self):
+        proc = self.run_cli(LA_INI_TEXT, "\n".join(LT_LINES))
+        self.assertEqual(proc.returncode, 2)
+        self.assertIn("WRONG", proc.stdout)
+        self.assertIn("CANNOT VERIFY", proc.stdout)
+
+    def test_all_clean_fixture_exits_zero(self):
+        clean_text = """
+[test]
+
+alpha = Festum Aurorae Caelestis
+; LT.txt:10.
+"""
+        proc = self.run_cli(clean_text, "\n".join(LT_LINES))
+        self.assertEqual(proc.returncode, 0)
+        self.assertIn("0 look wrong", proc.stdout)
+
+
+if __name__ == "__main__":
+    unittest.main()