bible(grb): restrict to the Vulgate canon; restore Hosea and Zechariah

grb failed corpus-check outright: 15 errors, and two whole books absent from
lectio at runtime with no error and no warning.

Upstream grb is a standalone Septuagint reader carrying 87 books. lectio needs
one uniform canon, so scripts/gen-grb-lectio.py derives the 73 books vul has
rather than forking -- upstream stays intact and keeps everything.

Eleven books have no Vulgate counterpart and are dropped, including 2 Esdras,
which upstream's README explains IS Ezra + Nehemiah under the Septuagint's
name; verified byte-identical, so nothing is lost. Five books ARE canonical
under other names and are remapped, and the verse counts show which witness
the Vulgate follows -- Jerome translated Theodotion, not the Old Greek:

    Bel and the Dragon (Theodotion)  42 = vul Daniel 14 (42)   exact
    Bel and the Dragon (LXX)         37                        no
    Sussana (Theodotion)             64 = vul Daniel 13 (65)
    Sussana (LXX)                    37                        no
    Letter of Jeremiah               73 = vul Baruch 6  (72)
    Wisdom of Solomon               435 = vul Wisdom   (439)

Upstream's plain "Daniel" is the Old Greek and is missing chapter 4 outright,
so Theodotion supplies Daniel throughout: complete, and the tradition the
lectionary actually cites. lectio thereby GAINS Wisdom, Daniel 4, Daniel 13-14
and Baruch 6 in Greek rather than losing them.

Three upstream defects are repaired in transit:

  * 408 rows carried 5 fields, not 6 -- a lost tab fused the book number and
    chapter ("281" = book 28, chapter 1). Every one was in Hosea or Zechariah,
    and neither book had a single well-formed row, so bible.go skipped both
    entirely. Hosea (197 verses) and Zechariah (211) are back.
  * 5 merged verse labels ("27-28") that strconv.Atoi turns into verse 0.
  * A UTF-8 BOM welded to the first book name, making "Genesis" a phantom
    74th book matching nothing.

The SBLGNT apparatus sigla are stripped, as TestGrbNoApparatusMarkers
requires; regenerating from raw upstream reintroduces ~8700 of them.

grb now passes. The remaining warnings are the Septuagint being itself --
Jeremiah is LXX-numbered (grb 33:2 is the Vulgate's 26:2), Esther integrates
its additions into chapters 1-10, LXX Malachi has three chapters, and 3
Kingdoms carries supplements like 10:22a that upstream stores as a duplicate
verse 22. Silencing those would mean deleting real Greek text.

1 files changed, 175 insertions, 0 deletions

diff --git a/scripts/gen-grb-lectio.py b/scripts/gen-grb-lectio.py
new file mode 100644
index 0000000..0ad27da
--- /dev/null
+++ b/scripts/gen-grb-lectio.py
@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+"""Derive lectio's Greek corpus from upstream grb (github.com/lukesmithxyz/grb).
+
+Upstream grb is a standalone reader and carries the whole Septuagint -- 87
+books, including manuscript variants and books with no Vulgate counterpart.
+lectio needs one uniform canon: the 73 books vul has, under vul's names. So
+this filters rather than forks, and upstream stays untouched.
+
+Five books ARE in the Vulgate canon but appear under other names, and the
+verse counts confirm which witness the Vulgate follows -- Jerome translated
+Theodotion, not the Old Greek:
+
+    Bel and the Dragon (Theodotion)  42 verses = vul Daniel 14   (42)  exact
+    Bel and the Dragon (LXX)         37 verses                          no
+    Sussana (Theodotion)             64 verses = vul Daniel 13   (65)
+    Sussana (LXX)                    37 verses                          no
+    Letter of Jeremiah               73 verses = vul Baruch 6    (72)
+    Wisdom of Solomon               435 verses = vul Wisdom     (439)
+
+Upstream's plain "Daniel" is the Old Greek AND is missing chapter 4 outright
+(it has 1,2,3,5..12), so Theodotion supplies Daniel throughout: complete, and
+the tradition the Vulgate and the lectionary actually use.
+
+Two upstream defects are repaired on the way through:
+
+  * 408 rows carry 5 fields, not 6: a lost tab fused the book number and the
+    chapter into one token ("281" = book 28, chapter 1). Every one is in Hosea
+    or Zechariah, and neither book has a single well-formed row -- bible.go
+    skips rows that are not exactly 6 fields, so both books were absent from
+    lectio at runtime with no error and no warning.
+  * The first row's book name carries a UTF-8 BOM, making "Genesis" a 74th
+    book that matches nothing.
+
+From the repo root:
+    python3 scripts/gen-grb-lectio.py            # report only
+    python3 scripts/gen-grb-lectio.py --write    # rewrite the corpus
+"""
+import collections
+import os
+import re
+import sys
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+UPSTREAM = os.environ.get("GRB_TSV", os.path.expanduser("~/git/grb/grb.tsv"))
+OUT = os.path.join(ROOT, "internal", "bible", "corpora_optional", "grb.tsv")
+SCHEMA = os.path.join(ROOT, "internal", "bible", "corpora_optional", "wuj.tsv")
+
+# fused book number + chapter; both books have 14 chapters
+FUSED = {"Hosea": 28, "Zechariah": 38}
+MAXCH = 14
+
+# book -> (canonical name, forced chapter or None)
+REMAP = {
+    "Daniel (Theodotion)": ("Daniel", None),
+    "Sussana (Theodotion)": ("Daniel", 13),
+    "Bel and the Dragon (Theodotion)": ("Daniel", 14),
+    "Letter of Jeremiah": ("Baruch", 6),
+    "Wisdom of Solomon": ("Wisdom", None),
+}
+
+DROP = {
+    "Daniel",                      # Old Greek, and missing chapter 4
+    "Sussana", "Bel and the Dragon",
+    "Judges (Vaticanus)", "Tobit (Sinaiticus)",
+    "1 Esdras", "2 Esdras", "3 Maccabees", "4 Maccabees",
+    "Odes", "Psalms of Solomon",
+}
+
+
+def schema():
+    """canonical (number, abbrev) per book name, from an existing corpus."""
+    out = {}
+    for line in open(SCHEMA, encoding="utf-8"):
+        f = line.rstrip("\n").split("\t")
+        if len(f) == 6 and f[0] not in out:
+            out[f[0]] = (f[2], f[1])
+    return out
+
+
+# SBLGNT critical-apparatus sigla, e.g. "⸀μαθητάς" or a "⸂...⸃" bracketed
+# variant. They are editorial marks on the text, not the text, and
+# TestGrbNoApparatusMarkers guards their absence -- regenerating from raw
+# upstream reintroduces roughly 8700 of them.
+APPARATUS = re.compile(r"[⸀-⸍]")
+
+
+def strip_apparatus(s):
+    return re.sub(r"\s{2,}", " ", APPARATUS.sub("", s)).strip()
+
+
+def read_upstream():
+    rows, repaired, ranges, dropped_bad = [], 0, 0, 0
+    for line in open(UPSTREAM, encoding="utf-8"):
+        f = line.rstrip("\n").split("\t")
+        if f:
+            f[0] = f[0].lstrip("")
+        if len(f) >= 5:
+            f[-1] = strip_apparatus(f[-1])
+
+        if len(f) == 5:
+            book, abbr, fused, verse, text = f
+            nr = FUSED.get(book)
+            if nr and fused.startswith(str(nr)):
+                ch = fused[len(str(nr)):]
+                if ch.isdigit() and 1 <= int(ch) <= MAXCH:
+                    rows.append([book, abbr, str(nr), ch, verse, text])
+                    repaired += 1
+                    continue
+            dropped_bad += 1
+            continue
+
+        if len(f) != 6:
+            dropped_bad += 1
+            continue
+
+        if not f[4].isdigit():                 # merged label like "27-28"
+            m = re.match(r"(\d+)", f[4])
+            if not m:
+                dropped_bad += 1
+                continue
+            f[4] = m.group(1)
+            ranges += 1
+        rows.append(f)
+    return rows, repaired, ranges, dropped_bad
+
+
+def main():
+    if not os.path.exists(UPSTREAM):
+        sys.exit("upstream grb.tsv not found at %s (set GRB_TSV)" % UPSTREAM)
+
+    rows, repaired, ranges, bad = read_upstream()
+    canon = schema()
+
+    kept, dropped = [], collections.Counter()
+    for r in rows:
+        book = r[0]
+        if book in DROP:
+            dropped[book] += 1
+            continue
+        if book in REMAP:
+            book, forced = REMAP[book]
+            r[0] = book
+            if forced is not None:
+                r[3] = str(forced)
+        if book not in canon:
+            dropped[book] += 1
+            continue
+        r[2], r[1] = canon[book][0], r[1]
+        kept.append(r)
+
+    kept.sort(key=lambda r: (int(r[2]), int(r[3]), int(r[4])))
+
+    books = {r[0] for r in kept}
+    print("upstream rows repaired (fused column) : %d" % repaired)
+    print("merged verse labels normalised        : %d" % ranges)
+    print("unusable rows discarded               : %d" % bad)
+    print("books dropped (outside vul canon)     : %d" % len(dropped))
+    for b, n in sorted(dropped.items(), key=lambda kv: -kv[1]):
+        print("    %-34s %5d verses" % (b, n))
+    print("books kept                            : %d" % len(books))
+    print("rows kept                             : %d" % len(kept))
+    missing = sorted(set(canon) - books)
+    if missing:
+        print("canonical books with no Greek text    : %s" % ", ".join(missing))
+
+    if "--write" not in sys.argv[1:]:
+        print("\nnot written; pass --write to rewrite %s" % OUT)
+        return
+    with open(OUT, "w", encoding="utf-8") as f:
+        for r in kept:
+            f.write("\t".join(r) + "\n")
+    print("\nwrote %s" % OUT)
+
+
+main()