path: root/tools/extract_litcal_ordo.py

#!/usr/bin/env python3
# tools/extract_litcal_ordo.py -- of-litcal-layer task (2026-08-25), external
# validation of the OF rite module's Phase 1 temporal cycle
# (lib/rites/rite_of/temporal_of.ml), specifically the two-block Ordinary
# Time week arithmetic named as the single highest-risk function in
# .superpowers/sdd/of-phase1-report.md's own "Concerns" section.
#
# Turns the ALREADY-ARCHIVED, SHA-pinned litcal JSON in the sibling project
# (~/git/projects/lectio/sources/litcal/calendar-YYYY-en.json, YYYY =
# 2024..2035, recorded in ~/git/projects/lectio/sources/manifest.tsv) into
# ONE sexp fixture, test/fixtures/litcal-temporal-2024-2035.sexp: one row
# per civil day, giving litcal's own season and (where witnessed) Ordinary
# Time week number.
#
# PYTHON, NOT OCAML -- same reasoning tools/extract_missalemeum_oracle.py
# and tools/extract_efdotorg_ordo.py already give: no JSON library in this
# project's frozen deps.
#
# Usage:
#   python3 tools/extract_litcal_ordo.py \
#     ~/git/projects/lectio/sources test/fixtures/litcal-temporal-2024-2035.sexp
#
# ---------------------------------------------------------------------------
# WHY 12 FILES, NOT 11. The task brief that started this layer said "2024
# through 2034" / "eleven available years". The manifest and the directory
# both actually carry TWELVE files, 2024 through 2035 inclusive -- checked
# directly (`grep litcal manifest.tsv | wc -l` = 12; `ls litcal/*.json | wc
# -l` = 12), every one SHA-256-verified against the manifest below before
# use. All twelve are used: they are already archived and pinned, "breadth
# of years is the point" per the task's own instruction, and there is no
# reason to discard a pinned, verified file to match a miscount in the
# brief. Recorded here, and in the fixture header, so a future reader does
# not go looking for an "eleventh year" that was never dropped.
#
# ---------------------------------------------------------------------------
# EACH FILE IS A LITURGICAL YEAR, NOT A CIVIL ONE. litcal's own
# `calendar/YYYY` endpoint returns the liturgical year that OPENS the
# Sunday nearest 30 November in civil year YYYY-1 and closes the Saturday
# before the next such Sunday in civil year YYYY -- e.g. calendar-2026
# covers 2025-11-30 (Advent I) through 2026-11-28 (the last Ordinary Time
# Saturday). Checked programmatically below (`_check_contiguous`): the
# twelve files' own non-vigil date ranges are EXACTLY contiguous, no gap,
# no overlap, covering 2023-12-03 through 2035-12-01 as one unbroken
# 4 382-day span -- confirmed, not assumed, before a single row is written.
#
# ---------------------------------------------------------------------------
# TWO ROW-SELECTION RULES, both found by direct inspection of the JSON
# (`is_vigil_mass` co-occurrence and same-date `liturgical_season`
# disagreement), not guessed:
#
# 1. VIGIL MASSES ARE EXCLUDED (`is_vigil_mass: true`). litcal lists an
#    anticipated Mass (Advent I, Christmas, Epiphany and so on) UNDER THE
#    CIVIL DATE IT ANTICIPATES, alongside that date's own real office --
#    e.g. 2025-11-29 (the Saturday before Advent I) carries BOTH
#    `OrdWeekday34Saturday` (season ORDINARY_TIME, litcal's own file for
#    civil year 2025) and, in the FOLLOWING file (calendar-2026),
#    `Advent1_vigil` (season ADVENT) for the identical civil date. Since
#    each file only ever carries ITS OWN non-vigil rows for a date (the
#    vigil is the file's own first entry, appended to an otherwise-absent
#    date), simply dropping every `is_vigil_mass: true` row removes this
#    source of cross-file/same-date season disagreement entirely -- no
#    file-merging logic was needed beyond that.
#
# 2. HOLY THURSDAY'S OWN CHRISM MASS (`HolyThursChrism`) IS ALSO EXCLUDED,
#    by name, as a single documented exception -- the ONLY same-date,
#    same-file pair found where two NON-vigil rows disagree on
#    `liturgical_season`: `HolyThursChrism` (LENT, the diocesan-cathedral
#    morning rite) and `HolyThurs` (EASTER_TRIDUUM, the evening Mass of
#    the Lord's Supper, the day's actual parish liturgy) share Holy
#    Thursday's own civil date in every one of the twelve files. Excluding
#    the Chrism row leaves `HolyThurs` as Holy Thursday's sole
#    representative, EASTER_TRIDUUM -- the same season Good Friday and
#    Holy Saturday already carry with no ambiguity at all (each has only
#    ONE non-vigil row, and it is always EASTER_TRIDUUM). This groups all
#    three Triduum days into ONE uniform, cleanly-citable divergence class
#    (colitur's `Vocab_of.season` has no Triduum value at all --
#    temporal_of.ml's own top-of-module comment already argues this from
#    the Normae's own prose, keeping the whole civil day of Holy Thursday/
#    Good Friday/Holy Saturday inside `Lent`) rather than an inconsistent
#    "2 of 3 Triduum days diverge" shape that would follow from picking
#    whichever Holy-Thursday row happens to agree with colitur.
#
# Checked exhaustively (not sampled) after applying both exclusions: zero
# remaining same-date `liturgical_season` disagreements anywhere in the
# 4 382-day span, and at most one event_key per date matches the Ordinary
# Time regex (`^Ord(Sunday|Weekday)\d+`) -- both asserted in
# `_check_contiguous`/`build_rows` below, not merely claimed here.
#
# ---------------------------------------------------------------------------
# WHY THE ORDINARY TIME WEEK IS SOMETIMES UNWITNESSED, two genuinely
# different shapes, both found by inspection, neither guessed:
#
#   (a) An OBLIGATORY celebration (a solemnity, feast, or obligatory
#       memorial) suppresses the weekday's own `OrdWeekday`/`OrdSunday`
#       row outright -- litcal prints ONLY the celebration, e.g.
#       25 May 2026 shows only `MaryMotherChurch` ("Memorial"), no
#       `OrdWeekday8Monday` at all. THIS is why the task's own hand-check
#       compares the first Ordinary Time WEEKDAY after Pentecost, not the
#       Monday itself: Mary, Mother of the Church (obligatory since 2018)
#       falls on that Monday in all three of the task's worked years
#       (2024-05-20, 2025-06-09, 2026-05-25), and is unwitnessed for the
#       same structural reason on every occurrence in this corpus.
#
#   (b) An OPTIONAL celebration does NOT suppress the weekday row --
#       litcal prints BOTH, e.g. 2025-06-14 carries both
#       `OrdWeekday10Saturday` and `SatMemBVM6` ("optional memorial"). So
#       an optional memorial's presence is never, by itself, a reason for
#       an unwitnessed week; only an obligatory one is.
#
#   A third, rarer shape (5 dates total, all in the Immaculate-Heart-of-
#   Mary window across the twelve years) shows TWO 'optional memorial'
#   rows and no weekday row at all -- litcal's own data omits the weekday
#   there for reasons this script does not adjudicate (plausibly a
#   version-5.7 data quirk, since the General Roman Calendar has treated
#   Immaculate Heart of Mary as OBLIGATORY since 2021, but litcal's own
#   `grade_lcl` still reads "optional memorial" here). Handled the same
#   as shape (a): week unwitnessed, [None], classified by the comparator,
#   never silently dropped.
#
# Lent's own PRIVILEGED ferias show a THIRD pattern that never produces an
# unwitnessed week and needed no special handling: a Lenten optional
# memorial is downgraded in print to a bare "commemoration" grade, but
# ALWAYS co-listed alongside its own weekday row (e.g. 2026-03-17 carries
# both `LentWeekday4Tuesday` and `StPatrick`, grade "commemoration") --
# structurally the same as shape (b) above, just Lent's own grade word for
# it. Ordinary Time week is irrelevant during Lent regardless, so this
# only matters for confirming SEASON is never ambiguous there either
# (checked, see above).
# ---------------------------------------------------------------------------

import glob
import hashlib
import json
import os
import re
import sys

ORD_RE = re.compile(r"^Ord(Sunday|Weekday)(\d+)")
EXCLUDED_EVENT_KEYS = {"HolyThursChrism"}


def sha256_of(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()


def load_manifest(sources_dir):
    manifest_path = os.path.join(sources_dir, "manifest.tsv")
    rows = {}
    with open(manifest_path) as f:
        header = f.readline()
        assert header.strip().split("\t") == ["path", "url", "sha256", "bytes"], header
        for line in f:
            path, url, sha, nbytes = line.rstrip("\n").split("\t")
            if path.startswith("litcal/"):
                rows[path] = (url, sha, int(nbytes))
    return rows


def verify_and_load(sources_dir, manifest_rows):
    """Verify every litcal/*.json on disk matches the manifest's SHA-256 and
    byte count, then return [(year, path, url, sha, data), ...] sorted by
    year. Fails loudly (non-zero exit) on any mismatch or missing file --
    this task's own brief says "verify those SHAs before using them"."""
    litcal_dir = os.path.join(sources_dir, "litcal")
    on_disk = sorted(glob.glob(os.path.join(litcal_dir, "calendar-*-en.json")))
    if not on_disk:
        print(f"ERROR: no calendar-*-en.json files found in {litcal_dir}", file=sys.stderr)
        sys.exit(1)
    out = []
    for path in on_disk:
        rel = "litcal/" + os.path.basename(path)
        if rel not in manifest_rows:
            print(f"ERROR: {rel} is on disk but not in manifest.tsv", file=sys.stderr)
            sys.exit(1)
        url, sha, nbytes = manifest_rows[rel]
        actual_sha = sha256_of(path)
        actual_bytes = os.path.getsize(path)
        if actual_sha != sha or actual_bytes != nbytes:
            print(
                f"ERROR: {rel} does not match manifest.tsv\n"
                f"  manifest: sha256={sha} bytes={nbytes}\n"
                f"  on disk:  sha256={actual_sha} bytes={actual_bytes}",
                file=sys.stderr,
            )
            sys.exit(1)
        m = re.match(r"calendar-(\d{4})-en\.json$", os.path.basename(path))
        year = int(m.group(1))
        with open(path) as f:
            data = json.load(f)
        out.append((year, rel, url, sha, data))
    out.sort(key=lambda t: t[0])
    # Confirm every manifest litcal/ row was also found on disk (catches a
    # partial archive silently under-covering the manifest's own claim).
    missing = set(manifest_rows) - {rel for _, rel, _, _, _ in out}
    if missing:
        print(f"ERROR: manifest.tsv names litcal files not found on disk: {sorted(missing)}", file=sys.stderr)
        sys.exit(1)
    return out


def easter_date_of(data):
    for r in data["litcal"]:
        if r["event_key"] == "Easter":
            return r["date"][:10]
    return None


def collect_rows(files):
    """[(year, path, url, sha, data), ...] -> dict date -> list of raw litcal
    rows, after dropping is_vigil_mass rows and EXCLUDED_EVENT_KEYS."""
    by_date = {}
    for _year, _rel, _url, _sha, data in files:
        for r in data["litcal"]:
            if r.get("is_vigil_mass"):
                continue
            if r["event_key"] in EXCLUDED_EVENT_KEYS:
                continue
            by_date.setdefault(r["date"][:10], []).append(r)
    return by_date


def check_contiguous(by_date):
    """Asserts the combined date set is one unbroken span with no gap, and
    that every date's rows agree on liturgical_season. Returns
    (first_date, last_date, n_dates)."""
    import datetime

    dates = sorted(by_date)
    first, last = dates[0], dates[-1]
    d0 = datetime.date.fromisoformat(first)
    for i, dt in enumerate(dates):
        expect = (d0 + datetime.timedelta(days=i)).isoformat()
        if dt != expect:
            print(f"ERROR: date span is not contiguous -- expected {expect}, got {dt} at index {i}", file=sys.stderr)
            sys.exit(1)
    for dt, rows in by_date.items():
        seasons = {r["liturgical_season"] for r in rows}
        if len(seasons) > 1:
            print(f"ERROR: {dt} has rows disagreeing on liturgical_season: {seasons}", file=sys.stderr)
            sys.exit(1)
    return first, last, len(dates)


def pick_representative(rows):
    """Which row's event_key/grade_lcl the fixture records for a date, for
    audit purposes ONLY -- comparison correctness never depends on this
    choice (season is already verified unique per date; the Ordinary Time
    week, if any, comes from the Ord* regex match directly, independent of
    which row is "representative"). Prefers the Ord* row; else the
    lowest-event_idx row that is not an 'optional memorial'; else the
    lowest-event_idx row of any kind."""
    for r in rows:
        if ORD_RE.match(r["event_key"]):
            return r
    non_optional = [r for r in rows if r.get("grade_lcl") != "optional memorial"]
    pool = non_optional if non_optional else rows
    return min(pool, key=lambda r: r["event_idx"])


def ordinary_time_week(rows):
    matches = [m for r in rows for m in [ORD_RE.match(r["event_key"])] if m]
    if not matches:
        return None
    if len(matches) > 1:
        dt = rows[0]["date"][:10]
        print(f"ERROR: {dt} has more than one Ord* event_key: {[r['event_key'] for r in rows]}", file=sys.stderr)
        sys.exit(1)
    return int(matches[0].group(2))


def build_rows(by_date):
    out = []
    for dt in sorted(by_date):
        rows = by_date[dt]
        season = rows[0]["liturgical_season"].lower()
        week = ordinary_time_week(rows)
        rep = pick_representative(rows)
        out.append((dt, season, week, rep["event_key"], rep.get("grade_lcl") or ""))
    return out


def sexp_string(s):
    out = []
    for ch in s:
        b = ch.encode("utf-8")
        if ch == '"':
            out.append('\\"')
        elif ch == "\\":
            out.append("\\\\")
        elif len(b) == 1 and 32 <= b[0] < 127:
            out.append(ch)
        else:
            for byte in b:
                out.append("\\%03d" % byte)
    return '"' + "".join(out) + '"'


def sexp_opt_int(n):
    return f"({n})" if n is not None else "()"


def write_fixture(out_path, rows, files, first, last):
    import datetime

    manifest_lines = []
    for year, rel, url, sha, data in files:
        easter = easter_date_of(data)
        manifest_lines.append(f";   {rel}  Easter {easter}  sha256={sha}")
    easter_dates = [easter_date_of(data) for _, _, _, _, data in files]
    easter_min, easter_max = min(easter_dates), max(easter_dates)
    easter_span_days = (datetime.date.fromisoformat(easter_max) - datetime.date.fromisoformat(easter_min)).days
    easter_list_str = ", ".join(f"{y}: {e}" for y, e in zip([f[0] for f in files], easter_dates))
    n_ot = sum(1 for _, s, w, _, _ in rows if s == "ordinary_time")
    n_ot_witnessed = sum(1 for _, s, w, _, _ in rows if s == "ordinary_time" and w is not None)
    n_triduum = sum(1 for _, s, _, _, _ in rows if s == "easter_triduum")

    header = f"""; test/fixtures/litcal-temporal-2024-2035.sexp -- of-litcal-layer task
; (2026-08-25): external validation fixture for the OF rite module's Phase 1
; temporal cycle (lib/rites/rite_of/temporal_of.ml), specifically the
; two-block Ordinary Time week-resumption arithmetic Normae n. 44 requires
; (see .superpowers/sdd/of-litcal-layer-report.md for the full account).
;
; SOURCE: litcal -- the Liturgical Calendar API, by Fr John Romano
; D'Orazio (github.com/Liturgical-Calendar/LiturgicalCalendarAPI),
; Apache-2.0 licensed, built for reuse. Data already archived and
; SHA-256-pinned by the sibling project lectio at
; ~/git/projects/lectio/sources/litcal/, recorded in that project's own
; sources/manifest.tsv -- verified against that manifest (not re-fetched)
; by tools/extract_litcal_ordo.py before this fixture was built:
;
{chr(10).join(manifest_lines)}
;
; TWELVE files, not eleven: the task brief that started this layer said
; "2024 through 2034" / "eleven available years", but both the manifest
; and the archive actually carry 2024 through 2035 (twelve files). All
; twelve are used -- see tools/extract_litcal_ordo.py's own header for the
; full account of why nothing pinned and verified was discarded to match
; the miscount.
;
; EASTER DATES ACTUALLY EXERCISED (the risk this layer validates is
; Easter-dependent, so this is the real measure of what these twelve years
; cover -- {easter_min} through {easter_max}, {len(set(easter_dates))} distinct
; dates across a {easter_span_days}-day span within the canonical 35-day
; Easter window (22 March - 25 April)):
;   {easter_list_str}
; NOT covered: the extremes of the window (no Easter in 22-24 March or
; 22-25 April among these twelve) -- a real gap, named rather than
; papered over; a future extension of the litcal archive to more years
; would close it.
;
; COVERAGE: {len(rows)} day-rows, {first}..{last} (one unbroken civil-date span --
; each file is a LITURGICAL year, opening near 30 November of the civil
; year before its own number, so the twelve files' own non-vigil date
; ranges are exactly contiguous; verified programmatically, not assumed).
; {n_ot} of those rows are litcal's own ORDINARY_TIME season; of those,
; {n_ot_witnessed} carry a witnessed Ordinary Time week number (an
; `OrdSundayN`/`OrdWeekdayN<Weekday>` event_key on that date) and
; {n_ot - n_ot_witnessed} do not (an obligatory celebration suppresses the
; weekday's own entry that day -- see tools/extract_litcal_ordo.py's own
; header for the two shapes this takes, both counted, neither silent).
; {n_triduum} rows are litcal's own EASTER_TRIDUUM season (Holy Thursday's
; evening Mass, Good Friday, Holy Saturday, all twelve years) -- a season
; colitur's own `Vocab_of.season` has no value for at all (see
; temporal_of.ml's own top-of-module comment); a documented, expected
; divergence class, not a defect on either side.
;
; SECOND-IMPLEMENTATION, NOT A SECOND PUBLICATION. litcal is software that
; reads the same universal Normae colitur's own temporal_of.ml was built
; from -- agreement between the two means two independent engines read
; nn. 43-44 the same way, which catches an IMPLEMENTATION bug (the actual
; risk Phase 1's own report names). It is weaker evidence than a published
; ordo, which is what the EF module's own layers 3-7 mostly rest on
; instead (CLAUDE.md's "know what each layer cannot see" section, and
; test_fiuv_ordo.ml/test_lms_ordo.ml's own headers, on why that gradation
; matters). Confirmed to be reading the UNIVERSAL calendar, not a national
; variant: every one of the twelve files' own `settings` block reads
; `epiphany: JAN6, ascension: THURSDAY, corpus_christi: THURSDAY` with no
; `national_calendar`/`diocesan_calendar` parameter in the archived URL --
; the same universal choices colitur's own temporal_of.ml makes (Epiphany
; fixed 6 January, Ascension and Corpus Christi both on their primary
; Thursday, never the widespread local Sunday-transfer option) -- checked
; directly against all twelve files' own `settings.epiphany`/`ascension`/
; `corpus_christi` fields, not assumed from one sample.
;
; LINEAGE: litcal and catholic-resources were previously established as
; ONE lineage for READING citations (both downstream of the USA-English
; edition) -- docs/superpowers/specs/2026-08-24-colitur-of-rite-module-
; design.md section 5. That finding is about WORDING/versification, which
; a translation can introduce independently of the calendar structure
; underneath it. For the TEMPORAL cycle specifically the concern is
; weaker: week numbering derives from the universal Normae (nn. 43-44),
; not from any translated edition, so a translation-lineage argument does
; not, by itself, implicate the week arithmetic the same way it implicates
; wording. Stated as a real distinction, not used to claim the lineage
; concern is fully dissolved -- litcal is still one compiler's own
; reading of the Normae, not a second one.
;
; Generator: tools/extract_litcal_ordo.py -- do not hand-edit; re-run
; against the same archived JSON (or a refreshed lectio archive/manifest)
; and commit the diff instead:
;   python3 tools/extract_litcal_ordo.py ~/git/projects/lectio/sources \\
;     test/fixtures/litcal-temporal-2024-2035.sexp
;
; Row shape: ((date D) (season S) (ot_week W) (event_key K) (grade_lcl G)).
;   date       ISO-8601 atom.
;   season     litcal's own `liturgical_season`, lowercased atom
;              (advent|christmas|lent|easter|ordinary_time|easter_triduum).
;   ot_week    Some n when an Ord* event_key witnesses the Ordinary Time
;              week on this date, else None -- see this generator's own
;              header for the two/three shapes an unwitnessed week takes.
;   event_key  the representative litcal event_key (audit only -- see
;              [pick_representative]'s own docstring: never load-bearing
;              for season/ot_week correctness, only for a human reading
;              this fixture to see what was on that date).
;   grade_lcl  that same row's own grade word, raw from litcal.
"""
    with open(out_path, "w") as f:
        f.write(header)
        f.write("(\n")
        for dt, season, week, event_key, grade_lcl in rows:
            f.write(
                f"  ((date {dt}) (season {season}) (ot_week {sexp_opt_int(week)}) "
                f"(event_key {sexp_string(event_key)}) (grade_lcl {sexp_string(grade_lcl)}))\n"
            )
        f.write(")\n")
    print(f"wrote {len(rows)} rows to {out_path}", file=sys.stderr)
    print(f"SHA-256 of {out_path}: {sha256_of(out_path)}", file=sys.stderr)


def main():
    if len(sys.argv) != 3:
        print(f"usage: {sys.argv[0]} <lectio-sources-dir> <out-fixture-path>", file=sys.stderr)
        sys.exit(2)
    sources_dir, out_path = sys.argv[1], sys.argv[2]
    manifest_rows = load_manifest(sources_dir)
    files = verify_and_load(sources_dir, manifest_rows)
    print(f"verified {len(files)} litcal files against manifest.tsv, years {files[0][0]}..{files[-1][0]}", file=sys.stderr)
    by_date = collect_rows(files)
    first, last, n = check_contiguous(by_date)
    print(f"contiguous span {first}..{last}, {n} dates, all same-date rows agree on season", file=sys.stderr)
    rows = build_rows(by_date)
    write_fixture(out_path, rows, files, first, last)


if __name__ == "__main__":
    main()