max-read bounds what a file becomes, not only what is read

max-read gates on file size before reading, then the text was read
whole and decoded: a file that is not valid UTF-8 decodes one byte per
code point and doubles, and normalising and folding copy that again per
set of options, with GOMAXPROCS files in flight. Twelve 40 MB files
reached 3.3 GB - enough to put a laptop into the OOM killer, with no
attacker involved, just a few big .log or .csv files.

Two bounds. The decoded text is cut to max-read at a rune boundary, so
the ceiling means what a reader takes it to mean. And extraction of
files at or above 4 MiB is rationed to two at a time, since holding
several large texts at once is what multiplies the ceiling; smaller
files, which is nearly all of them, are untouched.

  twelve 40 MB files: peak RSS 3294 MB -> 728 MB, wall 27s -> 46s
  two thousand small files: 0.05s both ways

The wall-clock cost falls entirely on large files needing extraction,
and buys a program that finishes instead of being killed.

0 files changed, 0 insertions, 0 deletions