created sorter for sorting downloads with custom rules; created straper for recreation of server

4 files changed, 703 insertions, 0 deletions

diff --git a/sorter/readme.md b/sorter/readme.md
new file mode 100644
index 0000000..e81c411
--- /dev/null
+++ b/sorter/readme.md
@@ -0,0 +1,210 @@
+# Downloads Sorter
+
+A Python script that automatically sorts files from your `~/Downloads` folder into organised subdirectories based on filename keywords and — more importantly — the **actual text content** of each file.
+
+Built for Debian Linux, Polish and English documents, Python 3.11+.
+
+---
+
+## How it works
+
+When you run the script it does the following:
+
+1. Scans every supported file in `~/Downloads` (not subdirectories)
+2. Extracts the full text content from each file
+3. Checks that text against the keyword rules in `rules.toml`
+4. Falls back to checking the filename if no content match is found
+5. Shows you a **dry-run preview** of where each file would go
+6. Asks for confirmation before moving anything
+
+Content always beats filename — so a file called `scan001.pdf` that contains an Arc of Asia invoice will correctly go to `Work/ARC`, even though the filename gives no hint.
+
+### Supported file types
+
+`.pdf` `.docx` `.doc` `.txt` `.xlsx` `.xls` `.md` `.odt`
+
+You can add more in `rules.toml` (see below).
+
+---
+
+## Files
+
+```
+sorter/
+├── sort_downloads.sh   # Run this — installs deps, then calls the Python script
+├── sort_downloads.py   # The actual logic
+└── rules.toml          # Your rules — edit this to add keywords and categories
+```
+
+Keep all three files in the same directory.
+
+---
+
+## Daily use
+
+```bash
+cd ~/.local/bin/sorter
+./sort_downloads.sh
+```
+
+That's it. The script will show a preview, then ask:
+
+```
+Proceed with moving files? [y/N]
+```
+
+Type `y` to move, or just press Enter to cancel without touching anything.
+
+### Useful flags
+
+```bash
+# Skip the confirmation prompt and move immediately
+./sort_downloads.sh --yes
+
+# Preview rules without scanning any files
+./sort_downloads.sh --list-rules
+
+# Sort a different folder instead of ~/Downloads
+./sort_downloads.sh --dir /path/to/folder
+
+# Use a different rules file
+./sort_downloads.sh --config /path/to/other-rules.toml
+```
+
+### Output folders
+
+Subfolders are created automatically inside `~/Downloads` the first time a file routes there. Based on the default rules you will get:
+
+```
+~/Downloads/
+├── Work/
+│   ├── ARC/
+│   └── LKIT/
+├── AKW/
+└── Other/
+```
+
+If two files would land on the same destination path, the script appends `_1`, `_2` etc. rather than overwriting.
+
+---
+
+## How to extend it
+
+Everything is controlled by `rules.toml`. You never need to touch the Python script to add new keywords or categories.
+
+### Adding a keyword to an existing category
+
+Open `rules.toml` and add a line to the relevant list:
+
+```toml
+[[categories]]
+name = "Work/ARC"
+content_keywords = [
+    "arc of asia",
+    "9571181577",
+    "your new keyword here",   # ← add here
+]
+filename_keywords = [
+    "arc",
+    "new_filename_hint",       # ← or here
+]
+```
+
+`content_keywords` are matched against the full extracted text of the file.
+`filename_keywords` are matched against the filename only, and only used if no content match was found first.
+
+Both are case-insensitive and partial — `"kasprzak"` will match `"Łukasz Kasprzak International Trade"`.
+
+### Adding a new category
+
+Append a new `[[categories]]` block anywhere in the list:
+
+```toml
+[[categories]]
+name        = "Finance/Banking"
+description = "Bank statements and account exports"
+content_keywords = [
+    "revolut",
+    "account statement",
+    "wyciąg bankowy",
+    "mbank",
+]
+filename_keywords = [
+    "revolut",
+    "statement",
+    "wyciag",
+]
+```
+
+The folder path (`Finance/Banking`, `Personal/Tax`, or any depth you like) will be created automatically inside `~/Downloads`.
+
+**Order matters** — categories are checked top to bottom and the first match wins. Put more specific categories above broader ones.
+
+### Adding a new file extension
+
+Add it to `supported_extensions` in `rules.toml`:
+
+```toml
+supported_extensions = [
+    ".pdf",
+    ".docx",
+    ".txt",
+    # ... existing entries ...
+    ".log",    # plain text — works out of the box
+    ".csv",    # plain text — works out of the box
+]
+```
+
+Plain text formats (`.log`, `.csv`, `.json`, `.xml`, `.ini`, `.conf`) work immediately with no code changes. Binary formats that need a dedicated parser (e.g. `.pptx`, `.ods`) would require a small addition to `sort_downloads.py`.
+
+### Changing the catch-all folder
+
+Files that match no category go here:
+
+```toml
+fallback_folder = "Other"
+```
+
+Change it to anything you like, e.g. `"Unsorted"` or `"Inbox"`.
+
+---
+
+## Keyword tips
+
+- **NIP / REGON / KRS numbers** are the most reliable content keywords — they are unique per company and appear on every invoice and document
+- Put **broad keywords** (like `"invoice"`) lower in the list so they don't accidentally catch documents that should match a more specific category above
+- If a bank statement matches the wrong category because a supplier's address appears as a payee, move that supplier's address out of `content_keywords` and rely on the NIP/REGON instead
+- Polish characters work fine in both content and filename keywords (`ł`, `ó`, `ą`, `ś`, `ź`, etc.)
+- Run `./sort_downloads.sh --list-rules` after editing to confirm your changes loaded correctly
+
+---
+
+## Dependencies
+
+| Library | Purpose | Installed by |
+|---|---|---|
+| `pdfplumber` | Read text from PDF files | `sort_downloads.sh` automatically |
+| `python-docx` | Read text from .docx/.doc files | `sort_downloads.sh` automatically |
+| `openpyxl` | Read text from .xlsx/.xls files | `sort_downloads.sh` automatically |
+| `odfpy` | Read text from .odt files | `sort_downloads.sh` automatically |
+| `tomllib` | Parse rules.toml | Built into Python 3.11+ — nothing to install |
+
+The bash wrapper (`sort_downloads.sh`) checks for and installs any missing libraries automatically on each run using `pip install --break-system-packages`.
+
+**Python 3.11 or newer is required.** On Debian 12+ this is the default.
+
+---
+
+## Troubleshooting
+
+**A file landed in the wrong folder**
+Run `--list-rules` to check what keywords are loaded. The preview also shows the match reason in brackets, e.g. `[content: 'wiercany 60a']` — use this to identify which keyword caused the misroute and either remove it or move a more specific category above it in `rules.toml`.
+
+**A file ended up in Other**
+The script found no matching keyword in the file's content or filename. Open the file, find a unique phrase or number, and add it as a `content_keyword` to the appropriate category.
+
+**PDF content is not being read**
+Check that `pdfplumber` is installed (`pip show pdfplumber`). Some PDFs are image-only scans with no embedded text — these cannot be read without OCR, which is not currently supported.
+
+**TOML syntax error on startup**
+TOML is strict about quoting — all strings must be in double quotes. Numbers like NIP/REGON must also be quoted (`"9571181577"`, not `9571181577`) to be treated as text for matching.
diff --git a/sorter/rules.toml b/sorter/rules.toml
new file mode 100644
index 0000000..0c0a7ac
--- /dev/null
+++ b/sorter/rules.toml
@@ -0,0 +1,124 @@
+# ─────────────────────────────────────────────────────────────────────────────
+#  Downloads Sorter — Rules Configuration (TOML)
+#
+#  PRIORITY: content_keywords > filename_keywords
+#  File contents are scanned first; filename is used as fallback.
+#
+#  FOLDER PATHS: relative to your Downloads directory.
+#  Use / for subfolders, e.g. "Work/ARC"
+#
+#  KEYWORD TIPS:
+#  - Case-insensitive ("arc" matches "ARC", "Arc", "arc")
+#  - Partial match ("kasprzak" matches "Łukasz Kasprzak International Trade")
+#  - Always quote strings — avoids TOML type surprises with numbers
+#  - NIP/REGON/KRS numbers are the most reliable content keywords
+#  - Polish characters work fine (ł, ó, ą, ś, ź, etc.)
+#  - Categories are checked in order — first match wins
+# ─────────────────────────────────────────────────────────────────────────────
+
+# Catch-all folder for files that match no category
+fallback_folder = "Other"
+
+# File types to scan (add or remove extensions as needed)
+supported_extensions = [
+    ".pdf",
+    ".docx",
+    ".doc",
+    ".txt",
+    ".xlsx",
+    ".xls",
+    ".md",
+    ".odt",
+]
+
+# ── Categories ────────────────────────────────────────────────────────────────
+# Each [[categories]] block defines one destination folder.
+# Add as many blocks as you need.
+
+[[categories]]
+name        = "Work/ARC"
+description = "Arc of Asia Sp. z o.o. — documents, invoices, correspondence"
+content_keywords = [
+    "arc of asia",
+    "arc of asia spółka z ograniczoną odpowiedzialnością",
+    "arc of asia spolka z ograniczona odpowiedzialnoscia",
+    "aleja grunwaldzka 56",
+    "9571181577",        # NIP
+    "540356138",         # REGON
+    "0001140839",        # KRS
+    "pl957118157700000", # EORI
+]
+filename_keywords = [
+    "arc",
+    "aoa",
+    "arc_of_asia",
+]
+
+[[categories]]
+name        = "Work/LKIT"
+description = "Łukasz Kasprzak International Trade — documents, invoices, correspondence"
+content_keywords = [
+    "łukasz kasprzak international trade",
+    "lukasz kasprzak international trade",
+    "8181739189",  # NIP
+    "540804571",   # REGON
+]
+filename_keywords = [
+    "lkit",
+    "kasprzak",
+]
+
+[[categories]]
+name        = "AKW"
+description = "Akademia Katolicka w Warszawie"
+content_keywords = [
+    "akademia katolicka w warszawie",
+    "akademia katolicka",
+]
+filename_keywords = [
+    "akw",
+]
+
+# ── Add more categories below ─────────────────────────────────────────────────
+
+# [[categories]]
+# name        = "Finance/Banking"
+# description = "Bank statements, Revolut, account history"
+# content_keywords = [
+#     "revolut",
+#     "account statement",
+#     "wyciąg bankowy",
+# ]
+# filename_keywords = [
+#     "revolut",
+#     "statement",
+#     "wyciag",
+# ]
+
+# [[categories]]
+# name        = "Finance/Invoices"
+# description = "VAT invoices and receipts"
+# content_keywords = [
+#     "faktura vat",
+#     "invoice",
+#     "23% vat",
+# ]
+# filename_keywords = [
+#     "faktura",
+#     "invoice",
+# ]
+
+# [[categories]]
+# name        = "Personal/Tax"
+# description = "Tax documents and PIT forms"
+# content_keywords = [
+#     "pit-37",
+#     "pit-36",
+#     "urząd skarbowy",
+#     "zeznanie podatkowe",
+# ]
+# filename_keywords = [
+#     "pit",
+#     "podatek",
+#     "tax",
+# ]
diff --git a/sorter/sort_downloads.py b/sorter/sort_downloads.py
new file mode 100644
index 0000000..10cc696
--- /dev/null
+++ b/sorter/sort_downloads.py
@@ -0,0 +1,309 @@
+#!/usr/bin/env python3
+"""
+Downloads Folder Sorter
+Rules are loaded from rules.toml — edit that file to add keywords/categories.
+Content match takes priority over filename match.
+Requires Python 3.11+ (uses built-in tomllib).
+"""
+
+import sys
+import shutil
+import tomllib
+import argparse
+from pathlib import Path
+
+# ── Optional deps (graceful fallback) ────────────────────────────────────────
+try:
+    import pdfplumber
+    HAS_PDF = True
+except ImportError:
+    HAS_PDF = False
+
+try:
+    from docx import Document
+    HAS_DOCX = True
+except ImportError:
+    HAS_DOCX = False
+
+try:
+    import openpyxl
+    HAS_XLSX = True
+except ImportError:
+    HAS_XLSX = False
+
+try:
+    from odf import teletype
+    from odf.opendocument import load as odf_load
+    HAS_ODT = True
+except ImportError:
+    HAS_ODT = False
+
+# ── Config loading ────────────────────────────────────────────────────────────
+
+def load_config(config_path: Path) -> dict:
+    if not config_path.exists():
+        print(f"Error: Config file not found: {config_path}")
+        print("Make sure rules.toml is in the same folder as this script.")
+        sys.exit(1)
+    with open(config_path, "rb") as f:
+        config = tomllib.load(f)
+    return config
+
+def validate_config(config: dict):
+    if "categories" not in config or not config["categories"]:
+        print("Error: 'categories' is missing or empty in rules.toml")
+        sys.exit(1)
+    for cat in config["categories"]:
+        if "name" not in cat:
+            print("Error: A category in rules.toml is missing a 'name' field.")
+            sys.exit(1)
+
+# ── Text extraction ───────────────────────────────────────────────────────────
+
+def extract_text_pdf(path: Path) -> str:
+    if not HAS_PDF:
+        return ""
+    try:
+        with pdfplumber.open(path) as pdf:
+            return "\n".join(page.extract_text() or "" for page in pdf.pages)
+    except Exception:
+        return ""
+
+def extract_text_docx(path: Path) -> str:
+    if not HAS_DOCX:
+        return ""
+    try:
+        doc = Document(path)
+        return "\n".join(p.text for p in doc.paragraphs)
+    except Exception:
+        return ""
+
+def extract_text_xlsx(path: Path) -> str:
+    if not HAS_XLSX:
+        return ""
+    try:
+        wb = openpyxl.load_workbook(path, read_only=True, data_only=True)
+        parts = []
+        for ws in wb.worksheets:
+            for row in ws.iter_rows(values_only=True):
+                parts.append(" ".join(str(c) for c in row if c is not None))
+        return "\n".join(parts)
+    except Exception:
+        return ""
+
+def extract_text_odt(path: Path) -> str:
+    if not HAS_ODT:
+        return ""
+    try:
+        doc = odf_load(path)
+        return teletype.extractText(doc.text)
+    except Exception:
+        return ""
+
+def extract_text_plain(path: Path) -> str:
+    for enc in ("utf-8", "utf-16", "latin-1"):
+        try:
+            return path.read_text(encoding=enc)
+        except Exception:
+            continue
+    return ""
+
+def extract_text(path: Path) -> str:
+    ext = path.suffix.lower()
+    if ext == ".pdf":
+        return extract_text_pdf(path)
+    elif ext in (".docx", ".doc"):
+        return extract_text_docx(path)
+    elif ext in (".xlsx", ".xls"):
+        return extract_text_xlsx(path)
+    elif ext == ".odt":
+        return extract_text_odt(path)
+    elif ext in (".txt", ".md"):
+        return extract_text_plain(path)
+    return ""
+
+# ── Classification ────────────────────────────────────────────────────────────
+
+def classify(text: str, filename: str, categories: list, fallback: str) -> tuple:
+    """
+    Returns (destination_folder, match_reason).
+    Content keywords checked first, then filename keywords.
+    Categories are evaluated in order — first match wins.
+    """
+    text_low = text.lower()
+    name_low = filename.lower()
+
+    for cat in categories:
+        for kw in cat.get("content_keywords", []):
+            if str(kw).lower() in text_low:
+                return cat["name"], f"content: '{kw}'"
+
+    for cat in categories:
+        for kw in cat.get("filename_keywords", []):
+            if str(kw).lower() in name_low:
+                return cat["name"], f"filename: '{kw}'"
+
+    return fallback, "no match"
+
+# ── Core ──────────────────────────────────────────────────────────────────────
+
+def collect_files(downloads: Path, extensions: set) -> list:
+    files = []
+    for entry in downloads.iterdir():
+        if entry.is_file() and entry.suffix.lower() in extensions:
+            files.append(entry)
+    return sorted(files)
+
+def plan_moves(files: list, downloads: Path, config: dict) -> list:
+    categories = config["categories"]
+    fallback   = config.get("fallback_folder", "Other")
+    moves = []
+    for f in files:
+        print(f"  Scanning: {f.name} ...", end=" ", flush=True)
+        text = extract_text(f)
+        dest_folder, reason = classify(text, f.name, categories, fallback)
+        dest_path = downloads / dest_folder / f.name
+        moves.append((f, dest_path, reason))
+        print(f"-> {dest_folder}/  ({reason})")
+    return moves
+
+def print_plan(moves: list, downloads: Path):
+    print("\n" + "=" * 70)
+    print("  DRY-RUN PREVIEW")
+    print("=" * 70)
+    col_w = max(len(src.name) for src, _, _ in moves) + 2
+    for src, dst, reason in moves:
+        rel = dst.relative_to(downloads)
+        print(f"  {src.name:<{col_w}} ->  {rel}   [{reason}]")
+    print("=" * 70)
+
+def execute_moves(moves: list, downloads: Path) -> list:
+    errors = []
+    for src, dst, _ in moves:
+        try:
+            dst.parent.mkdir(parents=True, exist_ok=True)
+            final_dst = dst
+            counter = 1
+            while final_dst.exists():
+                final_dst = dst.with_stem(f"{dst.stem}_{counter}")
+                counter += 1
+            shutil.move(str(src), str(final_dst))
+            print(f"  OK   {src.name}  ->  {final_dst.relative_to(downloads)}")
+        except Exception as e:
+            errors.append((src, e))
+            print(f"  ERR  {src.name}  ->  {e}")
+    return errors
+
+def check_deps() -> list:
+    missing = []
+    if not HAS_PDF:
+        missing.append("pdfplumber")
+    if not HAS_DOCX:
+        missing.append("python-docx")
+    if not HAS_XLSX:
+        missing.append("openpyxl")
+    if not HAS_ODT:
+        missing.append("odfpy")
+    return missing
+
+# ── Entry point ───────────────────────────────────────────────────────────────
+
+def main():
+    if sys.version_info < (3, 11):
+        print("Error: Python 3.11 or newer is required (for built-in tomllib).")
+        print(f"       You have Python {sys.version}")
+        sys.exit(1)
+
+    parser = argparse.ArgumentParser(
+        description="Sort ~/Downloads using rules defined in rules.toml"
+    )
+    parser.add_argument(
+        "--dir",
+        default=str(Path.home() / "Downloads"),
+        help="Path to Downloads folder (default: ~/Downloads)",
+    )
+    parser.add_argument(
+        "--config",
+        default=None,
+        help="Path to rules.toml (default: same folder as this script)",
+    )
+    parser.add_argument(
+        "--yes", "-y",
+        action="store_true",
+        help="Skip confirmation prompt and move immediately",
+    )
+    parser.add_argument(
+        "--list-rules",
+        action="store_true",
+        help="Print loaded categories and keywords, then exit",
+    )
+    args = parser.parse_args()
+
+    script_dir  = Path(__file__).parent
+    config_path = Path(args.config) if args.config else script_dir / "rules.toml"
+    config      = load_config(config_path)
+    validate_config(config)
+
+    extensions = set(config.get("supported_extensions", [
+        ".pdf", ".docx", ".doc", ".txt", ".xlsx", ".xls", ".md", ".odt"
+    ]))
+
+    if args.list_rules:
+        print(f"\nConfig: {config_path}\n")
+        for cat in config["categories"]:
+            print(f"  [folder] {cat['name']}")
+            if cat.get("description"):
+                print(f"           {cat['description']}")
+            if cat.get("content_keywords"):
+                kws = ", ".join(str(k) for k in cat["content_keywords"])
+                print(f"           content : {kws}")
+            if cat.get("filename_keywords"):
+                kws = ", ".join(str(k) for k in cat["filename_keywords"])
+                print(f"           filename: {kws}")
+            print()
+        print(f"  [folder] {config.get('fallback_folder', 'Other')}  (catch-all)\n")
+        return
+
+    downloads = Path(args.dir).expanduser().resolve()
+    if not downloads.is_dir():
+        print(f"Error: '{downloads}' is not a directory.")
+        sys.exit(1)
+
+    missing = check_deps()
+    if missing:
+        print("Warning: some content-extraction libraries are missing.")
+        print("         Install them for full content scanning:")
+        for m in missing:
+            print(f"           pip install {m}")
+        print()
+
+    print(f"Scanning: {downloads}")
+    print(f"Rules:    {config_path}\n")
+
+    files = collect_files(downloads, extensions)
+    if not files:
+        print("No supported files found.")
+        sys.exit(0)
+
+    print(f"Found {len(files)} file(s). Classifying...\n")
+    moves = plan_moves(files, downloads, config)
+
+    print_plan(moves, downloads)
+
+    if not args.yes:
+        answer = input("\nProceed with moving files? [y/N] ").strip().lower()
+        if answer != "y":
+            print("Aborted. No files were moved.")
+            sys.exit(0)
+
+    print("\nMoving files...\n")
+    errors = execute_moves(moves, downloads)
+
+    print()
+    if errors:
+        print(f"Done with {len(errors)} error(s).")
+    else:
+        print("All files moved successfully.")
+
+if __name__ == "__main__":
+    main()
diff --git a/sorter/sort_downloads.sh b/sorter/sort_downloads.sh
new file mode 100755
index 0000000..d2023a8
--- /dev/null
+++ b/sorter/sort_downloads.sh
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# sort_downloads.sh — wrapper for sort_downloads.py
+# Installs missing Python deps, then runs the sorter.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PYTHON_SCRIPT="$SCRIPT_DIR/sort_downloads.py"
+
+# ── Colour helpers ────────────────────────────────────────────────────────────
+GREEN='\033[0;32m'; YELLOW='\033[1;33m'; RED='\033[0;31m'; NC='\033[0m'
+info()  { echo -e "${GREEN}[sort]${NC} $*"; }
+warn()  { echo -e "${YELLOW}[warn]${NC} $*"; }
+error() { echo -e "${RED}[error]${NC} $*" >&2; }
+
+# ── Check Python 3.11+ ────────────────────────────────────────────────────────
+if ! command -v python3 &>/dev/null; then
+    error "python3 not found. Install it with: sudo apt install python3"
+    exit 1
+fi
+
+PYVER=$(python3 -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')")
+PYMAJ=$(python3 -c "import sys; print(sys.version_info.major)")
+PYMIN=$(python3 -c "import sys; print(sys.version_info.minor)")
+
+if [[ "$PYMAJ" -lt 3 ]] || [[ "$PYMAJ" -eq 3 && "$PYMIN" -lt 11 ]]; then
+    error "Python 3.11+ required for built-in tomllib. You have Python $PYVER."
+    exit 1
+fi
+
+info "Using Python $(python3 --version) — tomllib built-in, no extra config deps needed."
+
+# ── Install pip if missing ────────────────────────────────────────────────────
+if ! python3 -m pip --version &>/dev/null; then
+    warn "pip not found — installing..."
+    sudo apt-get install -y python3-pip
+fi
+
+# ── Install content-extraction libraries ─────────────────────────────────────
+# Note: pyyaml is no longer needed — TOML is built into Python 3.11+
+declare -A DEPS=(
+    [pdfplumber]="pdfplumber"
+    [docx]="python-docx"
+    [openpyxl]="openpyxl"
+    [odf]="odfpy"
+)
+
+for import_name in "${!DEPS[@]}"; do
+    pip_name="${DEPS[$import_name]}"
+    if ! python3 -c "import $import_name" &>/dev/null 2>&1; then
+        info "Installing $pip_name..."
+        python3 -m pip install --quiet "$pip_name" --break-system-packages
+    fi
+done
+
+info "All dependencies ready."
+echo ""
+
+# ── Run the sorter ────────────────────────────────────────────────────────────
+exec python3 "$PYTHON_SCRIPT" "$@"