diff options
Diffstat (limited to 'sorter')
| -rw-r--r-- | sorter/readme.md | 210 | ||||
| -rw-r--r-- | sorter/rules.toml | 124 | ||||
| -rw-r--r-- | sorter/sort_downloads.py | 309 | ||||
| -rwxr-xr-x | sorter/sort_downloads.sh | 60 |
4 files changed, 703 insertions, 0 deletions
diff --git a/sorter/readme.md b/sorter/readme.md new file mode 100644 index 0000000..e81c411 --- /dev/null +++ b/sorter/readme.md @@ -0,0 +1,210 @@ +# Downloads Sorter + +A Python script that automatically sorts files from your `~/Downloads` folder into organised subdirectories based on filename keywords and — more importantly — the **actual text content** of each file. + +Built for Debian Linux, Polish and English documents, Python 3.11+. + +--- + +## How it works + +When you run the script it does the following: + +1. Scans every supported file in `~/Downloads` (not subdirectories) +2. Extracts the full text content from each file +3. Checks that text against the keyword rules in `rules.toml` +4. Falls back to checking the filename if no content match is found +5. Shows you a **dry-run preview** of where each file would go +6. Asks for confirmation before moving anything + +Content always beats filename — so a file called `scan001.pdf` that contains an Arc of Asia invoice will correctly go to `Work/ARC`, even though the filename gives no hint. + +### Supported file types + +`.pdf` `.docx` `.doc` `.txt` `.xlsx` `.xls` `.md` `.odt` + +You can add more in `rules.toml` (see below). + +--- + +## Files + +``` +sorter/ +├── sort_downloads.sh # Run this — installs deps, then calls the Python script +├── sort_downloads.py # The actual logic +└── rules.toml # Your rules — edit this to add keywords and categories +``` + +Keep all three files in the same directory. + +--- + +## Daily use + +```bash +cd ~/.local/bin/sorter +./sort_downloads.sh +``` + +That's it. The script will show a preview, then ask: + +``` +Proceed with moving files? [y/N] +``` + +Type `y` to move, or just press Enter to cancel without touching anything. + +### Useful flags + +```bash +# Skip the confirmation prompt and move immediately +./sort_downloads.sh --yes + +# Preview rules without scanning any files +./sort_downloads.sh --list-rules + +# Sort a different folder instead of ~/Downloads +./sort_downloads.sh --dir /path/to/folder + +# Use a different rules file +./sort_downloads.sh --config /path/to/other-rules.toml +``` + +### Output folders + +Subfolders are created automatically inside `~/Downloads` the first time a file routes there. Based on the default rules you will get: + +``` +~/Downloads/ +├── Work/ +│ ├── ARC/ +│ └── LKIT/ +├── AKW/ +└── Other/ +``` + +If two files would land on the same destination path, the script appends `_1`, `_2` etc. rather than overwriting. + +--- + +## How to extend it + +Everything is controlled by `rules.toml`. You never need to touch the Python script to add new keywords or categories. + +### Adding a keyword to an existing category + +Open `rules.toml` and add a line to the relevant list: + +```toml +[[categories]] +name = "Work/ARC" +content_keywords = [ + "arc of asia", + "9571181577", + "your new keyword here", # ← add here +] +filename_keywords = [ + "arc", + "new_filename_hint", # ← or here +] +``` + +`content_keywords` are matched against the full extracted text of the file. +`filename_keywords` are matched against the filename only, and only used if no content match was found first. + +Both are case-insensitive and partial — `"kasprzak"` will match `"Łukasz Kasprzak International Trade"`. + +### Adding a new category + +Append a new `[[categories]]` block anywhere in the list: + +```toml +[[categories]] +name = "Finance/Banking" +description = "Bank statements and account exports" +content_keywords = [ + "revolut", + "account statement", + "wyciąg bankowy", + "mbank", +] +filename_keywords = [ + "revolut", + "statement", + "wyciag", +] +``` + +The folder path (`Finance/Banking`, `Personal/Tax`, or any depth you like) will be created automatically inside `~/Downloads`. + +**Order matters** — categories are checked top to bottom and the first match wins. Put more specific categories above broader ones. + +### Adding a new file extension + +Add it to `supported_extensions` in `rules.toml`: + +```toml +supported_extensions = [ + ".pdf", + ".docx", + ".txt", + # ... existing entries ... + ".log", # plain text — works out of the box + ".csv", # plain text — works out of the box +] +``` + +Plain text formats (`.log`, `.csv`, `.json`, `.xml`, `.ini`, `.conf`) work immediately with no code changes. Binary formats that need a dedicated parser (e.g. `.pptx`, `.ods`) would require a small addition to `sort_downloads.py`. + +### Changing the catch-all folder + +Files that match no category go here: + +```toml +fallback_folder = "Other" +``` + +Change it to anything you like, e.g. `"Unsorted"` or `"Inbox"`. + +--- + +## Keyword tips + +- **NIP / REGON / KRS numbers** are the most reliable content keywords — they are unique per company and appear on every invoice and document +- Put **broad keywords** (like `"invoice"`) lower in the list so they don't accidentally catch documents that should match a more specific category above +- If a bank statement matches the wrong category because a supplier's address appears as a payee, move that supplier's address out of `content_keywords` and rely on the NIP/REGON instead +- Polish characters work fine in both content and filename keywords (`ł`, `ó`, `ą`, `ś`, `ź`, etc.) +- Run `./sort_downloads.sh --list-rules` after editing to confirm your changes loaded correctly + +--- + +## Dependencies + +| Library | Purpose | Installed by | +|---|---|---| +| `pdfplumber` | Read text from PDF files | `sort_downloads.sh` automatically | +| `python-docx` | Read text from .docx/.doc files | `sort_downloads.sh` automatically | +| `openpyxl` | Read text from .xlsx/.xls files | `sort_downloads.sh` automatically | +| `odfpy` | Read text from .odt files | `sort_downloads.sh` automatically | +| `tomllib` | Parse rules.toml | Built into Python 3.11+ — nothing to install | + +The bash wrapper (`sort_downloads.sh`) checks for and installs any missing libraries automatically on each run using `pip install --break-system-packages`. + +**Python 3.11 or newer is required.** On Debian 12+ this is the default. + +--- + +## Troubleshooting + +**A file landed in the wrong folder** +Run `--list-rules` to check what keywords are loaded. The preview also shows the match reason in brackets, e.g. `[content: 'wiercany 60a']` — use this to identify which keyword caused the misroute and either remove it or move a more specific category above it in `rules.toml`. + +**A file ended up in Other** +The script found no matching keyword in the file's content or filename. Open the file, find a unique phrase or number, and add it as a `content_keyword` to the appropriate category. + +**PDF content is not being read** +Check that `pdfplumber` is installed (`pip show pdfplumber`). Some PDFs are image-only scans with no embedded text — these cannot be read without OCR, which is not currently supported. + +**TOML syntax error on startup** +TOML is strict about quoting — all strings must be in double quotes. Numbers like NIP/REGON must also be quoted (`"9571181577"`, not `9571181577`) to be treated as text for matching. diff --git a/sorter/rules.toml b/sorter/rules.toml new file mode 100644 index 0000000..0c0a7ac --- /dev/null +++ b/sorter/rules.toml @@ -0,0 +1,124 @@ +# ───────────────────────────────────────────────────────────────────────────── +# Downloads Sorter — Rules Configuration (TOML) +# +# PRIORITY: content_keywords > filename_keywords +# File contents are scanned first; filename is used as fallback. +# +# FOLDER PATHS: relative to your Downloads directory. +# Use / for subfolders, e.g. "Work/ARC" +# +# KEYWORD TIPS: +# - Case-insensitive ("arc" matches "ARC", "Arc", "arc") +# - Partial match ("kasprzak" matches "Łukasz Kasprzak International Trade") +# - Always quote strings — avoids TOML type surprises with numbers +# - NIP/REGON/KRS numbers are the most reliable content keywords +# - Polish characters work fine (ł, ó, ą, ś, ź, etc.) +# - Categories are checked in order — first match wins +# ───────────────────────────────────────────────────────────────────────────── + +# Catch-all folder for files that match no category +fallback_folder = "Other" + +# File types to scan (add or remove extensions as needed) +supported_extensions = [ + ".pdf", + ".docx", + ".doc", + ".txt", + ".xlsx", + ".xls", + ".md", + ".odt", +] + +# ── Categories ──────────────────────────────────────────────────────────────── +# Each [[categories]] block defines one destination folder. +# Add as many blocks as you need. + +[[categories]] +name = "Work/ARC" +description = "Arc of Asia Sp. z o.o. — documents, invoices, correspondence" +content_keywords = [ + "arc of asia", + "arc of asia spółka z ograniczoną odpowiedzialnością", + "arc of asia spolka z ograniczona odpowiedzialnoscia", + "aleja grunwaldzka 56", + "9571181577", # NIP + "540356138", # REGON + "0001140839", # KRS + "pl957118157700000", # EORI +] +filename_keywords = [ + "arc", + "aoa", + "arc_of_asia", +] + +[[categories]] +name = "Work/LKIT" +description = "Łukasz Kasprzak International Trade — documents, invoices, correspondence" +content_keywords = [ + "łukasz kasprzak international trade", + "lukasz kasprzak international trade", + "8181739189", # NIP + "540804571", # REGON +] +filename_keywords = [ + "lkit", + "kasprzak", +] + +[[categories]] +name = "AKW" +description = "Akademia Katolicka w Warszawie" +content_keywords = [ + "akademia katolicka w warszawie", + "akademia katolicka", +] +filename_keywords = [ + "akw", +] + +# ── Add more categories below ───────────────────────────────────────────────── + +# [[categories]] +# name = "Finance/Banking" +# description = "Bank statements, Revolut, account history" +# content_keywords = [ +# "revolut", +# "account statement", +# "wyciąg bankowy", +# ] +# filename_keywords = [ +# "revolut", +# "statement", +# "wyciag", +# ] + +# [[categories]] +# name = "Finance/Invoices" +# description = "VAT invoices and receipts" +# content_keywords = [ +# "faktura vat", +# "invoice", +# "23% vat", +# ] +# filename_keywords = [ +# "faktura", +# "invoice", +# ] + +# [[categories]] +# name = "Personal/Tax" +# description = "Tax documents and PIT forms" +# content_keywords = [ +# "pit-37", +# "pit-36", +# "urząd skarbowy", +# "zeznanie podatkowe", +# ] +# filename_keywords = [ +# "pit", +# "podatek", +# "tax", +# ] diff --git a/sorter/sort_downloads.py b/sorter/sort_downloads.py new file mode 100644 index 0000000..10cc696 --- /dev/null +++ b/sorter/sort_downloads.py @@ -0,0 +1,309 @@ +#!/usr/bin/env python3 +""" +Downloads Folder Sorter +Rules are loaded from rules.toml — edit that file to add keywords/categories. +Content match takes priority over filename match. +Requires Python 3.11+ (uses built-in tomllib). +""" + +import sys +import shutil +import tomllib +import argparse +from pathlib import Path + +# ── Optional deps (graceful fallback) ──────────────────────────────────────── +try: + import pdfplumber + HAS_PDF = True +except ImportError: + HAS_PDF = False + +try: + from docx import Document + HAS_DOCX = True +except ImportError: + HAS_DOCX = False + +try: + import openpyxl + HAS_XLSX = True +except ImportError: + HAS_XLSX = False + +try: + from odf import teletype + from odf.opendocument import load as odf_load + HAS_ODT = True +except ImportError: + HAS_ODT = False + +# ── Config loading ──────────────────────────────────────────────────────────── + +def load_config(config_path: Path) -> dict: + if not config_path.exists(): + print(f"Error: Config file not found: {config_path}") + print("Make sure rules.toml is in the same folder as this script.") + sys.exit(1) + with open(config_path, "rb") as f: + config = tomllib.load(f) + return config + +def validate_config(config: dict): + if "categories" not in config or not config["categories"]: + print("Error: 'categories' is missing or empty in rules.toml") + sys.exit(1) + for cat in config["categories"]: + if "name" not in cat: + print("Error: A category in rules.toml is missing a 'name' field.") + sys.exit(1) + +# ── Text extraction ─────────────────────────────────────────────────────────── + +def extract_text_pdf(path: Path) -> str: + if not HAS_PDF: + return "" + try: + with pdfplumber.open(path) as pdf: + return "\n".join(page.extract_text() or "" for page in pdf.pages) + except Exception: + return "" + +def extract_text_docx(path: Path) -> str: + if not HAS_DOCX: + return "" + try: + doc = Document(path) + return "\n".join(p.text for p in doc.paragraphs) + except Exception: + return "" + +def extract_text_xlsx(path: Path) -> str: + if not HAS_XLSX: + return "" + try: + wb = openpyxl.load_workbook(path, read_only=True, data_only=True) + parts = [] + for ws in wb.worksheets: + for row in ws.iter_rows(values_only=True): + parts.append(" ".join(str(c) for c in row if c is not None)) + return "\n".join(parts) + except Exception: + return "" + +def extract_text_odt(path: Path) -> str: + if not HAS_ODT: + return "" + try: + doc = odf_load(path) + return teletype.extractText(doc.text) + except Exception: + return "" + +def extract_text_plain(path: Path) -> str: + for enc in ("utf-8", "utf-16", "latin-1"): + try: + return path.read_text(encoding=enc) + except Exception: + continue + return "" + +def extract_text(path: Path) -> str: + ext = path.suffix.lower() + if ext == ".pdf": + return extract_text_pdf(path) + elif ext in (".docx", ".doc"): + return extract_text_docx(path) + elif ext in (".xlsx", ".xls"): + return extract_text_xlsx(path) + elif ext == ".odt": + return extract_text_odt(path) + elif ext in (".txt", ".md"): + return extract_text_plain(path) + return "" + +# ── Classification ──────────────────────────────────────────────────────────── + +def classify(text: str, filename: str, categories: list, fallback: str) -> tuple: + """ + Returns (destination_folder, match_reason). + Content keywords checked first, then filename keywords. + Categories are evaluated in order — first match wins. + """ + text_low = text.lower() + name_low = filename.lower() + + for cat in categories: + for kw in cat.get("content_keywords", []): + if str(kw).lower() in text_low: + return cat["name"], f"content: '{kw}'" + + for cat in categories: + for kw in cat.get("filename_keywords", []): + if str(kw).lower() in name_low: + return cat["name"], f"filename: '{kw}'" + + return fallback, "no match" + +# ── Core ────────────────────────────────────────────────────────────────────── + +def collect_files(downloads: Path, extensions: set) -> list: + files = [] + for entry in downloads.iterdir(): + if entry.is_file() and entry.suffix.lower() in extensions: + files.append(entry) + return sorted(files) + +def plan_moves(files: list, downloads: Path, config: dict) -> list: + categories = config["categories"] + fallback = config.get("fallback_folder", "Other") + moves = [] + for f in files: + print(f" Scanning: {f.name} ...", end=" ", flush=True) + text = extract_text(f) + dest_folder, reason = classify(text, f.name, categories, fallback) + dest_path = downloads / dest_folder / f.name + moves.append((f, dest_path, reason)) + print(f"-> {dest_folder}/ ({reason})") + return moves + +def print_plan(moves: list, downloads: Path): + print("\n" + "=" * 70) + print(" DRY-RUN PREVIEW") + print("=" * 70) + col_w = max(len(src.name) for src, _, _ in moves) + 2 + for src, dst, reason in moves: + rel = dst.relative_to(downloads) + print(f" {src.name:<{col_w}} -> {rel} [{reason}]") + print("=" * 70) + +def execute_moves(moves: list, downloads: Path) -> list: + errors = [] + for src, dst, _ in moves: + try: + dst.parent.mkdir(parents=True, exist_ok=True) + final_dst = dst + counter = 1 + while final_dst.exists(): + final_dst = dst.with_stem(f"{dst.stem}_{counter}") + counter += 1 + shutil.move(str(src), str(final_dst)) + print(f" OK {src.name} -> {final_dst.relative_to(downloads)}") + except Exception as e: + errors.append((src, e)) + print(f" ERR {src.name} -> {e}") + return errors + +def check_deps() -> list: + missing = [] + if not HAS_PDF: + missing.append("pdfplumber") + if not HAS_DOCX: + missing.append("python-docx") + if not HAS_XLSX: + missing.append("openpyxl") + if not HAS_ODT: + missing.append("odfpy") + return missing + +# ── Entry point ─────────────────────────────────────────────────────────────── + +def main(): + if sys.version_info < (3, 11): + print("Error: Python 3.11 or newer is required (for built-in tomllib).") + print(f" You have Python {sys.version}") + sys.exit(1) + + parser = argparse.ArgumentParser( + description="Sort ~/Downloads using rules defined in rules.toml" + ) + parser.add_argument( + "--dir", + default=str(Path.home() / "Downloads"), + help="Path to Downloads folder (default: ~/Downloads)", + ) + parser.add_argument( + "--config", + default=None, + help="Path to rules.toml (default: same folder as this script)", + ) + parser.add_argument( + "--yes", "-y", + action="store_true", + help="Skip confirmation prompt and move immediately", + ) + parser.add_argument( + "--list-rules", + action="store_true", + help="Print loaded categories and keywords, then exit", + ) + args = parser.parse_args() + + script_dir = Path(__file__).parent + config_path = Path(args.config) if args.config else script_dir / "rules.toml" + config = load_config(config_path) + validate_config(config) + + extensions = set(config.get("supported_extensions", [ + ".pdf", ".docx", ".doc", ".txt", ".xlsx", ".xls", ".md", ".odt" + ])) + + if args.list_rules: + print(f"\nConfig: {config_path}\n") + for cat in config["categories"]: + print(f" [folder] {cat['name']}") + if cat.get("description"): + print(f" {cat['description']}") + if cat.get("content_keywords"): + kws = ", ".join(str(k) for k in cat["content_keywords"]) + print(f" content : {kws}") + if cat.get("filename_keywords"): + kws = ", ".join(str(k) for k in cat["filename_keywords"]) + print(f" filename: {kws}") + print() + print(f" [folder] {config.get('fallback_folder', 'Other')} (catch-all)\n") + return + + downloads = Path(args.dir).expanduser().resolve() + if not downloads.is_dir(): + print(f"Error: '{downloads}' is not a directory.") + sys.exit(1) + + missing = check_deps() + if missing: + print("Warning: some content-extraction libraries are missing.") + print(" Install them for full content scanning:") + for m in missing: + print(f" pip install {m}") + print() + + print(f"Scanning: {downloads}") + print(f"Rules: {config_path}\n") + + files = collect_files(downloads, extensions) + if not files: + print("No supported files found.") + sys.exit(0) + + print(f"Found {len(files)} file(s). Classifying...\n") + moves = plan_moves(files, downloads, config) + + print_plan(moves, downloads) + + if not args.yes: + answer = input("\nProceed with moving files? [y/N] ").strip().lower() + if answer != "y": + print("Aborted. No files were moved.") + sys.exit(0) + + print("\nMoving files...\n") + errors = execute_moves(moves, downloads) + + print() + if errors: + print(f"Done with {len(errors)} error(s).") + else: + print("All files moved successfully.") + +if __name__ == "__main__": + main() diff --git a/sorter/sort_downloads.sh b/sorter/sort_downloads.sh new file mode 100755 index 0000000..d2023a8 --- /dev/null +++ b/sorter/sort_downloads.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# sort_downloads.sh — wrapper for sort_downloads.py +# Installs missing Python deps, then runs the sorter. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PYTHON_SCRIPT="$SCRIPT_DIR/sort_downloads.py" + +# ── Colour helpers ──────────────────────────────────────────────────────────── +GREEN='\033[0;32m'; YELLOW='\033[1;33m'; RED='\033[0;31m'; NC='\033[0m' +info() { echo -e "${GREEN}[sort]${NC} $*"; } +warn() { echo -e "${YELLOW}[warn]${NC} $*"; } +error() { echo -e "${RED}[error]${NC} $*" >&2; } + +# ── Check Python 3.11+ ──────────────────────────────────────────────────────── +if ! command -v python3 &>/dev/null; then + error "python3 not found. Install it with: sudo apt install python3" + exit 1 +fi + +PYVER=$(python3 -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')") +PYMAJ=$(python3 -c "import sys; print(sys.version_info.major)") +PYMIN=$(python3 -c "import sys; print(sys.version_info.minor)") + +if [[ "$PYMAJ" -lt 3 ]] || [[ "$PYMAJ" -eq 3 && "$PYMIN" -lt 11 ]]; then + error "Python 3.11+ required for built-in tomllib. You have Python $PYVER." + exit 1 +fi + +info "Using Python $(python3 --version) — tomllib built-in, no extra config deps needed." + +# ── Install pip if missing ──────────────────────────────────────────────────── +if ! python3 -m pip --version &>/dev/null; then + warn "pip not found — installing..." + sudo apt-get install -y python3-pip +fi + +# ── Install content-extraction libraries ───────────────────────────────────── +# Note: pyyaml is no longer needed — TOML is built into Python 3.11+ +declare -A DEPS=( + [pdfplumber]="pdfplumber" + [docx]="python-docx" + [openpyxl]="openpyxl" + [odf]="odfpy" +) + +for import_name in "${!DEPS[@]}"; do + pip_name="${DEPS[$import_name]}" + if ! python3 -c "import $import_name" &>/dev/null 2>&1; then + info "Installing $pip_name..." + python3 -m pip install --quiet "$pip_name" --break-system-packages + fi +done + +info "All dependencies ready." +echo "" + +# ── Run the sorter ──────────────────────────────────────────────────────────── +exec python3 "$PYTHON_SCRIPT" "$@" |