diff options
Diffstat (limited to 'sorter')
| -rw-r--r-- | sorter/readme.md | 210 | ||||
| -rw-r--r-- | sorter/rules.toml | 137 | ||||
| -rw-r--r-- | sorter/sort_downloads.py | 310 | ||||
| -rwxr-xr-x | sorter/sort_downloads.sh | 60 |
4 files changed, 0 insertions, 717 deletions
diff --git a/sorter/readme.md b/sorter/readme.md deleted file mode 100644 index e81c411..0000000 --- a/sorter/readme.md +++ /dev/null @@ -1,210 +0,0 @@ -# Downloads Sorter - -A Python script that automatically sorts files from your `~/Downloads` folder into organised subdirectories based on filename keywords and — more importantly — the **actual text content** of each file. - -Built for Debian Linux, Polish and English documents, Python 3.11+. - ---- - -## How it works - -When you run the script it does the following: - -1. Scans every supported file in `~/Downloads` (not subdirectories) -2. Extracts the full text content from each file -3. Checks that text against the keyword rules in `rules.toml` -4. Falls back to checking the filename if no content match is found -5. Shows you a **dry-run preview** of where each file would go -6. Asks for confirmation before moving anything - -Content always beats filename — so a file called `scan001.pdf` that contains an Arc of Asia invoice will correctly go to `Work/ARC`, even though the filename gives no hint. - -### Supported file types - -`.pdf` `.docx` `.doc` `.txt` `.xlsx` `.xls` `.md` `.odt` - -You can add more in `rules.toml` (see below). - ---- - -## Files - -``` -sorter/ -├── sort_downloads.sh # Run this — installs deps, then calls the Python script -├── sort_downloads.py # The actual logic -└── rules.toml # Your rules — edit this to add keywords and categories -``` - -Keep all three files in the same directory. - ---- - -## Daily use - -```bash -cd ~/.local/bin/sorter -./sort_downloads.sh -``` - -That's it. The script will show a preview, then ask: - -``` -Proceed with moving files? [y/N] -``` - -Type `y` to move, or just press Enter to cancel without touching anything. - -### Useful flags - -```bash -# Skip the confirmation prompt and move immediately -./sort_downloads.sh --yes - -# Preview rules without scanning any files -./sort_downloads.sh --list-rules - -# Sort a different folder instead of ~/Downloads -./sort_downloads.sh --dir /path/to/folder - -# Use a different rules file -./sort_downloads.sh --config /path/to/other-rules.toml -``` - -### Output folders - -Subfolders are created automatically inside `~/Downloads` the first time a file routes there. Based on the default rules you will get: - -``` -~/Downloads/ -├── Work/ -│ ├── ARC/ -│ └── LKIT/ -├── AKW/ -└── Other/ -``` - -If two files would land on the same destination path, the script appends `_1`, `_2` etc. rather than overwriting. - ---- - -## How to extend it - -Everything is controlled by `rules.toml`. You never need to touch the Python script to add new keywords or categories. - -### Adding a keyword to an existing category - -Open `rules.toml` and add a line to the relevant list: - -```toml -[[categories]] -name = "Work/ARC" -content_keywords = [ - "arc of asia", - "9571181577", - "your new keyword here", # ← add here -] -filename_keywords = [ - "arc", - "new_filename_hint", # ← or here -] -``` - -`content_keywords` are matched against the full extracted text of the file. -`filename_keywords` are matched against the filename only, and only used if no content match was found first. - -Both are case-insensitive and partial — `"kasprzak"` will match `"Łukasz Kasprzak International Trade"`. - -### Adding a new category - -Append a new `[[categories]]` block anywhere in the list: - -```toml -[[categories]] -name = "Finance/Banking" -description = "Bank statements and account exports" -content_keywords = [ - "revolut", - "account statement", - "wyciąg bankowy", - "mbank", -] -filename_keywords = [ - "revolut", - "statement", - "wyciag", -] -``` - -The folder path (`Finance/Banking`, `Personal/Tax`, or any depth you like) will be created automatically inside `~/Downloads`. - -**Order matters** — categories are checked top to bottom and the first match wins. Put more specific categories above broader ones. - -### Adding a new file extension - -Add it to `supported_extensions` in `rules.toml`: - -```toml -supported_extensions = [ - ".pdf", - ".docx", - ".txt", - # ... existing entries ... - ".log", # plain text — works out of the box - ".csv", # plain text — works out of the box -] -``` - -Plain text formats (`.log`, `.csv`, `.json`, `.xml`, `.ini`, `.conf`) work immediately with no code changes. Binary formats that need a dedicated parser (e.g. `.pptx`, `.ods`) would require a small addition to `sort_downloads.py`. - -### Changing the catch-all folder - -Files that match no category go here: - -```toml -fallback_folder = "Other" -``` - -Change it to anything you like, e.g. `"Unsorted"` or `"Inbox"`. - ---- - -## Keyword tips - -- **NIP / REGON / KRS numbers** are the most reliable content keywords — they are unique per company and appear on every invoice and document -- Put **broad keywords** (like `"invoice"`) lower in the list so they don't accidentally catch documents that should match a more specific category above -- If a bank statement matches the wrong category because a supplier's address appears as a payee, move that supplier's address out of `content_keywords` and rely on the NIP/REGON instead -- Polish characters work fine in both content and filename keywords (`ł`, `ó`, `ą`, `ś`, `ź`, etc.) -- Run `./sort_downloads.sh --list-rules` after editing to confirm your changes loaded correctly - ---- - -## Dependencies - -| Library | Purpose | Installed by | -|---|---|---| -| `pdfplumber` | Read text from PDF files | `sort_downloads.sh` automatically | -| `python-docx` | Read text from .docx/.doc files | `sort_downloads.sh` automatically | -| `openpyxl` | Read text from .xlsx/.xls files | `sort_downloads.sh` automatically | -| `odfpy` | Read text from .odt files | `sort_downloads.sh` automatically | -| `tomllib` | Parse rules.toml | Built into Python 3.11+ — nothing to install | - -The bash wrapper (`sort_downloads.sh`) checks for and installs any missing libraries automatically on each run using `pip install --break-system-packages`. - -**Python 3.11 or newer is required.** On Debian 12+ this is the default. - ---- - -## Troubleshooting - -**A file landed in the wrong folder** -Run `--list-rules` to check what keywords are loaded. The preview also shows the match reason in brackets, e.g. `[content: 'wiercany 60a']` — use this to identify which keyword caused the misroute and either remove it or move a more specific category above it in `rules.toml`. - -**A file ended up in Other** -The script found no matching keyword in the file's content or filename. Open the file, find a unique phrase or number, and add it as a `content_keyword` to the appropriate category. - -**PDF content is not being read** -Check that `pdfplumber` is installed (`pip show pdfplumber`). Some PDFs are image-only scans with no embedded text — these cannot be read without OCR, which is not currently supported. - -**TOML syntax error on startup** -TOML is strict about quoting — all strings must be in double quotes. Numbers like NIP/REGON must also be quoted (`"9571181577"`, not `9571181577`) to be treated as text for matching. diff --git a/sorter/rules.toml b/sorter/rules.toml deleted file mode 100644 index ae1cbfd..0000000 --- a/sorter/rules.toml +++ /dev/null @@ -1,137 +0,0 @@ -# ───────────────────────────────────────────────────────────────────────────── -# Downloads Sorter — Rules Configuration (TOML) -# -# PRIORITY: content_keywords > filename_keywords -# File contents are scanned first; filename is used as fallback. -# -# FOLDER PATHS: relative to your Downloads directory. -# Use / for subfolders, e.g. "Work/ARC" -# -# KEYWORD TIPS: -# - Case-insensitive ("arc" matches "ARC", "Arc", "arc") -# - Partial match ("kasprzak" matches "Łukasz Kasprzak International Trade") -# - Always quote strings — avoids TOML type surprises with numbers -# - NIP/REGON/KRS numbers are the most reliable content keywords -# - Polish characters work fine (ł, ó, ą, ś, ź, etc.) -# - Categories are checked in order — first match wins -# ───────────────────────────────────────────────────────────────────────────── - -# Catch-all folder for files that match no category -fallback_folder = "Other" - -# File types to scan (add or remove extensions as needed) -supported_extensions = [ - ".pdf", - ".docx", - ".doc", - ".txt", - ".xlsx", - ".xls", - ".md", - ".odt", - ".sh", - ".py", - ".js", - ".ts", - ".html", - ".css", - ".json", - ".yaml", - ".toml", -] - -# ── Categories ──────────────────────────────────────────────────────────────── -# Each [[categories]] block defines one destination folder. -# Add as many blocks as you need. - -[[categories]] -name = "Work/ARC" -description = "Arc of Asia Sp. z o.o. — documents, invoices, correspondence" -content_keywords = [ - "arc of asia", - "arc of asia spółka z ograniczoną odpowiedzialnością", - "arc of asia spolka z ograniczona odpowiedzialnoscia", - "aleja grunwaldzka 56", - "9571181577", # NIP - "540356138", # REGON - "0001140839", # KRS - "pl957118157700000", # EORI -] -filename_keywords = [ - "arc", - "aoa", - "arc_of_asia", -] - -[[categories]] -name = "Work/LKIT" -description = "Łukasz Kasprzak International Trade — documents, invoices, correspondence" -content_keywords = [ - "łukasz kasprzak international trade", - "lukasz kasprzak international trade", - "8181739189", # NIP - "540804571", # REGON -] -filename_keywords = [ - "lkit", - "kasprzak", -] - -[[categories]] -name = "AKW" -description = "Akademia Katolicka w Warszawie" -content_keywords = [ - "akademia katolicka w warszawie", - "akademia katolicka", -] -filename_keywords = [ - "akw", -] - -[[categories]] -name = "Code" -description = "Scripts and source code files" -filename_keywords = [".sh", ".py", ".js", ".ts", ".html", ".css", ".json", ".yaml", ".toml"] -# ── Add more categories below ───────────────────────────────────────────────── - -# [[categories]] -# name = "Finance/Banking" -# description = "Bank statements, Revolut, account history" -# content_keywords = [ -# "revolut", -# "account statement", -# "wyciąg bankowy", -# ] -# filename_keywords = [ -# "revolut", -# "statement", -# "wyciag", -# ] - -# [[categories]] -# name = "Finance/Invoices" -# description = "VAT invoices and receipts" -# content_keywords = [ -# "faktura vat", -# "invoice", -# "23% vat", -# ] -# filename_keywords = [ -# "faktura", -# "invoice", -# ] - -# [[categories]] -# name = "Personal/Tax" -# description = "Tax documents and PIT forms" -# content_keywords = [ -# "pit-37", -# "pit-36", -# "urząd skarbowy", -# "zeznanie podatkowe", -# ] -# filename_keywords = [ -# "pit", -# "podatek", -# "tax", -# ] diff --git a/sorter/sort_downloads.py b/sorter/sort_downloads.py deleted file mode 100644 index 1ced5e3..0000000 --- a/sorter/sort_downloads.py +++ /dev/null @@ -1,310 +0,0 @@ -#!/usr/bin/env python3 -""" -Downloads Folder Sorter -Rules are loaded from rules.toml — edit that file to add keywords/categories. -Content match takes priority over filename match. -Requires Python 3.11+ (uses built-in tomllib). -""" - -import sys -import shutil -import tomllib -import argparse -from pathlib import Path - -# ── Optional deps (graceful fallback) ──────────────────────────────────────── -try: - import pdfplumber - HAS_PDF = True -except ImportError: - HAS_PDF = False - -try: - from docx import Document - HAS_DOCX = True -except ImportError: - HAS_DOCX = False - -try: - import openpyxl - HAS_XLSX = True -except ImportError: - HAS_XLSX = False - -try: - from odf import teletype - from odf.opendocument import load as odf_load - HAS_ODT = True -except ImportError: - HAS_ODT = False - -# ── Config loading ──────────────────────────────────────────────────────────── - -def load_config(config_path: Path) -> dict: - if not config_path.exists(): - print(f"Error: Config file not found: {config_path}") - print("Make sure rules.toml is in the same folder as this script.") - sys.exit(1) - with open(config_path, "rb") as f: - config = tomllib.load(f) - return config - -def validate_config(config: dict): - if "categories" not in config or not config["categories"]: - print("Error: 'categories' is missing or empty in rules.toml") - sys.exit(1) - for cat in config["categories"]: - if "name" not in cat: - print("Error: A category in rules.toml is missing a 'name' field.") - sys.exit(1) - -# ── Text extraction ─────────────────────────────────────────────────────────── - -def extract_text_pdf(path: Path) -> str: - if not HAS_PDF: - return "" - try: - with pdfplumber.open(path) as pdf: - return "\n".join(page.extract_text() or "" for page in pdf.pages) - except Exception: - return "" - -def extract_text_docx(path: Path) -> str: - if not HAS_DOCX: - return "" - try: - doc = Document(path) - return "\n".join(p.text for p in doc.paragraphs) - except Exception: - return "" - -def extract_text_xlsx(path: Path) -> str: - if not HAS_XLSX: - return "" - try: - wb = openpyxl.load_workbook(path, read_only=True, data_only=True) - parts = [] - for ws in wb.worksheets: - for row in ws.iter_rows(values_only=True): - parts.append(" ".join(str(c) for c in row if c is not None)) - return "\n".join(parts) - except Exception: - return "" - -def extract_text_odt(path: Path) -> str: - if not HAS_ODT: - return "" - try: - doc = odf_load(path) - return teletype.extractText(doc.text) - except Exception: - return "" - -def extract_text_plain(path: Path) -> str: - for enc in ("utf-8", "utf-16", "latin-1"): - try: - return path.read_text(encoding=enc) - except Exception: - continue - return "" - -def extract_text(path: Path) -> str: - ext = path.suffix.lower() - if ext == ".pdf": - return extract_text_pdf(path) - elif ext in (".docx", ".doc"): - return extract_text_docx(path) - elif ext in (".xlsx", ".xls"): - return extract_text_xlsx(path) - elif ext == ".odt": - return extract_text_odt(path) - elif ext in (".txt", ".md"): - return extract_text_plain(path) - return "" - -# ── Classification ──────────────────────────────────────────────────────────── - -def classify(text: str, filename: str, categories: list, fallback: str) -> tuple: - """ - Returns (destination_folder, match_reason). - Content keywords checked first, then filename keywords. - Categories are evaluated in order — first match wins. - """ - text_low = text.lower() - name_low = filename.lower() - - for cat in categories: - for kw in cat.get("content_keywords", []): - if str(kw).lower() in text_low: - return cat["name"], f"content: '{kw}'" - - for cat in categories: - for kw in cat.get("filename_keywords", []): - if str(kw).lower() in name_low: - return cat["name"], f"filename: '{kw}'" - - return fallback, "no match" - -# ── Core ────────────────────────────────────────────────────────────────────── - -def collect_files(downloads: Path, extensions: set) -> list: - files = [] - for entry in downloads.iterdir(): - if entry.is_file() and entry.suffix.lower() in extensions: - files.append(entry) - return sorted(files) - -def plan_moves(files: list, downloads: Path, config: dict) -> list: - categories = config["categories"] - fallback = config.get("fallback_folder", "Other") - moves = [] - for f in files: - print(f" Scanning: {f.name} ...", end=" ", flush=True) - text = extract_text(f) - dest_folder, reason = classify(text, f.name, categories, fallback) - dest_path = downloads / dest_folder / f.name - moves.append((f, dest_path, reason)) - print(f"-> {dest_folder}/ ({reason})") - return moves - -def print_plan(moves: list, downloads: Path): - print("\n" + "=" * 70) - print(" DRY-RUN PREVIEW") - print("=" * 70) - col_w = max(len(src.name) for src, _, _ in moves) + 2 - for src, dst, reason in moves: - rel = dst.relative_to(downloads) - print(f" {src.name:<{col_w}} -> {rel} [{reason}]") - print("=" * 70) - -def execute_moves(moves: list, downloads: Path) -> list: - errors = [] - for src, dst, _ in moves: - try: - dst.parent.mkdir(parents=True, exist_ok=True) - final_dst = dst - counter = 1 - while final_dst.exists(): - final_dst = dst.with_stem(f"{dst.stem}_{counter}") - counter += 1 - shutil.move(str(src), str(final_dst)) - print(f" OK {src.name} -> {final_dst.relative_to(downloads)}") - except Exception as e: - errors.append((src, e)) - print(f" ERR {src.name} -> {e}") - return errors - -def check_deps() -> list: - missing = [] - if not HAS_PDF: - missing.append("pdfplumber") - if not HAS_DOCX: - missing.append("python-docx") - if not HAS_XLSX: - missing.append("openpyxl") - if not HAS_ODT: - missing.append("odfpy") - return missing - -# ── Entry point ─────────────────────────────────────────────────────────────── - -def main(): - if sys.version_info < (3, 11): - print("Error: Python 3.11 or newer is required (for built-in tomllib).") - print(f" You have Python {sys.version}") - sys.exit(1) - - parser = argparse.ArgumentParser( - description="Sort ~/downloads using rules defined in rules.toml" - ) - parser.add_argument( - "--dir", - default=str(Path.home() / "downloads"), - help="Path to Downloads folder (default: ~/Downloads)", - ) - parser.add_argument( - "--config", - default=None, - help="Path to rules.toml (default: same folder as this script)", - ) - parser.add_argument( - "--yes", "-y", - action="store_true", - help="Skip confirmation prompt and move immediately", - ) - parser.add_argument( - "--list-rules", - action="store_true", - help="Print loaded categories and keywords, then exit", - ) - args = parser.parse_args() - - script_dir = Path(__file__).parent - config_path = Path(args.config) if args.config else script_dir / "rules.toml" - config = load_config(config_path) - validate_config(config) - - extensions = set(config.get("supported_extensions", [ - ".pdf", ".docx", ".doc", ".txt", ".xlsx", ".xls", ".md", ".odt" - ".sh", ".py", ".js", ".ts", ".html", ".css", ".json", ".yaml", ".toml" - ])) - - if args.list_rules: - print(f"\nConfig: {config_path}\n") - for cat in config["categories"]: - print(f" [folder] {cat['name']}") - if cat.get("description"): - print(f" {cat['description']}") - if cat.get("content_keywords"): - kws = ", ".join(str(k) for k in cat["content_keywords"]) - print(f" content : {kws}") - if cat.get("filename_keywords"): - kws = ", ".join(str(k) for k in cat["filename_keywords"]) - print(f" filename: {kws}") - print() - print(f" [folder] {config.get('fallback_folder', 'Other')} (catch-all)\n") - return - - downloads = Path(args.dir).expanduser().resolve() - if not downloads.is_dir(): - print(f"Error: '{downloads}' is not a directory.") - sys.exit(1) - - missing = check_deps() - if missing: - print("Warning: some content-extraction libraries are missing.") - print(" Install them for full content scanning:") - for m in missing: - print(f" pip install {m}") - print() - - print(f"Scanning: {downloads}") - print(f"Rules: {config_path}\n") - - files = collect_files(downloads, extensions) - if not files: - print("No supported files found.") - sys.exit(0) - - print(f"Found {len(files)} file(s). Classifying...\n") - moves = plan_moves(files, downloads, config) - - print_plan(moves, downloads) - - if not args.yes: - answer = input("\nProceed with moving files? [y/N] ").strip().lower() - if answer != "y": - print("Aborted. No files were moved.") - sys.exit(0) - - print("\nMoving files...\n") - errors = execute_moves(moves, downloads) - - print() - if errors: - print(f"Done with {len(errors)} error(s).") - else: - print("All files moved successfully.") - -if __name__ == "__main__": - main() diff --git a/sorter/sort_downloads.sh b/sorter/sort_downloads.sh deleted file mode 100755 index d2023a8..0000000 --- a/sorter/sort_downloads.sh +++ /dev/null @@ -1,60 +0,0 @@ -#!/usr/bin/env bash -# sort_downloads.sh — wrapper for sort_downloads.py -# Installs missing Python deps, then runs the sorter. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PYTHON_SCRIPT="$SCRIPT_DIR/sort_downloads.py" - -# ── Colour helpers ──────────────────────────────────────────────────────────── -GREEN='\033[0;32m'; YELLOW='\033[1;33m'; RED='\033[0;31m'; NC='\033[0m' -info() { echo -e "${GREEN}[sort]${NC} $*"; } -warn() { echo -e "${YELLOW}[warn]${NC} $*"; } -error() { echo -e "${RED}[error]${NC} $*" >&2; } - -# ── Check Python 3.11+ ──────────────────────────────────────────────────────── -if ! command -v python3 &>/dev/null; then - error "python3 not found. Install it with: sudo apt install python3" - exit 1 -fi - -PYVER=$(python3 -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')") -PYMAJ=$(python3 -c "import sys; print(sys.version_info.major)") -PYMIN=$(python3 -c "import sys; print(sys.version_info.minor)") - -if [[ "$PYMAJ" -lt 3 ]] || [[ "$PYMAJ" -eq 3 && "$PYMIN" -lt 11 ]]; then - error "Python 3.11+ required for built-in tomllib. You have Python $PYVER." - exit 1 -fi - -info "Using Python $(python3 --version) — tomllib built-in, no extra config deps needed." - -# ── Install pip if missing ──────────────────────────────────────────────────── -if ! python3 -m pip --version &>/dev/null; then - warn "pip not found — installing..." - sudo apt-get install -y python3-pip -fi - -# ── Install content-extraction libraries ───────────────────────────────────── -# Note: pyyaml is no longer needed — TOML is built into Python 3.11+ -declare -A DEPS=( - [pdfplumber]="pdfplumber" - [docx]="python-docx" - [openpyxl]="openpyxl" - [odf]="odfpy" -) - -for import_name in "${!DEPS[@]}"; do - pip_name="${DEPS[$import_name]}" - if ! python3 -c "import $import_name" &>/dev/null 2>&1; then - info "Installing $pip_name..." - python3 -m pip install --quiet "$pip_name" --break-system-packages - fi -done - -info "All dependencies ready." -echo "" - -# ── Run the sorter ──────────────────────────────────────────────────────────── -exec python3 "$PYTHON_SCRIPT" "$@" |