aboutsummaryrefslogtreecommitdiff
path: root/sorter/readme.md
blob: e81c411ce5dccf1651b6bdb2bf031ba5f7f14da3 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
# Downloads Sorter

A Python script that automatically sorts files from your `~/Downloads` folder into organised subdirectories based on filename keywords and — more importantly — the **actual text content** of each file.

Built for Debian Linux, Polish and English documents, Python 3.11+.

---

## How it works

When you run the script it does the following:

1. Scans every supported file in `~/Downloads` (not subdirectories)
2. Extracts the full text content from each file
3. Checks that text against the keyword rules in `rules.toml`
4. Falls back to checking the filename if no content match is found
5. Shows you a **dry-run preview** of where each file would go
6. Asks for confirmation before moving anything

Content always beats filename — so a file called `scan001.pdf` that contains an Arc of Asia invoice will correctly go to `Work/ARC`, even though the filename gives no hint.

### Supported file types

`.pdf` `.docx` `.doc` `.txt` `.xlsx` `.xls` `.md` `.odt`

You can add more in `rules.toml` (see below).

---

## Files

```
sorter/
├── sort_downloads.sh   # Run this — installs deps, then calls the Python script
├── sort_downloads.py   # The actual logic
└── rules.toml          # Your rules — edit this to add keywords and categories
```

Keep all three files in the same directory.

---

## Daily use

```bash
cd ~/.local/bin/sorter
./sort_downloads.sh
```

That's it. The script will show a preview, then ask:

```
Proceed with moving files? [y/N]
```

Type `y` to move, or just press Enter to cancel without touching anything.

### Useful flags

```bash
# Skip the confirmation prompt and move immediately
./sort_downloads.sh --yes

# Preview rules without scanning any files
./sort_downloads.sh --list-rules

# Sort a different folder instead of ~/Downloads
./sort_downloads.sh --dir /path/to/folder

# Use a different rules file
./sort_downloads.sh --config /path/to/other-rules.toml
```

### Output folders

Subfolders are created automatically inside `~/Downloads` the first time a file routes there. Based on the default rules you will get:

```
~/Downloads/
├── Work/
│   ├── ARC/
│   └── LKIT/
├── AKW/
└── Other/
```

If two files would land on the same destination path, the script appends `_1`, `_2` etc. rather than overwriting.

---

## How to extend it

Everything is controlled by `rules.toml`. You never need to touch the Python script to add new keywords or categories.

### Adding a keyword to an existing category

Open `rules.toml` and add a line to the relevant list:

```toml
[[categories]]
name = "Work/ARC"
content_keywords = [
    "arc of asia",
    "9571181577",
    "your new keyword here",   # ← add here
]
filename_keywords = [
    "arc",
    "new_filename_hint",       # ← or here
]
```

`content_keywords` are matched against the full extracted text of the file.
`filename_keywords` are matched against the filename only, and only used if no content match was found first.

Both are case-insensitive and partial — `"kasprzak"` will match `"Łukasz Kasprzak International Trade"`.

### Adding a new category

Append a new `[[categories]]` block anywhere in the list:

```toml
[[categories]]
name        = "Finance/Banking"
description = "Bank statements and account exports"
content_keywords = [
    "revolut",
    "account statement",
    "wyciąg bankowy",
    "mbank",
]
filename_keywords = [
    "revolut",
    "statement",
    "wyciag",
]
```

The folder path (`Finance/Banking`, `Personal/Tax`, or any depth you like) will be created automatically inside `~/Downloads`.

**Order matters** — categories are checked top to bottom and the first match wins. Put more specific categories above broader ones.

### Adding a new file extension

Add it to `supported_extensions` in `rules.toml`:

```toml
supported_extensions = [
    ".pdf",
    ".docx",
    ".txt",
    # ... existing entries ...
    ".log",    # plain text — works out of the box
    ".csv",    # plain text — works out of the box
]
```

Plain text formats (`.log`, `.csv`, `.json`, `.xml`, `.ini`, `.conf`) work immediately with no code changes. Binary formats that need a dedicated parser (e.g. `.pptx`, `.ods`) would require a small addition to `sort_downloads.py`.

### Changing the catch-all folder

Files that match no category go here:

```toml
fallback_folder = "Other"
```

Change it to anything you like, e.g. `"Unsorted"` or `"Inbox"`.

---

## Keyword tips

- **NIP / REGON / KRS numbers** are the most reliable content keywords — they are unique per company and appear on every invoice and document
- Put **broad keywords** (like `"invoice"`) lower in the list so they don't accidentally catch documents that should match a more specific category above
- If a bank statement matches the wrong category because a supplier's address appears as a payee, move that supplier's address out of `content_keywords` and rely on the NIP/REGON instead
- Polish characters work fine in both content and filename keywords (`ł`, `ó`, `ą`, `ś`, `ź`, etc.)
- Run `./sort_downloads.sh --list-rules` after editing to confirm your changes loaded correctly

---

## Dependencies

| Library | Purpose | Installed by |
|---|---|---|
| `pdfplumber` | Read text from PDF files | `sort_downloads.sh` automatically |
| `python-docx` | Read text from .docx/.doc files | `sort_downloads.sh` automatically |
| `openpyxl` | Read text from .xlsx/.xls files | `sort_downloads.sh` automatically |
| `odfpy` | Read text from .odt files | `sort_downloads.sh` automatically |
| `tomllib` | Parse rules.toml | Built into Python 3.11+ — nothing to install |

The bash wrapper (`sort_downloads.sh`) checks for and installs any missing libraries automatically on each run using `pip install --break-system-packages`.

**Python 3.11 or newer is required.** On Debian 12+ this is the default.

---

## Troubleshooting

**A file landed in the wrong folder**
Run `--list-rules` to check what keywords are loaded. The preview also shows the match reason in brackets, e.g. `[content: 'wiercany 60a']` — use this to identify which keyword caused the misroute and either remove it or move a more specific category above it in `rules.toml`.

**A file ended up in Other**
The script found no matching keyword in the file's content or filename. Open the file, find a unique phrase or number, and add it as a `content_keyword` to the appropriate category.

**PDF content is not being read**
Check that `pdfplumber` is installed (`pip show pdfplumber`). Some PDFs are image-only scans with no embedded text — these cannot be read without OCR, which is not currently supported.

**TOML syntax error on startup**
TOML is strict about quoting — all strings must be in double quotes. Numbers like NIP/REGON must also be quoted (`"9571181577"`, not `9571181577`) to be treated as text for matching.