WillTools 2026-09-09 -- clean snapshot of the development repo for review
ci / test (push) Successful in 6m14s
ci / test (push) Successful in 6m14s
History-free snapshot of the private development repository at 77a82eb. Code, docs and the public catalogue only: no plant or job knowledge layers, no client, job or producer names (tools/scrub.py clean against the widened term list; a second word-level grep clean). The intro deck is left out because its slide images cannot be checked for job drawings.
This commit is contained in:
@@ -0,0 +1,236 @@
|
||||
# The library index — finding the page that says it
|
||||
|
||||
`tools/libindex.py` builds a **page-granular full-text index** over the
|
||||
knowledge library so that "which document, which page, says X" is one query
|
||||
instead of a re-read. SQLite + FTS5, lexical only.
|
||||
|
||||
```bash
|
||||
python tools/libindex.py build
|
||||
python tools/libindex.py search "lifting anchor" --limit 5
|
||||
python tools/libindex.py status
|
||||
```
|
||||
|
||||
The database is `knowledge/library/index.sqlite`. It is **never committed** —
|
||||
`knowledge/library/` is gitignored as a folder — and it never needs to be,
|
||||
because it is a **rebuildable artifact**: delete the file, run `build`, and you
|
||||
have it back in a couple of seconds. Nothing else in the repo reads it.
|
||||
|
||||
Set `PYTHONIOENCODING=utf-8` on any command whose output you are going to read.
|
||||
The corpus is bilingual and a Windows console is cp1252, so a Chinese line, a
|
||||
typographic dash or a curly quote out of a PDF would otherwise raise
|
||||
`UnicodeEncodeError` and the run would look like a crash rather than like a
|
||||
result. The tool reconfigures its own streams to UTF-8 with `errors='replace'`
|
||||
as a second line of defence, but the environment variable is the one that also
|
||||
covers whatever you pipe it into.
|
||||
|
||||
---
|
||||
|
||||
## What it is not
|
||||
|
||||
**There is no semantic search here.** A query finds the *words you typed*. A
|
||||
result of `0 hits` means "those words are not in the index" — it does **not**
|
||||
mean the library is silent on the subject, and it may never be reported as
|
||||
though it did. The word you searched is very often not the word the document
|
||||
used; try the number, the abbreviation, and the other spelling before you
|
||||
conclude anything.
|
||||
|
||||
A vector layer over the code standards is a plausible later addition and the
|
||||
schema is shaped to accept one — `pages.method` already records *how* a page's
|
||||
text was obtained — but nothing here is one today.
|
||||
|
||||
---
|
||||
|
||||
## Schema
|
||||
|
||||
```sql
|
||||
meta(key PK, value) -- schema_version, last_build, roots
|
||||
|
||||
docs(hash PK, -- sha256 of the file: the identity
|
||||
relpath, -- where it was last seen
|
||||
pages, bytes, mtime, ingested_at)
|
||||
|
||||
pages(doc_hash, page_no, text, method, -- PRIMARY KEY (doc_hash, page_no)
|
||||
...) -- method ∈ {'text-layer'}
|
||||
|
||||
notes(relpath PK, doc_ref, text, mtime, bytes, hash, ingested_at)
|
||||
|
||||
pages_fts -- FTS5 external-content over pages.text
|
||||
notes_fts -- FTS5 external-content over notes.text
|
||||
```
|
||||
|
||||
Both FTS tables are **external-content** rather than contentless, which is what
|
||||
makes `snippet()` work: the highlighted excerpt is read back out of the content
|
||||
table, so a hit shows the sentence rather than only its rowid. They are kept in
|
||||
step by `AFTER INSERT / DELETE / UPDATE` triggers on `pages` and `notes`, so
|
||||
there is no path that writes a row and forgets to index it.
|
||||
|
||||
**Identity is the content hash**, matching the convention the library ledger
|
||||
keys on. A file that moved is the same document and is not re-read — its
|
||||
`relpath` is updated and the build says `1 moved`. A file whose bytes changed
|
||||
is a *new* document: the old row and all of its pages go, the new ones arrive.
|
||||
Two paths holding identical bytes are one document; the second is reported as a
|
||||
duplicate rather than indexed twice.
|
||||
|
||||
**An empty `text` with method `'text-layer'` is a finding, not a gap.** It means
|
||||
the page was opened, its text layer was read, and there was nothing in it — a
|
||||
scanned image whose words currently live only in the extraction notes. That is
|
||||
why the page still gets a row and still gets a method: *looked and found
|
||||
nothing* is a different answer from *never looked*, and `status` prints the
|
||||
count so the difference stays visible. A per-page vision transcription would
|
||||
land as a second `method` value beside this one, not as a rewrite of it.
|
||||
|
||||
`notes.doc_ref` is the `M#` / `D#` / `H#` id the note's filename begins with,
|
||||
when it has one — the join back to the ledger's own numbering. A note whose
|
||||
name carries no id gets `NULL`, not `''`.
|
||||
|
||||
---
|
||||
|
||||
## What gets ingested
|
||||
|
||||
Default roots are `knowledge/library` and `knowledge/MC`, both walked
|
||||
recursively. `--root PATH` (repeatable) replaces them — useful for a document
|
||||
that sits outside either drop folder, or for indexing one folder in isolation.
|
||||
|
||||
* **`*.pdf`** anywhere under a root → one `pages` row per page, text from
|
||||
`pymupdf`'s `page.get_text()`.
|
||||
* **`*.md` under a folder named `notes`** → one `notes` row each. Deliberately
|
||||
*not* every `.md` under the root: the ledger and the runbook are bookkeeping
|
||||
*about* the library rather than content *out of* it, and mixing them into the
|
||||
same ranked list would answer "which document says X" with a row from the
|
||||
index of documents.
|
||||
|
||||
A PDF that cannot be read — corrupt, encrypted — is **named on stdout and not
|
||||
recorded at all**, so the next build retries it instead of treating it as
|
||||
already known.
|
||||
|
||||
---
|
||||
|
||||
## Incremental, and safe to run at any time
|
||||
|
||||
`build` is idempotent. Everything is hashed on every run (that is the identity,
|
||||
and it is cheap); only *new* hashes are opened and text-extracted, which is the
|
||||
expensive part. Notes are compared by content hash too.
|
||||
|
||||
Measured on a full library: **1.4 s** for a first build, **0.05 s** for a re-run
|
||||
with nothing changed. That matters because notes are being written into the
|
||||
library while builds are running — a build is expected to be re-run constantly
|
||||
and to cost nothing when there is nothing to do.
|
||||
|
||||
Removal is **confined to the roots the run was given**. A build named with one
|
||||
`--root` will not delete the rows of a folder it was not asked to look at;
|
||||
otherwise every narrowed re-run would silently gut the index. Rows for files
|
||||
that really have vanished from a scanned root are deleted and the count is
|
||||
printed.
|
||||
|
||||
`--rebuild` throws the tables away first. An index written by an older schema
|
||||
version is wiped and rebuilt automatically, with a line saying so — a silent
|
||||
wipe and a silent no-op look identical from the outside, so it says which one
|
||||
happened.
|
||||
|
||||
---
|
||||
|
||||
## Searching
|
||||
|
||||
Every chunk of your query becomes a quoted FTS5 phrase, so nothing you type is
|
||||
read as operator syntax by accident:
|
||||
|
||||
| you type | FTS5 gets | matches |
|
||||
| --- | --- | --- |
|
||||
| `shear wall` | `"shear" "wall"` | both words, anywhere on the page |
|
||||
| `"shear wall"` | `"shear wall"` | that phrase, in that order |
|
||||
| `7.5` | `"7.5"` | 7 and 5 adjacent — a decimal |
|
||||
| `f'c` | `"f'c"` | the strength symbol |
|
||||
| `#4` | `"#4"` | the bar size, *not* every page with a 4 |
|
||||
| `8"` | `"8"""` | the inch dimension |
|
||||
| `beam*` | `"beam"*` | prefix |
|
||||
|
||||
`--raw` hands the string to FTS5 untouched, for the rare query that really
|
||||
wants `OR` / `NEAR` / a column filter.
|
||||
|
||||
Other flags: `--limit N` (default 10), `--kind pages|notes|all`, and
|
||||
`--doc X` where `X` is a hash prefix, a ledger id, or a path substring.
|
||||
|
||||
Each hit prints the relpath, the page number (or the note's ledger id), the
|
||||
bm25 score and a `snippet()` excerpt with the matched terms in `[brackets]`.
|
||||
**bm25 is negative and lower is better**; results are sorted best-first. Page
|
||||
hits and note hits are merged into one ranked list and tagged `[page]` /
|
||||
`[note]`.
|
||||
|
||||
### Two different kinds of nothing
|
||||
|
||||
The refusals are worded apart on purpose, and the tests assert that neither
|
||||
sentence can be printed for the other:
|
||||
|
||||
| situation | what it prints | exit |
|
||||
| --- | --- | --- |
|
||||
| ran, matched nothing | `0 hits` | 0 |
|
||||
| no database yet | `no index built yet -- run: python tools/libindex.py build` | 1 |
|
||||
| database from another schema | `index is from another schema version -- run: ...` | 1 |
|
||||
| query FTS5 could not parse | `could not run that query -- ...` plus the expression it tried | 1 |
|
||||
| sqlite3 without FTS5 | `this python has no sqlite3 FTS5 ...` | 1 |
|
||||
|
||||
There is no `LIKE` fallback. A machine whose sqlite cannot do FTS5 gets a
|
||||
refusal, not a silently worse index that nobody notices is worse.
|
||||
|
||||
---
|
||||
|
||||
## Tokenizer, and why this one
|
||||
|
||||
```
|
||||
unicode61 remove_diacritics 2 tokenchars '#''"'
|
||||
```
|
||||
|
||||
Three characters are promoted to token characters because each one carries
|
||||
meaning in this corpus that the default tokenizer throws away:
|
||||
|
||||
* **`#`** — `#4` is a bar size. Without it, `#4` and a bare `4` are the same
|
||||
token and every page with a 4 on it answers a bar-size query.
|
||||
* **`'`** — `f'c` is one symbol, not `f` AND `c`.
|
||||
* **`"`** — `8"` is an inch dimension, not a bare 8.
|
||||
|
||||
Plate marks (`P316`) and part numbers need nothing special: unicode61 does not
|
||||
split a letter/digit boundary.
|
||||
|
||||
Three characters were deliberately **left as separators**:
|
||||
|
||||
* **`.`** — making it a token character would keep `7.5` whole, but it would
|
||||
also keep `7.5.` whole at the end of a sentence, and that no longer matches a
|
||||
search for `7.5`. Decimals are handled on the *query* side instead: `7.5`
|
||||
becomes the phrase `"7" "5"`, which requires adjacency — so it matches `7.5`
|
||||
wherever it appears, including before a full stop, and does **not** match a
|
||||
page that merely has a 7 and a 5 somewhere on it. Both directions are tested.
|
||||
* **`-`** — hyphens join words far more often than they join marks; as a token
|
||||
character, a search for `wall` would stop finding `wall-panel`.
|
||||
* **`_`** — same reasoning: leaving it a separator is what lets a search for
|
||||
`anchor` find `ANCHOR_FACE_4T`.
|
||||
|
||||
`remove_diacritics 2` is the Unicode-aware folding, so an accented spelling in a
|
||||
European datasheet matches its unaccented form. CJK is indexed as-is by
|
||||
unicode61 and is searchable; it is not word-segmented, so a Chinese query works
|
||||
best as the exact run of characters used in the source.
|
||||
|
||||
The quoting of that tokenizer string in `libindex.py` is fussy and easy to break
|
||||
— an SQL double-quoted string containing an FTS5 single-quoted string, each with
|
||||
its own quote character doubled. FTS5 rejects `[...]` quoting there whatever
|
||||
the SQL documentation suggests about dequoting. If you change the tokenchars,
|
||||
run the tokenizer tests; they cover exactly the four shapes above.
|
||||
|
||||
---
|
||||
|
||||
## Its output is gitignored material
|
||||
|
||||
`libindex.py` prints the verbatim text of documents that name producers, jobs
|
||||
and clients. **The tool's source never does, and its output must not be pasted
|
||||
into a tracked file** — `CLAUDE.md` §3 has the placeholders (`sample_client`,
|
||||
`sample_job`). The same rule as `tools/library.py`, for the same reason.
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
`WillTools.extension/tests/test_libindex.py`, beside the tests for the other
|
||||
host-CPython tools. Every PDF is built with `pymupdf` into `tmp_path`; nothing
|
||||
reads the real library and nothing writes into the working directory. What is
|
||||
covered is the tokenizer decision (each shape as its own assertion), the
|
||||
incremental behaviour (unchanged / moved / changed / vanished / duplicate /
|
||||
root confinement), and the wording of the two different kinds of nothing.
|
||||
Reference in New Issue
Block a user