# The library index — finding the page that says it `tools/libindex.py` builds a **page-granular full-text index** over the knowledge library so that "which document, which page, says X" is one query instead of a re-read. SQLite + FTS5, lexical only. ```bash python tools/libindex.py build python tools/libindex.py search "lifting anchor" --limit 5 python tools/libindex.py status ``` The database is `knowledge/library/index.sqlite`. It is **never committed** — `knowledge/library/` is gitignored as a folder — and it never needs to be, because it is a **rebuildable artifact**: delete the file, run `build`, and you have it back in a couple of seconds. Nothing else in the repo reads it. Set `PYTHONIOENCODING=utf-8` on any command whose output you are going to read. The corpus is bilingual and a Windows console is cp1252, so a Chinese line, a typographic dash or a curly quote out of a PDF would otherwise raise `UnicodeEncodeError` and the run would look like a crash rather than like a result. The tool reconfigures its own streams to UTF-8 with `errors='replace'` as a second line of defence, but the environment variable is the one that also covers whatever you pipe it into. --- ## What it is not **There is no semantic search here.** A query finds the *words you typed*. A result of `0 hits` means "those words are not in the index" — it does **not** mean the library is silent on the subject, and it may never be reported as though it did. The word you searched is very often not the word the document used; try the number, the abbreviation, and the other spelling before you conclude anything. A vector layer over the code standards is a plausible later addition and the schema is shaped to accept one — `pages.method` already records *how* a page's text was obtained — but nothing here is one today. --- ## Schema ```sql meta(key PK, value) -- schema_version, last_build, roots docs(hash PK, -- sha256 of the file: the identity relpath, -- where it was last seen pages, bytes, mtime, ingested_at) pages(doc_hash, page_no, text, method, -- PRIMARY KEY (doc_hash, page_no) ...) -- method ∈ {'text-layer'} notes(relpath PK, doc_ref, text, mtime, bytes, hash, ingested_at) pages_fts -- FTS5 external-content over pages.text notes_fts -- FTS5 external-content over notes.text ``` Both FTS tables are **external-content** rather than contentless, which is what makes `snippet()` work: the highlighted excerpt is read back out of the content table, so a hit shows the sentence rather than only its rowid. They are kept in step by `AFTER INSERT / DELETE / UPDATE` triggers on `pages` and `notes`, so there is no path that writes a row and forgets to index it. **Identity is the content hash**, matching the convention the library ledger keys on. A file that moved is the same document and is not re-read — its `relpath` is updated and the build says `1 moved`. A file whose bytes changed is a *new* document: the old row and all of its pages go, the new ones arrive. Two paths holding identical bytes are one document; the second is reported as a duplicate rather than indexed twice. **An empty `text` with method `'text-layer'` is a finding, not a gap.** It means the page was opened, its text layer was read, and there was nothing in it — a scanned image whose words currently live only in the extraction notes. That is why the page still gets a row and still gets a method: *looked and found nothing* is a different answer from *never looked*, and `status` prints the count so the difference stays visible. A per-page vision transcription would land as a second `method` value beside this one, not as a rewrite of it. `notes.doc_ref` is the `M#` / `D#` / `H#` id the note's filename begins with, when it has one — the join back to the ledger's own numbering. A note whose name carries no id gets `NULL`, not `''`. --- ## What gets ingested Default roots are `knowledge/library` and `knowledge/MC`, both walked recursively. `--root PATH` (repeatable) replaces them — useful for a document that sits outside either drop folder, or for indexing one folder in isolation. * **`*.pdf`** anywhere under a root → one `pages` row per page, text from `pymupdf`'s `page.get_text()`. * **`*.md` under a folder named `notes`** → one `notes` row each. Deliberately *not* every `.md` under the root: the ledger and the runbook are bookkeeping *about* the library rather than content *out of* it, and mixing them into the same ranked list would answer "which document says X" with a row from the index of documents. A PDF that cannot be read — corrupt, encrypted — is **named on stdout and not recorded at all**, so the next build retries it instead of treating it as already known. --- ## Incremental, and safe to run at any time `build` is idempotent. Everything is hashed on every run (that is the identity, and it is cheap); only *new* hashes are opened and text-extracted, which is the expensive part. Notes are compared by content hash too. Measured on a full library: **1.4 s** for a first build, **0.05 s** for a re-run with nothing changed. That matters because notes are being written into the library while builds are running — a build is expected to be re-run constantly and to cost nothing when there is nothing to do. Removal is **confined to the roots the run was given**. A build named with one `--root` will not delete the rows of a folder it was not asked to look at; otherwise every narrowed re-run would silently gut the index. Rows for files that really have vanished from a scanned root are deleted and the count is printed. `--rebuild` throws the tables away first. An index written by an older schema version is wiped and rebuilt automatically, with a line saying so — a silent wipe and a silent no-op look identical from the outside, so it says which one happened. --- ## Searching Every chunk of your query becomes a quoted FTS5 phrase, so nothing you type is read as operator syntax by accident: | you type | FTS5 gets | matches | | --- | --- | --- | | `shear wall` | `"shear" "wall"` | both words, anywhere on the page | | `"shear wall"` | `"shear wall"` | that phrase, in that order | | `7.5` | `"7.5"` | 7 and 5 adjacent — a decimal | | `f'c` | `"f'c"` | the strength symbol | | `#4` | `"#4"` | the bar size, *not* every page with a 4 | | `8"` | `"8"""` | the inch dimension | | `beam*` | `"beam"*` | prefix | `--raw` hands the string to FTS5 untouched, for the rare query that really wants `OR` / `NEAR` / a column filter. Other flags: `--limit N` (default 10), `--kind pages|notes|all`, and `--doc X` where `X` is a hash prefix, a ledger id, or a path substring. Each hit prints the relpath, the page number (or the note's ledger id), the bm25 score and a `snippet()` excerpt with the matched terms in `[brackets]`. **bm25 is negative and lower is better**; results are sorted best-first. Page hits and note hits are merged into one ranked list and tagged `[page]` / `[note]`. ### Two different kinds of nothing The refusals are worded apart on purpose, and the tests assert that neither sentence can be printed for the other: | situation | what it prints | exit | | --- | --- | --- | | ran, matched nothing | `0 hits` | 0 | | no database yet | `no index built yet -- run: python tools/libindex.py build` | 1 | | database from another schema | `index is from another schema version -- run: ...` | 1 | | query FTS5 could not parse | `could not run that query -- ...` plus the expression it tried | 1 | | sqlite3 without FTS5 | `this python has no sqlite3 FTS5 ...` | 1 | There is no `LIKE` fallback. A machine whose sqlite cannot do FTS5 gets a refusal, not a silently worse index that nobody notices is worse. --- ## Tokenizer, and why this one ``` unicode61 remove_diacritics 2 tokenchars '#''"' ``` Three characters are promoted to token characters because each one carries meaning in this corpus that the default tokenizer throws away: * **`#`** — `#4` is a bar size. Without it, `#4` and a bare `4` are the same token and every page with a 4 on it answers a bar-size query. * **`'`** — `f'c` is one symbol, not `f` AND `c`. * **`"`** — `8"` is an inch dimension, not a bare 8. Plate marks (`P316`) and part numbers need nothing special: unicode61 does not split a letter/digit boundary. Three characters were deliberately **left as separators**: * **`.`** — making it a token character would keep `7.5` whole, but it would also keep `7.5.` whole at the end of a sentence, and that no longer matches a search for `7.5`. Decimals are handled on the *query* side instead: `7.5` becomes the phrase `"7" "5"`, which requires adjacency — so it matches `7.5` wherever it appears, including before a full stop, and does **not** match a page that merely has a 7 and a 5 somewhere on it. Both directions are tested. * **`-`** — hyphens join words far more often than they join marks; as a token character, a search for `wall` would stop finding `wall-panel`. * **`_`** — same reasoning: leaving it a separator is what lets a search for `anchor` find `ANCHOR_FACE_4T`. `remove_diacritics 2` is the Unicode-aware folding, so an accented spelling in a European datasheet matches its unaccented form. CJK is indexed as-is by unicode61 and is searchable; it is not word-segmented, so a Chinese query works best as the exact run of characters used in the source. The quoting of that tokenizer string in `libindex.py` is fussy and easy to break — an SQL double-quoted string containing an FTS5 single-quoted string, each with its own quote character doubled. FTS5 rejects `[...]` quoting there whatever the SQL documentation suggests about dequoting. If you change the tokenchars, run the tokenizer tests; they cover exactly the four shapes above. --- ## Its output is gitignored material `libindex.py` prints the verbatim text of documents that name producers, jobs and clients. **The tool's source never does, and its output must not be pasted into a tracked file** — `CLAUDE.md` §3 has the placeholders (`sample_client`, `sample_job`). The same rule as `tools/library.py`, for the same reason. --- ## Tests `WillTools.extension/tests/test_libindex.py`, beside the tests for the other host-CPython tools. Every PDF is built with `pymupdf` into `tmp_path`; nothing reads the real library and nothing writes into the working directory. What is covered is the tokenizer decision (each shape as its own assertion), the incremental behaviour (unchanged / moved / changed / vanished / duplicate / root confinement), and the wording of the two different kinds of nothing.