willtools-demo/docs/library-index.md

The library index — finding the page that says it

tools/libindex.py builds a page-granular full-text index over the knowledge library so that "which document, which page, says X" is one query instead of a re-read. SQLite + FTS5, lexical only.

python tools/libindex.py build
python tools/libindex.py search "lifting anchor" --limit 5
python tools/libindex.py status

The database is knowledge/library/index.sqlite. It is never committed — knowledge/library/ is gitignored as a folder — and it never needs to be, because it is a rebuildable artifact: delete the file, run build, and you have it back in a couple of seconds. Nothing else in the repo reads it.

Set PYTHONIOENCODING=utf-8 on any command whose output you are going to read. The corpus is bilingual and a Windows console is cp1252, so a Chinese line, a typographic dash or a curly quote out of a PDF would otherwise raise UnicodeEncodeError and the run would look like a crash rather than like a result. The tool reconfigures its own streams to UTF-8 with errors='replace' as a second line of defence, but the environment variable is the one that also covers whatever you pipe it into.

---

What it is not

There is no semantic search here. A query finds the words you typed. A result of 0 hits means "those words are not in the index" — it does not mean the library is silent on the subject, and it may never be reported as though it did. The word you searched is very often not the word the document used; try the number, the abbreviation, and the other spelling before you conclude anything.

A vector layer over the code standards is a plausible later addition and the schema is shaped to accept one — pages.method already records how a page's text was obtained — but nothing here is one today.

---

Schema

meta(key PK, value)                     -- schema_version, last_build, roots

docs(hash PK,                           -- sha256 of the file: the identity
     relpath,                           -- where it was last seen
     pages, bytes, mtime, ingested_at)

pages(doc_hash, page_no, text, method,  -- PRIMARY KEY (doc_hash, page_no)
      ...)                              -- method ∈ {'text-layer'}

notes(relpath PK, doc_ref, text, mtime, bytes, hash, ingested_at)

pages_fts  -- FTS5 external-content over pages.text
notes_fts  -- FTS5 external-content over notes.text

Both FTS tables are external-content rather than contentless, which is what makes snippet() work: the highlighted excerpt is read back out of the content table, so a hit shows the sentence rather than only its rowid. They are kept in step by AFTER INSERT / DELETE / UPDATE triggers on pages and notes, so there is no path that writes a row and forgets to index it.

Identity is the content hash, matching the convention the library ledger keys on. A file that moved is the same document and is not re-read — its relpath is updated and the build says 1 moved. A file whose bytes changed is a new document: the old row and all of its pages go, the new ones arrive. Two paths holding identical bytes are one document; the second is reported as a duplicate rather than indexed twice.

An empty text with method 'text-layer' is a finding, not a gap. It means the page was opened, its text layer was read, and there was nothing in it — a scanned image whose words currently live only in the extraction notes. That is why the page still gets a row and still gets a method: looked and found nothing is a different answer from never looked, and status prints the count so the difference stays visible. A per-page vision transcription would land as a second method value beside this one, not as a rewrite of it.

notes.doc_ref is the M# / D# / H# id the note's filename begins with, when it has one — the join back to the ledger's own numbering. A note whose name carries no id gets NULL, not ''.

---

What gets ingested

Default roots are knowledge/library and knowledge/MC, both walked recursively. --root PATH (repeatable) replaces them — useful for a document that sits outside either drop folder, or for indexing one folder in isolation.

• *.pdf anywhere under a root → one pages row per page, text from pymupdf's page.get_text().
• *.md under a folder named notes → one notes row each. Deliberately not every .md under the root: the ledger and the runbook are bookkeeping about the library rather than content out of it, and mixing them into the same ranked list would answer "which document says X" with a row from the index of documents.

A PDF that cannot be read — corrupt, encrypted — is named on stdout and not recorded at all, so the next build retries it instead of treating it as already known.

---

Incremental, and safe to run at any time

build is idempotent. Everything is hashed on every run (that is the identity, and it is cheap); only new hashes are opened and text-extracted, which is the expensive part. Notes are compared by content hash too.

Measured on a full library: 1.4 s for a first build, 0.05 s for a re-run with nothing changed. That matters because notes are being written into the library while builds are running — a build is expected to be re-run constantly and to cost nothing when there is nothing to do.

Removal is confined to the roots the run was given. A build named with one --root will not delete the rows of a folder it was not asked to look at; otherwise every narrowed re-run would silently gut the index. Rows for files that really have vanished from a scanned root are deleted and the count is printed.

--rebuild throws the tables away first. An index written by an older schema version is wiped and rebuilt automatically, with a line saying so — a silent wipe and a silent no-op look identical from the outside, so it says which one happened.

---

Searching

Every chunk of your query becomes a quoted FTS5 phrase, so nothing you type is read as operator syntax by accident:

you type	FTS5 gets	matches
shear wall	"shear" "wall"	both words, anywhere on the page
"shear wall"	"shear wall"	that phrase, in that order
7.5	"7.5"	7 and 5 adjacent — a decimal
f'c	"f'c"	the strength symbol
#4	"#4"	the bar size, not every page with a 4
8"	"8"""	the inch dimension
beam*	"beam"*	prefix

--raw hands the string to FTS5 untouched, for the rare query that really wants OR / NEAR / a column filter.

Other flags: --limit N (default 10), --kind pages|notes|all, and --doc X where X is a hash prefix, a ledger id, or a path substring.

Each hit prints the relpath, the page number (or the note's ledger id), the bm25 score and a snippet() excerpt with the matched terms in [brackets]. bm25 is negative and lower is better; results are sorted best-first. Page hits and note hits are merged into one ranked list and tagged [page] / [note].

Two different kinds of nothing

The refusals are worded apart on purpose, and the tests assert that neither sentence can be printed for the other:

situation	what it prints	exit
ran, matched nothing	0 hits	0
no database yet	no index built yet -- run: python tools/libindex.py build	1
database from another schema	index is from another schema version -- run: ...	1
query FTS5 could not parse	could not run that query -- ... plus the expression it tried	1
sqlite3 without FTS5	this python has no sqlite3 FTS5 ...	1

There is no LIKE fallback. A machine whose sqlite cannot do FTS5 gets a refusal, not a silently worse index that nobody notices is worse.

---

Tokenizer, and why this one

unicode61 remove_diacritics 2 tokenchars '#''"'

Three characters are promoted to token characters because each one carries meaning in this corpus that the default tokenizer throws away:

• # — #4 is a bar size. Without it, #4 and a bare 4 are the same token and every page with a 4 on it answers a bar-size query.
• ' — f'c is one symbol, not f AND c.
• " — 8" is an inch dimension, not a bare 8.

Plate marks (P316) and part numbers need nothing special: unicode61 does not split a letter/digit boundary.

Three characters were deliberately left as separators:

• . — making it a token character would keep 7.5 whole, but it would also keep 7.5. whole at the end of a sentence, and that no longer matches a search for 7.5. Decimals are handled on the query side instead: 7.5 becomes the phrase "7" "5", which requires adjacency — so it matches 7.5 wherever it appears, including before a full stop, and does not match a page that merely has a 7 and a 5 somewhere on it. Both directions are tested.
• - — hyphens join words far more often than they join marks; as a token character, a search for wall would stop finding wall-panel.
• _ — same reasoning: leaving it a separator is what lets a search for anchor find ANCHOR_FACE_4T.

remove_diacritics 2 is the Unicode-aware folding, so an accented spelling in a European datasheet matches its unaccented form. CJK is indexed as-is by unicode61 and is searchable; it is not word-segmented, so a Chinese query works best as the exact run of characters used in the source.

The quoting of that tokenizer string in libindex.py is fussy and easy to break — an SQL double-quoted string containing an FTS5 single-quoted string, each with its own quote character doubled. FTS5 rejects [...] quoting there whatever the SQL documentation suggests about dequoting. If you change the tokenchars, run the tokenizer tests; they cover exactly the four shapes above.

---

Its output is gitignored material

libindex.py prints the verbatim text of documents that name producers, jobs and clients. The tool's source never does, and its output must not be pasted into a tracked file — CLAUDE.md §3 has the placeholders (sample_client, sample_job). The same rule as tools/library.py, for the same reason.

---

Tests

WillTools.extension/tests/test_libindex.py, beside the tests for the other host-CPython tools. Every PDF is built with pymupdf into tmp_path; nothing reads the real library and nothing writes into the working directory. What is covered is the tokenizer decision (each shape as its own assertion), the incremental behaviour (unchanged / moved / changed / vanished / duplicate / root confinement), and the wording of the two different kinds of nothing.