will
25592fa6c2
ci / test (push) Successful in 6m14s
History-free snapshot of the private development repository at 77a82eb. Code, docs and the public catalogue only: no plant or job knowledge layers, no client, job or producer names (tools/scrub.py clean against the widened term list; a second word-level grep clean). The intro deck is left out because its slide images cannot be checked for job drawings.
237 lines
10 KiB
Markdown
237 lines
10 KiB
Markdown
# The library index — finding the page that says it
|
|
|
|
`tools/libindex.py` builds a **page-granular full-text index** over the
|
|
knowledge library so that "which document, which page, says X" is one query
|
|
instead of a re-read. SQLite + FTS5, lexical only.
|
|
|
|
```bash
|
|
python tools/libindex.py build
|
|
python tools/libindex.py search "lifting anchor" --limit 5
|
|
python tools/libindex.py status
|
|
```
|
|
|
|
The database is `knowledge/library/index.sqlite`. It is **never committed** —
|
|
`knowledge/library/` is gitignored as a folder — and it never needs to be,
|
|
because it is a **rebuildable artifact**: delete the file, run `build`, and you
|
|
have it back in a couple of seconds. Nothing else in the repo reads it.
|
|
|
|
Set `PYTHONIOENCODING=utf-8` on any command whose output you are going to read.
|
|
The corpus is bilingual and a Windows console is cp1252, so a Chinese line, a
|
|
typographic dash or a curly quote out of a PDF would otherwise raise
|
|
`UnicodeEncodeError` and the run would look like a crash rather than like a
|
|
result. The tool reconfigures its own streams to UTF-8 with `errors='replace'`
|
|
as a second line of defence, but the environment variable is the one that also
|
|
covers whatever you pipe it into.
|
|
|
|
---
|
|
|
|
## What it is not
|
|
|
|
**There is no semantic search here.** A query finds the *words you typed*. A
|
|
result of `0 hits` means "those words are not in the index" — it does **not**
|
|
mean the library is silent on the subject, and it may never be reported as
|
|
though it did. The word you searched is very often not the word the document
|
|
used; try the number, the abbreviation, and the other spelling before you
|
|
conclude anything.
|
|
|
|
A vector layer over the code standards is a plausible later addition and the
|
|
schema is shaped to accept one — `pages.method` already records *how* a page's
|
|
text was obtained — but nothing here is one today.
|
|
|
|
---
|
|
|
|
## Schema
|
|
|
|
```sql
|
|
meta(key PK, value) -- schema_version, last_build, roots
|
|
|
|
docs(hash PK, -- sha256 of the file: the identity
|
|
relpath, -- where it was last seen
|
|
pages, bytes, mtime, ingested_at)
|
|
|
|
pages(doc_hash, page_no, text, method, -- PRIMARY KEY (doc_hash, page_no)
|
|
...) -- method ∈ {'text-layer'}
|
|
|
|
notes(relpath PK, doc_ref, text, mtime, bytes, hash, ingested_at)
|
|
|
|
pages_fts -- FTS5 external-content over pages.text
|
|
notes_fts -- FTS5 external-content over notes.text
|
|
```
|
|
|
|
Both FTS tables are **external-content** rather than contentless, which is what
|
|
makes `snippet()` work: the highlighted excerpt is read back out of the content
|
|
table, so a hit shows the sentence rather than only its rowid. They are kept in
|
|
step by `AFTER INSERT / DELETE / UPDATE` triggers on `pages` and `notes`, so
|
|
there is no path that writes a row and forgets to index it.
|
|
|
|
**Identity is the content hash**, matching the convention the library ledger
|
|
keys on. A file that moved is the same document and is not re-read — its
|
|
`relpath` is updated and the build says `1 moved`. A file whose bytes changed
|
|
is a *new* document: the old row and all of its pages go, the new ones arrive.
|
|
Two paths holding identical bytes are one document; the second is reported as a
|
|
duplicate rather than indexed twice.
|
|
|
|
**An empty `text` with method `'text-layer'` is a finding, not a gap.** It means
|
|
the page was opened, its text layer was read, and there was nothing in it — a
|
|
scanned image whose words currently live only in the extraction notes. That is
|
|
why the page still gets a row and still gets a method: *looked and found
|
|
nothing* is a different answer from *never looked*, and `status` prints the
|
|
count so the difference stays visible. A per-page vision transcription would
|
|
land as a second `method` value beside this one, not as a rewrite of it.
|
|
|
|
`notes.doc_ref` is the `M#` / `D#` / `H#` id the note's filename begins with,
|
|
when it has one — the join back to the ledger's own numbering. A note whose
|
|
name carries no id gets `NULL`, not `''`.
|
|
|
|
---
|
|
|
|
## What gets ingested
|
|
|
|
Default roots are `knowledge/library` and `knowledge/MC`, both walked
|
|
recursively. `--root PATH` (repeatable) replaces them — useful for a document
|
|
that sits outside either drop folder, or for indexing one folder in isolation.
|
|
|
|
* **`*.pdf`** anywhere under a root → one `pages` row per page, text from
|
|
`pymupdf`'s `page.get_text()`.
|
|
* **`*.md` under a folder named `notes`** → one `notes` row each. Deliberately
|
|
*not* every `.md` under the root: the ledger and the runbook are bookkeeping
|
|
*about* the library rather than content *out of* it, and mixing them into the
|
|
same ranked list would answer "which document says X" with a row from the
|
|
index of documents.
|
|
|
|
A PDF that cannot be read — corrupt, encrypted — is **named on stdout and not
|
|
recorded at all**, so the next build retries it instead of treating it as
|
|
already known.
|
|
|
|
---
|
|
|
|
## Incremental, and safe to run at any time
|
|
|
|
`build` is idempotent. Everything is hashed on every run (that is the identity,
|
|
and it is cheap); only *new* hashes are opened and text-extracted, which is the
|
|
expensive part. Notes are compared by content hash too.
|
|
|
|
Measured on a full library: **1.4 s** for a first build, **0.05 s** for a re-run
|
|
with nothing changed. That matters because notes are being written into the
|
|
library while builds are running — a build is expected to be re-run constantly
|
|
and to cost nothing when there is nothing to do.
|
|
|
|
Removal is **confined to the roots the run was given**. A build named with one
|
|
`--root` will not delete the rows of a folder it was not asked to look at;
|
|
otherwise every narrowed re-run would silently gut the index. Rows for files
|
|
that really have vanished from a scanned root are deleted and the count is
|
|
printed.
|
|
|
|
`--rebuild` throws the tables away first. An index written by an older schema
|
|
version is wiped and rebuilt automatically, with a line saying so — a silent
|
|
wipe and a silent no-op look identical from the outside, so it says which one
|
|
happened.
|
|
|
|
---
|
|
|
|
## Searching
|
|
|
|
Every chunk of your query becomes a quoted FTS5 phrase, so nothing you type is
|
|
read as operator syntax by accident:
|
|
|
|
| you type | FTS5 gets | matches |
|
|
| --- | --- | --- |
|
|
| `shear wall` | `"shear" "wall"` | both words, anywhere on the page |
|
|
| `"shear wall"` | `"shear wall"` | that phrase, in that order |
|
|
| `7.5` | `"7.5"` | 7 and 5 adjacent — a decimal |
|
|
| `f'c` | `"f'c"` | the strength symbol |
|
|
| `#4` | `"#4"` | the bar size, *not* every page with a 4 |
|
|
| `8"` | `"8"""` | the inch dimension |
|
|
| `beam*` | `"beam"*` | prefix |
|
|
|
|
`--raw` hands the string to FTS5 untouched, for the rare query that really
|
|
wants `OR` / `NEAR` / a column filter.
|
|
|
|
Other flags: `--limit N` (default 10), `--kind pages|notes|all`, and
|
|
`--doc X` where `X` is a hash prefix, a ledger id, or a path substring.
|
|
|
|
Each hit prints the relpath, the page number (or the note's ledger id), the
|
|
bm25 score and a `snippet()` excerpt with the matched terms in `[brackets]`.
|
|
**bm25 is negative and lower is better**; results are sorted best-first. Page
|
|
hits and note hits are merged into one ranked list and tagged `[page]` /
|
|
`[note]`.
|
|
|
|
### Two different kinds of nothing
|
|
|
|
The refusals are worded apart on purpose, and the tests assert that neither
|
|
sentence can be printed for the other:
|
|
|
|
| situation | what it prints | exit |
|
|
| --- | --- | --- |
|
|
| ran, matched nothing | `0 hits` | 0 |
|
|
| no database yet | `no index built yet -- run: python tools/libindex.py build` | 1 |
|
|
| database from another schema | `index is from another schema version -- run: ...` | 1 |
|
|
| query FTS5 could not parse | `could not run that query -- ...` plus the expression it tried | 1 |
|
|
| sqlite3 without FTS5 | `this python has no sqlite3 FTS5 ...` | 1 |
|
|
|
|
There is no `LIKE` fallback. A machine whose sqlite cannot do FTS5 gets a
|
|
refusal, not a silently worse index that nobody notices is worse.
|
|
|
|
---
|
|
|
|
## Tokenizer, and why this one
|
|
|
|
```
|
|
unicode61 remove_diacritics 2 tokenchars '#''"'
|
|
```
|
|
|
|
Three characters are promoted to token characters because each one carries
|
|
meaning in this corpus that the default tokenizer throws away:
|
|
|
|
* **`#`** — `#4` is a bar size. Without it, `#4` and a bare `4` are the same
|
|
token and every page with a 4 on it answers a bar-size query.
|
|
* **`'`** — `f'c` is one symbol, not `f` AND `c`.
|
|
* **`"`** — `8"` is an inch dimension, not a bare 8.
|
|
|
|
Plate marks (`P316`) and part numbers need nothing special: unicode61 does not
|
|
split a letter/digit boundary.
|
|
|
|
Three characters were deliberately **left as separators**:
|
|
|
|
* **`.`** — making it a token character would keep `7.5` whole, but it would
|
|
also keep `7.5.` whole at the end of a sentence, and that no longer matches a
|
|
search for `7.5`. Decimals are handled on the *query* side instead: `7.5`
|
|
becomes the phrase `"7" "5"`, which requires adjacency — so it matches `7.5`
|
|
wherever it appears, including before a full stop, and does **not** match a
|
|
page that merely has a 7 and a 5 somewhere on it. Both directions are tested.
|
|
* **`-`** — hyphens join words far more often than they join marks; as a token
|
|
character, a search for `wall` would stop finding `wall-panel`.
|
|
* **`_`** — same reasoning: leaving it a separator is what lets a search for
|
|
`anchor` find `ANCHOR_FACE_4T`.
|
|
|
|
`remove_diacritics 2` is the Unicode-aware folding, so an accented spelling in a
|
|
European datasheet matches its unaccented form. CJK is indexed as-is by
|
|
unicode61 and is searchable; it is not word-segmented, so a Chinese query works
|
|
best as the exact run of characters used in the source.
|
|
|
|
The quoting of that tokenizer string in `libindex.py` is fussy and easy to break
|
|
— an SQL double-quoted string containing an FTS5 single-quoted string, each with
|
|
its own quote character doubled. FTS5 rejects `[...]` quoting there whatever
|
|
the SQL documentation suggests about dequoting. If you change the tokenchars,
|
|
run the tokenizer tests; they cover exactly the four shapes above.
|
|
|
|
---
|
|
|
|
## Its output is gitignored material
|
|
|
|
`libindex.py` prints the verbatim text of documents that name producers, jobs
|
|
and clients. **The tool's source never does, and its output must not be pasted
|
|
into a tracked file** — `CLAUDE.md` §3 has the placeholders (`sample_client`,
|
|
`sample_job`). The same rule as `tools/library.py`, for the same reason.
|
|
|
|
---
|
|
|
|
## Tests
|
|
|
|
`WillTools.extension/tests/test_libindex.py`, beside the tests for the other
|
|
host-CPython tools. Every PDF is built with `pymupdf` into `tmp_path`; nothing
|
|
reads the real library and nothing writes into the working directory. What is
|
|
covered is the tokenizer decision (each shape as its own assertion), the
|
|
incremental behaviour (unchanged / moved / changed / vanished / duplicate /
|
|
root confinement), and the wording of the two different kinds of nothing.
|