will/willtools-demo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
52d94303bcc061d236b6dccc339b946ad9941fc8
willtools-demo/tools/knowledge_drift.py
T
will 52d94303bc
ci / test (push) Has been cancelled
Details
WillTools: precast shop-ticket QC and panel handling tools 
Public demo snapshot of an in-house pyRevit extension for precast concrete
shop-ticket QC (37 checks / 46 checklist rows) and a standalone panel
handling solver (PCI Chapter 8).

- No client, job, plant or product data: knowledge layers ship as
  structural templates only.
- Ticket QC's model-read escalation targets a LAN model server endpoint
  (configurable), not a cloud provider.
- Built from the private development repo via tools/make_dist.py.
2026-09-03 16:33:47 -05:00

1057 lines
42 KiB
Python
Raw Blame History

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Compare the knowledge keys that are DECLARED against the ones that are READ.
python tools/knowledge_drift.py # findings, exit 1 if any
python tools/knowledge_drift.py --all # + every suppressed line
python tools/knowledge_drift.py --blind # only the blind-spot report
python tools/knowledge_drift.py --json # machine readable
WHY THIS EXISTS
---------------
The tool's behaviour is driven by layered YAML (knowledge.DEFAULTS ->
catalogue.yaml -> plant -> job -> project config) read through
`knowledge.get(know, 'dotted.path', default)`. Nothing anywhere compared the
two sides of that contract, and seven separate drifts were found in one day by
accident. Every one looked normal from the outside:
* a key written under the wrong parent -- parsed cleanly, did nothing
* a key documented in a job file with a careful comment, read by no code
* a value read by the AI prompt builder and by no check
* a value written, tested, and never wired to the check it was for
They are one missing mechanism, not seven bugs. A key that is declared and
never read is a documented promise the tool does not keep. A key that is read
and declared nowhere is a default nobody can see or override.
WHAT "READ" MEANS HERE, AND THE THING THAT MATTERS MOST
------------------------------------------------------
A reader is graded, not counted. `rails.left_rail_is_fixed` HAD a reader --
the AI prompt builder -- and was still drift, because no check consumed it, so
a plant that set it was silently ignored. `weight_three_way` had a reader in
the test suite. So every call site is put in a tier:
behaviour a check, a solver, an export -- the tool acts on it
surface the config screen, the config dump, the AI prompt. These
DISPLAY a value. A key only they read is settable, visible,
and inert.
test only the suite touches it
and a key whose best reader is `surface` or `test` is reported, separately,
with that word attached.
THE DISCIPLINE: "could not judge" MUST NOT READ AS "judged and found nothing"
----------------------------------------------------------------------------
A path built from a variable, a subtree fetched and walked by hand, a key
reached through a helper -- this detector cannot see all of those. Every run
therefore ends with a blind-spot report naming the call sites it could not
resolve and the subtrees it has deliberately stopped judging. A clean
findings list on top of a long blind-spot list is not a clean codebase, and
the output says so in those words.
CONSTRAINTS THIS RESPECTS
-------------------------
Host CPython 3.12. The code it reads -- the tree under CODE_ROOTS -- is
IronPython 2.7 and CPython, so THAT is only ever PARSED (`ast`), never
imported: importing `vibe` for real pulls in the Revit API and dies. YAML is
read with PyYAML, not `vibe.yamlkit` (a hand-rolled subset for design-spec
files that would silently parse a fraction of a knowledge file). The real
plant and job layers are gitignored: this runs, and is useful, when they are
absent, and it never prints a filename or a value from them -- only key
paths, under the generic labels 'plant layer' / 'job layer'.
One narrow exception to "never imports": `vibe.plantpath` and `vibe.joblocate`
have no Revit import at all -- see their own module docstrings -- so they are
loaded, via the same `sys.modules['vibe']` stub `tools/snapshot_cycle.py` and
`WillTools.extension/tests/conftest.py` use, purely to ask WHERE the plant and
job layers live on THIS machine (`_plant_root` / `_jobs_root` below). That is
resolution-rule reuse, not a parallel guess at the same question, and it is
the fix for the gap `machine-setup.md` in the private repo used to flag as
"a recommendation, not implemented": WILLTOOLS_KNOWLEDGE and WILLTOOLS_JOBS
now say where to look, the same as every other reader of these layers.
"""
from __future__ import annotations
import argparse
import ast
import io
import json
import os
import re
import sys
import types
try:
import yaml
except ImportError: # pragma: no cover
sys.stderr.write('PyYAML is required: pip install pyyaml\n')
raise SystemExit(2)
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KNOWLEDGE_PY = os.path.join(ROOT, 'WillTools.extension', 'lib', 'vibe',
'knowledge.py')
CONFIGMODEL_PY = os.path.join(ROOT, 'WillTools.extension', 'lib', 'vibe',
'configmodel.py')
CATALOGUE = os.path.join(ROOT, 'knowledge', 'catalogue.yaml')
ALLOWLIST = os.path.join(ROOT, 'tools', 'knowledge_drift_allow.txt')
# `vibe.plantpath` / `vibe.joblocate` only -- see the docstring above. Neither
# imports the Revit API, so the stub is enough to make `from vibe import
# plantpath` resolve as an ordinary submodule instead of running
# `vibe/__init__.py` (which does, and dies on host CPython).
_VIBE_DIR = os.path.join(ROOT, 'WillTools.extension', 'lib', 'vibe')
if 'vibe' not in sys.modules:
_stub = types.ModuleType('vibe')
_stub.__path__ = [_VIBE_DIR]
_stub.__file__ = os.path.join(_VIBE_DIR, '__init__.py')
sys.modules['vibe'] = _stub
from vibe import joblocate # noqa: E402
from vibe import plantpath # noqa: E402
CODE_ROOTS = ('WillTools.extension', 'panellift', 'tools')
SKIP_DIRS = {'__pycache__', '.git', 'dist', '_archive', '_scratch',
'.pytest_cache', 'node_modules'}
# The name a module is bound to when it is the knowledge module.
KNOWLEDGE_ALIASES = {'K', 'knowledge', 'kn'}
CATEGORIES = ('unread', 'undeclared', 'defaults', 'surface-only', 'test-only')
# --------------------------------------------------------------- reader tiers
def tier_of(relpath):
"""behaviour / surface / test, from where the call site lives.
The split is the whole point of the tool, so it is stated as data here
rather than being inferred. `configdump`, `configmodel` and the Project
Config screen render settings. None of
them makes the tool DO anything, so a key only they read is a key a plant
can set and be ignored on -- which is exactly the shape of the
`rails.left_rail_is_fixed` drift.
"""
p = relpath.replace('\\', '/')
if '/tests/' in p or os.path.basename(p).startswith('test_') \
or os.path.basename(p) == 'conftest.py':
return 'test'
for mark in ('lib/vibe/configdump.py', 'lib/vibe/configmodel.py',
'ProjectConfig.pushbutton'):
if mark in p:
return 'surface'
return 'behaviour'
TIER_RANK = {'behaviour': 0, 'surface': 1, 'test': 2}
# ------------------------------------------------------------- path flattening
def flatten(node, prefix=''):
"""Every dotted path in a nested mapping, containers included.
Containers are emitted as well as leaves because a container IS a
read target here -- `strands.area_in2` is fetched whole and walked --
so the reader side and the declaration side have to speak about the
same set of names.
"""
out = []
if isinstance(node, dict):
for key, value in node.items():
if not isinstance(key, str):
continue
path = prefix + '.' + key if prefix else key
out.append(path)
out.extend(flatten(value, path))
return out
# ------------------------------------------------------------- declaration side
def _literal(node):
try:
return ast.literal_eval(node)
except Exception:
return None
def declarations():
"""{path: set(source labels)} plus the raw trees, keyed by source.
Sources are labels, never paths: the plant and job layers are gitignored
and a colleague's clone has neither, so nothing that identifies a file --
or a client -- may reach the output.
"""
decls = {}
trees = {}
def add(label, tree):
if not isinstance(tree, dict):
return
trees.setdefault(label, {})
_deep_update(trees[label], tree)
for path in flatten(tree):
decls.setdefault(path, set()).add(label)
# 1. DEFAULTS / STARTER / PLANT_STARTER, parsed out of knowledge.py.
src = _read(KNOWLEDGE_PY)
if src is not None:
mod = ast.parse(src)
for stmt in mod.body:
if not isinstance(stmt, ast.Assign):
continue
names = [t.id for t in stmt.targets if isinstance(t, ast.Name)]
if 'DEFAULTS' in names:
add('knowledge.DEFAULTS', _literal(stmt.value))
elif 'STARTER' in names or 'PLANT_STARTER' in names:
text = _literal(stmt.value)
if isinstance(text, str):
add(names[0], _safe_yaml(text, names[0]))
# 2. The public catalogue.
if os.path.isfile(CATALOGUE):
add('catalogue.yaml', _safe_yaml(_read(CATALOGUE), 'catalogue.yaml'))
# 3. The private layers, if this checkout has them.
for label, files in (('plant layer', _plant_files()),
('job layer', _job_files())):
for path in files:
add(label, _unwrap_config(_safe_yaml(_read(path), label)))
return decls, trees
def _deep_update(base, override):
for key, value in (override or {}).items():
if isinstance(value, dict) and isinstance(base.get(key), dict):
_deep_update(base[key], value)
else:
base[key] = value
return base
def _unwrap_config(tree):
"""A project config file wraps its knowledge under `knowledge:`.
Mirrors `knowledge._merge_config_file`. The sibling sections describe the
configuration rather than being part of it, so merging the document whole
would plant junk branches and then report every one of them as unread.
"""
if isinstance(tree, dict) and isinstance(tree.get('knowledge'), dict) \
and ('project' in tree or 'output_locations' in tree):
return tree['knowledge']
return tree
def _plant_root():
"""Where this run looks for the plant layer(s).
WILLTOOLS_KNOWLEDGE wins when it is set, resolved the SAME way every
other reader resolves it -- `vibe.plantpath.plant_root()`, reused rather
than re-derived, so a machine whose env var points at a private clone
somewhere else on disk (the setup `machine-setup.md` in the private repo
documents) is walked from the folder the real tool would actually read.
Unset, this is `<repo>/knowledge/plant` -- NOT `plantpath`'s own
unset-fallback of `%APPDATA%/WillTools`, on purpose: that fallback names
a per-machine install folder that has nothing to do with auditing a
checkout and will not even exist on a CI runner. `<repo>/knowledge/plant`
is where a private clone has always lived when nobody has set the env
var, and is exactly what the hardcoded path here used to be.
"""
if os.environ.get(plantpath.PLANT_ENV):
return plantpath.plant_root()
return os.path.join(ROOT, 'knowledge', 'plant')
def _jobs_root():
"""Where this run looks for job layers -- WILLTOOLS_JOBS, or the checkout
default. Same reasoning as `_plant_root`, and the same env var every
other reader of the job layer (`designspec.root_dir`, `vibe.joblocate`)
honours; unlike the plant file, WILLTOOLS_JOBS already names a folder
directly, so there is no file-vs-folder resolution to reuse here."""
return os.environ.get(joblocate.JOBS_ENV) or os.path.join(
ROOT, 'knowledge', 'job')
def _plant_files():
"""Every plant knowledge file this run can see.
WALKED, not listed, because the plant layer is ONE FOLDER PER PLANT:
`<root>/<plant>/knowledge.yaml`, with the older single
`<root>/knowledge.yaml` still supported. Listing only the top directory
stopped finding anything the day the folders appeared, and the failure
was quiet in the worst way -- the run still succeeded, the layer count
silently dropped to four, every key declared only by a plant became
"never declared", and five allowlist entries were reported STALE. A
checker that stops checking and says it is fine is the thing this file
exists to prevent, so it is walked and the layer count is printed.
`.bak-*` snapshots are skipped for the same reason `_job_files()` skips
them: yesterday's file would resurrect a key that was removed on purpose.
"""
out = []
base = _plant_root()
if not os.path.isdir(base):
return out
for dirpath, dirnames, filenames in os.walk(base):
dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS]
for name in sorted(filenames):
if name.endswith(('.yaml', '.yml')) and '.bak-' not in name:
out.append(os.path.join(dirpath, name))
return out
def _job_files():
"""Job knowledge and project-config files, from any job folder this run
can see -- see `_jobs_root` for where that is.
`.bak-*` snapshots are skipped: they are yesterday's file, and a key that
was removed on purpose would come back as a finding forever.
"""
out = []
base = _jobs_root()
if not os.path.isdir(base):
return out
for dirpath, dirnames, filenames in os.walk(base):
dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS]
for name in sorted(filenames):
if name.endswith(('.yaml', '.yml')) and '.bak-' not in name:
out.append(os.path.join(dirpath, name))
return out
PARSE_NOTES = []
def _safe_yaml(text, label):
if not text:
return {}
try:
return yaml.safe_load(text) or {}
except Exception as exc:
PARSE_NOTES.append('%s did not parse as YAML (%s) -- its keys are '
'NOT in this run' % (label, exc.__class__.__name__))
return {}
def _read(path):
try:
with io.open(path, 'r', encoding='utf-8') as fh:
return fh.read()
except Exception:
return None
# ------------------------------------------------------------------ reader side
class Reader(object):
__slots__ = ('path', 'default', 'file', 'line', 'tier', 'how', 'covering')
def __init__(self, path, default, file, line, how, covering=True):
self.path = path
self.default = default
self.file = file
self.line = line
self.tier = tier_of(file)
self.how = how
# A COVERING read fetches the node and everything under it -- a real
# knowledge.get -- so nothing below it can be called unread. A config
# row or a subtree caption names the node without fetching its
# children, so it must not silence them: `parameters` appearing on the
# config screen says nothing about whether `parameters.sheet.weight`
# is used by anything.
self.covering = covering
@property
def where(self):
return '%s:%d' % (self.file, self.line)
# One-level wrappers around knowledge.get whose path is a literal prefix plus a
# caller-supplied key. Resolving them by hand is a judgement call: the
# alternative is to treat the whole prefix as opaque, which would hide every
# nomenclature and parameter key from the checker. Each entry names the arg
# index that carries the tail.
HELPERS = {
'_nomen': ('nomenclature.', 1), # designspec._nomen
'param_names': ('parameters.', 1), # knowledge.param_names
}
# Tuples of role names that are fed to a wrapper in a loop rather than one at a
# time. knowledge.unmapped_roles() walks PARAMETER_ROLES and calls
# param_names on each, so every role in it is genuinely read -- but by a
# variable, which the AST scan cannot follow.
ROLE_TUPLES = {'PARAMETER_ROLES': 'parameters.', 'OPTIONAL_ROLES': 'parameters.'}
# Module-level literals in the SURFACE layer that name whole knowledge
# subtrees to display. `configdump.SECTION_ORDER` and `configmodel.SUBTREES`
# are how a branch reaches the config dump and the config screen, and without
# them a branch shown on both would be reported as read by nothing at all --
# which is a different, and wrong, statement about it. Every string in the
# literal is taken; the ones that are not declared paths are dropped.
SURFACE_TABLES = {
('lib/vibe/configmodel.py', 'SUBTREES'),
}
# `configdump.SECTION_ORDER` / `SECTION_NOTES` are deliberately NOT here.
# They order and caption whole branches; treating them as reads made every
# top-level branch look read and swallowed the real findings underneath.
# Display order is not consumption.
class Scan(ast.NodeVisitor):
def __init__(self, relpath):
self.rel = relpath
self.readers = []
self.blind = []
# knowledge.py calls its own `get` bare, with no module prefix, and
# that is where is_strand / catalogue_item / param_names live -- so
# skipping bare calls would report the families and lifters tables as
# unread when they are the most-read keys in the tool. Only inside
# that one file, where `get` cannot mean anything else.
self.bare_get = relpath.endswith('lib/vibe/knowledge.py')
self.fn_stack = []
self.locals_stack = []
def visit_FunctionDef(self, node):
self.fn_stack.append(node.name)
self.locals_stack.append(_local_strings(node))
self.generic_visit(node)
self.locals_stack.pop()
self.fn_stack.pop()
visit_AsyncFunctionDef = visit_FunctionDef
def _resolve_name(self, node):
"""A local variable holding one or two string literals, or None.
`liftexport._fast_box` picks between two literal paths with a ternary
and passes the result to `knowledge.get`. Refusing to follow that
would report both `handling.fast_lifter_box` keys as read only by the
config screen -- a false accusation, and the worst kind, because it
is indistinguishable from the true ones.
"""
if not isinstance(node, ast.Name):
return None
for scope in reversed(self.locals_stack):
if node.id in scope:
return scope[node.id]
return None
def visit_Call(self, node):
self._knowledge_get(node)
self._helper(node)
self._config_row(node)
self.generic_visit(node)
# knowledge.get(know, 'a.b', default) -------------------------------
def _knowledge_get(self, node):
fn = node.func
if isinstance(fn, ast.Attribute) and fn.attr == 'get':
base = fn.value
name = base.id if isinstance(base, ast.Name) else (
base.attr if isinstance(base, ast.Attribute) else None)
if name not in KNOWLEDGE_ALIASES:
return
elif self.bare_get and isinstance(fn, ast.Name) and fn.id == 'get':
pass
else:
return
if len(node.args) < 2:
return
arg = node.args[1]
default = (_unparse(node.args[2]) if len(node.args) > 2 else 'None')
if isinstance(arg, ast.Constant) and isinstance(arg.value, str):
self.readers.append(Reader(arg.value, default, self.rel,
node.lineno, 'literal'))
elif isinstance(arg, ast.BinOp) and isinstance(arg.op, ast.Add) \
and isinstance(arg.left, ast.Constant) \
and isinstance(arg.left.value, str):
if self.fn_stack and self.fn_stack[-1] in HELPERS:
# This IS the body of a wrapper whose callers are resolved
# above. Blinding its own prefix here would hide every
# nomenclature and parameter key behind the very helper that
# makes them visible.
return
self.blind.append((self.rel, node.lineno, 'concatenated path',
arg.left.value + '<var>'))
else:
resolved = self._resolve_name(arg)
if resolved:
for path in sorted(resolved):
self.readers.append(Reader(path, default, self.rel,
node.lineno, 'local literal'))
else:
self.blind.append((self.rel, node.lineno, 'non-literal path',
_unparse(arg)))
# _nomen(know, 'design_folder', ...) / param_names(know, 'sheet.fc') --
def _helper(self, node):
fn = node.func
name = fn.id if isinstance(fn, ast.Name) else (
fn.attr if isinstance(fn, ast.Attribute) else None)
if name not in HELPERS:
return
prefix, idx = HELPERS[name]
if len(node.args) <= idx:
return
arg = node.args[idx]
if isinstance(arg, ast.Constant) and isinstance(arg.value, str):
default = (_unparse(node.args[idx + 1])
if len(node.args) > idx + 1 else 'None')
self.readers.append(Reader(prefix + arg.value, default, self.rel,
node.lineno, 'helper ' + name))
else:
self.blind.append((self.rel, node.lineno,
'non-literal key into ' + name,
prefix + _unparse(arg)))
# configmodel._s('handling.rig_mode', ...) --------------------------
def _config_row(self, node):
fn = node.func
name = fn.id if isinstance(fn, ast.Name) else None
if name != '_s' or not node.args:
return
arg = node.args[0]
if isinstance(arg, ast.Constant) and isinstance(arg.value, str):
self.readers.append(Reader(arg.value, 'None', self.rel,
node.lineno, 'config row',
covering=False))
def _local_strings(fn):
"""{local name: set of string literals it can hold}, one level deep.
Only plain `x = 'a'` and `x = 'a' if c else 'b'`. Anything cleverer is
left unresolved on purpose: a wrong resolution is worse than an admitted
blind spot, because it turns into a confident finding about the wrong key.
"""
out = {}
for node in ast.walk(fn):
if not isinstance(node, ast.Assign) or len(node.targets) != 1:
continue
target = node.targets[0]
if not isinstance(target, ast.Name):
continue
value = node.value
got = set()
if isinstance(value, ast.Constant) and isinstance(value.value, str):
got.add(value.value)
elif isinstance(value, ast.IfExp):
for side in (value.body, value.orelse):
if isinstance(side, ast.Constant) and isinstance(side.value,
str):
got.add(side.value)
else:
got = set()
break
if got:
out.setdefault(target.id, set()).update(got)
else:
out[target.id] = set() # reassigned to something opaque
return {k: v for k, v in out.items() if v}
def _surface_tables(tree, rel):
out = []
for stmt in ast.walk(tree):
if not isinstance(stmt, ast.Assign):
continue
for target in stmt.targets:
if not isinstance(target, ast.Name):
continue
if not any(rel.endswith(f) and target.id == v
for f, v in SURFACE_TABLES):
continue
for text in _strings_in(_literal(stmt.value)):
out.append(Reader(text, 'None', rel, stmt.lineno,
'surface table ' + target.id,
covering=False))
return out
def _strings_in(value):
if isinstance(value, str):
return [value]
out = []
if isinstance(value, dict):
for key, val in value.items():
out.extend(_strings_in(key))
elif isinstance(value, (list, tuple, set)):
for item in value:
out.extend(_strings_in(item))
return out
def _unparse(node):
try:
return ast.unparse(node)
except Exception:
return '<?>'
def scan_code():
readers, blind, parse_fail = [], [], []
for root in CODE_ROOTS:
base = os.path.join(ROOT, root)
for dirpath, dirnames, filenames in os.walk(base):
dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS]
for name in filenames:
if not name.endswith('.py'):
continue
full = os.path.join(dirpath, name)
rel = os.path.relpath(full, ROOT).replace('\\', '/')
text = _read(full)
if text is None:
continue
try:
tree = ast.parse(text)
except SyntaxError as exc:
# IronPython 2.7 print statements and the like. A file
# that will not parse is a file whose readers are INVISIBLE
# to this tool, so it is reported, never skipped quietly.
parse_fail.append((rel, 'line %s: %s' % (exc.lineno,
exc.msg)))
continue
scan = Scan(rel)
scan.visit(tree)
readers.extend(scan.readers)
blind.extend(scan.blind)
readers.extend(_role_tuples(tree, rel))
readers.extend(_surface_tables(tree, rel))
return readers, blind, parse_fail
def _role_tuples(tree, rel):
"""PARAMETER_ROLES = (...) -> a reader for each 'parameters.<role>'."""
out = []
for stmt in ast.walk(tree):
if not isinstance(stmt, ast.Assign):
continue
for target in stmt.targets:
if not isinstance(target, ast.Name) or target.id not in ROLE_TUPLES:
continue
prefix = ROLE_TUPLES[target.id]
for role in (_literal(stmt.value) or ()):
if isinstance(role, str):
out.append(Reader(prefix + role, 'None', rel,
stmt.lineno, 'role table'))
return out
# ------------------------------------------------------------------- allowlist
def load_allowlist():
"""`<category> <path> -- <reason>`, one per line.
An allowlist FILE rather than a YAML comment, chosen deliberately: two of
the three categories are about keys that exist only in Python or only in a
gitignored file, so a comment convention could not express them and half
the exceptions would have nowhere to live. One file, greppable, in the
repo, next to the tool that reads it. The reason is MANDATORY -- an
exception with no sentence is refused, because an unexplained suppression
is how a checker quietly stops checking -- and every run prints the
suppression count and any entry that no longer matches anything, so the
file cannot rot unseen.
"""
entries, errors = {}, []
text = _read(ALLOWLIST)
if text is None:
return entries, errors
for num, raw in enumerate(text.splitlines(), 1):
line = raw.strip()
if not line or line.startswith('#'):
continue
if ' -- ' not in line:
errors.append('%s:%d needs a reason: "<category> <path> -- why"'
% (os.path.basename(ALLOWLIST), num))
continue
head, reason = line.split(' -- ', 1)
bits = head.split()
if len(bits) != 2 or bits[0] not in CATEGORIES:
errors.append('%s:%d unreadable; category must be one of %s'
% (os.path.basename(ALLOWLIST), num,
', '.join(CATEGORIES)))
continue
if not reason.strip():
errors.append('%s:%d has an empty reason' %
(os.path.basename(ALLOWLIST), num))
continue
entries[(bits[0], bits[1])] = reason.strip()
return entries, errors
# ---------------------------------------------------------------------- analyse
def analyse():
decls, _trees = declarations()
readers, blind, parse_fail = scan_code()
allow, allow_errors = load_allowlist()
# A surface table is a list of branch names mixed with the help text
# beside them. Only the entries that are real declared paths are readers;
# the prose is dropped rather than reported as a hidden default.
readers = [r for r in readers
if not r.how.startswith('surface table') or r.path in decls]
by_path = {}
for r in readers:
by_path.setdefault(r.path, []).append(r)
read_paths = set(by_path)
covering_paths = {p for p, rs in by_path.items()
if any(r.covering for r in rs)}
# A path is covered when it, or any ancestor, is read by a COVERING read:
# fetching `strands.area_in2` whole and walking it reads every size in it.
covered = set()
for path in decls:
parts = path.split('.')
for i in range(1, len(parts) + 1):
head = '.'.join(parts[:i])
if head in covering_paths or (head == path and head in read_paths):
covered.add(path)
break
# Subtrees a dynamic path makes unjudgeable. `K.get(know, 'nomenclature.'
# + key)` means SOME nomenclature key is read and the tool cannot say
# which -- so the prefix is suppressed from findings and named in the
# blind-spot report with the number of keys it hides.
dyn_prefixes = sorted({b[3][:-len('<var>')].rstrip('.')
for b in blind if b[2] == 'concatenated path'})
def under_dyn(path):
return any(path == p or path.startswith(p + '.') for p in dyn_prefixes)
# ---- 1. declared, never read. Report the SHALLOWEST unread node: a
# 40-entry insert table nobody reads is one finding, not 200 lines.
tree = _paths_to_tree(sorted(decls))
unread = []
_collect_unread(tree, '', read_paths, covering_paths, unread)
unread = [p for p in unread if not under_dyn(p)]
dyn_hidden = sorted(p for p in decls
if under_dyn(p) and p not in covered)
# ---- 2. read, never declared.
undeclared = sorted(p for p in by_path
if p not in decls and not _is_prefix_of_decl(p, decls))
# ---- 3. same path, different fallbacks.
inconsistent = []
for path, rs in sorted(by_path.items()):
seen = {}
for r in rs:
# Two kinds of call site are excluded, both because they do not
# actually express an opinion about what "unset" means:
# * a config row or a subtree caption NAMES a key, it does not
# fetch it, so it has no fallback to disagree with;
# * a test that reads a value with no default is asserting what
# is there, not declaring what should happen when it is not.
# Counting either produced findings that were pure artefacts of
# this tool -- the fastest way to teach people to ignore it.
if not r.covering or r.tier == 'test':
continue
seen.setdefault(r.default, []).append(r)
if len(seen) > 1:
inconsistent.append((path, seen))
# ---- 4/5. graded readers.
surface_only, test_only = [], []
behaviour_paths = {p for p, rs in by_path.items()
if any(r.tier == 'behaviour' for r in rs)}
for path, rs in sorted(by_path.items()):
tiers = {r.tier for r in rs}
if 'behaviour' in tiers:
continue
# A BRANCH the config screen names, whose leaves a check does read,
# is not inert -- `parameters` is the example. Reporting the branch
# would be a second, wrong sentence about keys already judged one
# level down.
if any(b.startswith(path + '.') for b in behaviour_paths):
continue
(test_only if tiers == {'test'} else surface_only).append((path, rs))
findings = {
'unread': [(p, sorted(decls[p])) for p in unread],
'undeclared': [(p, sorted({r.where for r in by_path[p]}))
for p in undeclared],
'defaults': inconsistent,
'surface-only': surface_only,
'test-only': test_only,
}
suppressed = {}
for cat, rows in findings.items():
keep, gone = [], []
for row in rows:
if (cat, row[0]) in allow:
gone.append((row[0], allow[(cat, row[0])]))
else:
keep.append(row)
findings[cat] = keep
suppressed[cat] = gone
fired = {(c, r[0]) for c, rows in findings.items() for r in rows}
used = {(c, p) for c, rows in suppressed.items() for p, _why in rows}
stale = sorted(k for k in allow if k not in used and k not in fired)
return {
'findings': findings,
'suppressed': suppressed,
'stale_allow': stale,
'allow_errors': allow_errors,
'blind': blind,
'dyn_prefixes': dyn_prefixes,
'dyn_hidden': dyn_hidden,
'parse_fail': parse_fail,
'parse_notes': list(PARSE_NOTES),
'layers': sorted({lbl for s in decls.values() for lbl in s}),
'counts': {'declared': len(decls), 'read': len(by_path),
'readers': len(readers)},
}
def _is_prefix_of_decl(path, decls):
"""A read path that is a prefix of a declared one is declared enough.
Reading `catalogue.inserts` when only `catalogue.inserts.QLPD4T5WG...`
exists is not a hidden default; it is a table fetched whole.
"""
head = path + '.'
return any(d.startswith(head) for d in decls)
def _paths_to_tree(paths):
tree = {}
for path in paths:
node = tree
for part in path.split('.'):
node = node.setdefault(part, {})
return tree
def _collect_unread(node, prefix, read_paths, covering_paths, out):
for key in sorted(node):
path = prefix + '.' + key if prefix else key
if path in covering_paths:
continue # subtree fetched whole
kids = node[key]
named = path in read_paths # named, but not fetched whole
below = kids and _any_read_below(kids, path, read_paths)
if below:
_collect_unread(kids, path, read_paths, covering_paths, out)
elif named:
# The config screen names this branch and nothing under it is
# read. One finding about the branch (in surface-only) says the
# whole thing; listing every leaf here would say it again, worse.
continue
else:
out.append(path)
def _any_read_below(node, prefix, read_paths):
head = prefix + '.'
return any(p.startswith(head) for p in read_paths)
# ------------------------------------------------------------------- reporting
def _p(out, text=''):
out.write(text + '\n')
def report(res, out, show_all=False, blind_only=False):
c = res['counts']
_p(out, '=' * 72)
_p(out, 'KNOWLEDGE DRIFT')
_p(out, '=' * 72)
_p(out, '%d declared paths across %d layer(s): %s'
% (c['declared'], len(res['layers']), ', '.join(res['layers'])))
_p(out, '%d distinct paths read by %d call sites.'
% (c['read'], c['readers']))
private = [l for l in res['layers'] if l in ('plant layer', 'job layer')]
if not private:
_p(out, 'No private plant or job layer in this checkout -- only the '
'keys that ship in the repo were judged.')
_p(out)
total = 0
if not blind_only:
total = _findings(res, out)
_blind_report(res, out)
if not blind_only:
_p(out)
sup = sum(len(v) for v in res['suppressed'].values())
_p(out, '%d finding(s); %d suppressed by %s.'
% (total, sup, os.path.basename(ALLOWLIST)))
if show_all and sup:
for cat in CATEGORIES:
for path, why in res['suppressed'][cat]:
_p(out, ' [%s] %s -- %s' % (cat, path, why))
if res['stale_allow']:
_p(out, 'STALE allowlist entries (nothing matches them any more; '
'delete them):')
for cat, path in res['stale_allow']:
_p(out, ' %s %s' % (cat, path))
for err in res['allow_errors']:
_p(out, 'ALLOWLIST ERROR: ' + err)
return total
SECTIONS = (
('unread', 'DECLARED BUT NEVER READ',
'A key someone can set that no code looks at. The file says the tool '
'honours it; the tool does not.'),
('undeclared', 'READ BUT NEVER DECLARED',
'A hidden default: nothing in DEFAULTS, the starters, the catalogue or '
'a real layer names it, so nobody can discover or override it.'),
('defaults', 'READ WITH INCONSISTENT DEFAULTS',
'The same path fetched with different fallbacks. Two call sites '
'disagree about what "unset" means, and only one of them can be right.'),
('surface-only', 'READ ONLY BY THE CONFIG SCREEN OR THE PROMPT',
'Settable, displayed, and inert: no check, solver or export consumes '
'it. This is the shape rails.left_rail_is_fixed had.'),
('test-only', 'READ ONLY BY THE TEST SUITE',
'Written and tested but never wired to production code -- the shape '
'weight_three_way had.'),
)
def _findings(res, out):
total = 0
for cat, title, why in SECTIONS:
rows = res['findings'][cat]
total += len(rows)
_p(out, '-' * 72)
_p(out, '%s (%d)' % (title, len(rows)))
_p(out, ' ' + why)
_p(out)
if not rows:
_p(out, ' none')
_p(out)
continue
if cat == 'defaults':
for path, seen in rows:
_p(out, ' %s' % path)
for default, rs in sorted(seen.items()):
_p(out, ' default %-24s %s'
% (default, ', '.join(sorted(r.where for r in rs))))
elif cat in ('surface-only', 'test-only'):
for path, rs in rows:
_p(out, ' %-52s %s'
% (path, ', '.join(sorted(set(
'%s [%s]' % (r.where, r.tier) for r in rs)))))
else:
for path, where in rows:
_p(out, ' %-52s %s' % (path, ', '.join(where)))
_p(out)
return total
def _blind_report(res, out):
_p(out, '=' * 72)
_p(out, 'WHAT THIS TOOL COULD NOT SEE')
_p(out, '=' * 72)
_p(out, 'Read this before treating a short findings list as a clean '
'codebase.')
_p(out, '"Could not judge" is not "judged and found nothing".')
_p(out)
if res['dyn_prefixes']:
_p(out, 'Subtrees reached by a computed path -- SOME key under each '
'is read and')
_p(out, 'this tool cannot say which, so nothing under them is '
'reported:')
for prefix in res['dyn_prefixes']:
hidden = [p for p in res['dyn_hidden']
if p == prefix or p.startswith(prefix + '.')]
_p(out, ' %-28s hides %d declared key(s) from judgement'
% (prefix, len(hidden)))
_p(out)
other = [b for b in res['blind'] if b[2] != 'concatenated path']
if other:
_p(out, 'Call sites whose path is not a literal at all:')
for rel, line, kind, shown in sorted(other):
_p(out, ' %s:%d %s %s' % (rel, line, kind, shown))
_p(out)
if res['parse_fail']:
_p(out, 'Python files that would not parse on CPython 3.12 -- any '
'reader in them is INVISIBLE:')
for rel, msg in res['parse_fail']:
_p(out, ' %s (%s)' % (rel, msg))
_p(out)
for note in res['parse_notes']:
_p(out, 'NOTE: ' + note)
_p(out, 'Shapes this detector does not attempt at all:')
for line in (
'a subtree fetched once and then walked by hand -- the keys inside',
' it are read, but by dict access, not by a path this tool sees;',
'a path assembled from a variable at a call site listed above;',
'a wrapper other than _nomen / param_names / _s hiding a literal path;',
'anything read outside the .py files under %s;' % ', '.join(CODE_ROOTS),
'whether a reader that EXISTS actually changes an outcome -- the',
' tier is a proxy for that, not a proof.',
):
_p(out, (' ' + line) if line.startswith(' ') else (' - ' + line))
_p(out)
_p(out, 'And three kinds of drift that LOOK like this one but are NOT '
'judged here')
_p(out, 'at all, because what they are about is not a knowledge-layer key:')
for line in (
'a Revit shared parameter the model carries that no code reads',
' (TICKET_IS_REINFORCED was one);',
'a field in the panellift JSON job file, or in the reinforcing index',
' record, that nothing interprets -- those are dataclass and dict',
' schemas, not YAML layers;',
'a check that is registered and can never run. A dead check still',
' reads its keys, so every key it touches looks alive from here.',
):
_p(out, (' ' + line) if line.startswith(' ') else (' - ' + line))
_p(out)
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--all', action='store_true',
help='also list every allowlist-suppressed finding')
ap.add_argument('--blind', action='store_true',
help='print only the blind-spot report')
ap.add_argument('--json', action='store_true', help='machine readable')
args = ap.parse_args(argv)
res = analyse()
if args.json:
payload = {
'counts': res['counts'],
'layers': res['layers'],
'findings': {
cat: [row[0] for row in res['findings'][cat]]
for cat in CATEGORIES},
'suppressed': {cat: [p for p, _ in res['suppressed'][cat]]
for cat in CATEGORIES},
'stale_allow': ['%s %s' % k for k in res['stale_allow']],
'allow_errors': res['allow_errors'],
'blind_dynamic_prefixes': res['dyn_prefixes'],
'blind_call_sites': ['%s:%d %s' % (b[0], b[1], b[2])
for b in res['blind']],
'unparseable_files': [f for f, _ in res['parse_fail']],
}
sys.stdout.write(json.dumps(payload, indent=2, sort_keys=True) + '\n')
total = sum(len(v) for v in payload['findings'].values())
else:
total = report(res, sys.stdout, show_all=args.all,
blind_only=args.blind)
if args.blind:
return 0
if res['allow_errors']:
return 2
return 1 if total else 0
if __name__ == '__main__':
raise SystemExit(main())
Reference in New Issue View Git Blame Copy Permalink