willtools-demo/tools/make_dist.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Build a clean, knowledge-free distribution tree for GitHub.

    python tools/make_dist.py OUT_DIR
    python tools/make_dist.py OUT_DIR --force        # overwrite OUT_DIR
    python tools/make_dist.py --scan-only OUT_DIR      # re-run the scrub gate
                                                        # over an existing tree

THE COMMISSION, owner, 2026-09-02 (verbatim): "我们继续更新github，但是得是干净
的版本，不包含任何知识包括公共知识，尽量修复可能出现的安装问题，等等。我要准备
分发给我同事试用了。我也可以跟他们那里收集反馈。" -- a clean distribution, ZERO
knowledge (including the public catalogue), for colleagues to trial, with
installation problems fixed where found. NEVER PUSHES; see the bottom of this
file for the exact commands the owner runs after reviewing what this builds.

WHAT "CLEAN" MEANS HERE, PRECISELY
-----------------------------------
1. Zero PROPRIETARY knowledge. Not `knowledge/catalogue-rationale.md`
   (which has shipped in this repository's own history), and not the two
   real capacity tables inside `knowledge/catalogue.yaml` --
   `catalogue.inserts` (ALP lifting-insert ratings) and
   `connectors.products` (sandwich-wall tie ratings). Those never leave the
   private checkout.

   `knowledge/catalogue.yaml` ITSELF now ships, but not verbatim: see
   "THE CATALOGUE CARVE-OUT" below for exactly what of it does and does
   not, and 2026-09 commission notes (verbatim, owner): "你先做B，把B确认
   做通了以后，再抽掉这些知识... 缺了知识没法跑是对的，但是还是得确认有了
   知识就能跑这件事儿" -- prove the tool runs WITH knowledge before
   shipping a build that has none, so every red in a knowledge-less
   distribution is attributable to knowledge alone, never tangled with a
   real bug. `knowledge/catalogue.yaml.template` still ships too, as a
   blank-value authoring template for a colleague building their OWN
   knowledge layers from scratch -- it is not what this carve-out is built
   from.

THE CATALOGUE CARVE-OUT
------------------------
`build_catalogue_carveout()` reads THIS repo's real `knowledge/catalogue.yaml`
and re-emits four branches VERBATIM (parsed and re-dumped, not hand-typed):
`face_designations` (trade abbreviations printed on every shop ticket
everywhere -- the real file's own header already calls this branch public),
`orientation` (a fact about precast assemblies in general, the owner's own
ruling for why it lives here and not in a plant file), `catalogue.rebar`
(ASTM A706/A615 grade tables -- a published national standard, not a
manufacturer's rating) and `catalogue.safety_factors` (two nulls and a
reason). `checklist.master` -- the shop-ticket QC checklist's row ids,
names, grades and which check function answers each one -- ships in full:
it is this TOOL's own built-in configuration, not knowledge about any job.

`catalogue.inserts` and `connectors.products` ship EMPTY. A handful of
tests read those two tables directly and assert the owner's real published
numbers (a specific insert's factor, a specific tie's edge distance) --
knowledge-content tests, not tests of this tool's logic. See
`append_test_exclusions()` below for how the dist's own test trees skip
them, by name, with the reason, rather than being faked with invented
numbers that would just be the same real knowledge through a back door.
2. Fresh history. This script does not touch this repo's git history at all
   -- it assembles a plain directory tree and commits ONE orphan commit
   inside a brand new repo at OUT_DIR. The 19 versions of the catalogue this
   repo's own history carries never reach GitHub this way.
3. An INCLUDE list, not an exclude list. Every file that reaches OUT_DIR is
   named below, one at a time, read through `git ls-files` scoped to its own
   directory -- never a raw directory walk. That means a stray untracked
   file (a scratch script with a real path pasted into it, the exact thing
   .gitignore's own `_scratch/` comment describes) cannot reach the dist by
   accident the way it could reach a plain `git add -A`: if `git ls-files`
   does not name it, this script never opens it.

WHY EACH TOOLS/ AND DOCS/ FILE IS IN OR OUT: TOOLS_MANIFEST / DOCS_MANIFEST
below, one reason per file, not just "curated". Read them before assuming a
tool's absence is an oversight.

THE SCRUB GATE
---------------
After assembly, `scrub_tree()` runs the SAME regex machinery
`tools/scrub.py` uses -- imported from it, not reimplemented -- over EVERY
file in OUT_DIR, using the REAL `.scrub-terms` list read from THIS repo (
never copied into the dist) plus two more structural patterns this script
adds: the private repo's own host, and the owner's machine paths. A hit
anywhere aborts the build before the git commit is made, no exceptions --
not even in a doc, not even in a comment. See `EXTRA_PATTERNS` below.

WHAT THIS SCRIPT DOES NOT DO
------------------------------
Push, or touch this repository's own git state at all. It also does not
decide whether the GitHub repo should be public or stay private-with-
collaborators -- that is the owner's call, not encoded here.
"""
from __future__ import annotations

import argparse
import datetime
import json
import os
import re
import shutil
import subprocess
import sys

import yaml  # PyYAML -- already a requirements.txt dependency of this suite

ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
TOOLS_DIR = os.path.join(ROOT, 'tools')
sys.path.insert(0, TOOLS_DIR)
import scrub as scrubmod  # noqa: E402  -- the real term list stays in ROOT/.scrub-terms


# ============================================================== tools/ manifest
#
# (filename, include, reason). `include=False` entries are still listed, so
# the decision and its reason are read together rather than left to be
# reconstructed from a diff.
TOOLS_MANIFEST = [
    ('aiextract_run.py', True,
     'host half of Design Extract\'s vision escalation pass; reads its own '
     'model key from the environment and fails with a clear message with '
     'none set -- no owner-specific path or credential baked in'),
    ('deckcheck.py', True,
     'generic frozen-deck-vs-current-knowledge checker; useful once a '
     'colleague has real knowledge layers of their own'),
    ('dsh-here.cmd', False,
     "the script's own header says it: \"Nothing references this file any "
     'more; it is kept for the owner to bin.\" Dead, and named after '
     'owner-only tooling (DeepSeek Harness) besides'),
    ('knowledge_drift.py', True,
     'generic declared-vs-read knowledge-key checker; no owner-specific '
     'paths, degrades cleanly with no plant/job layers'),
    ('knowledge_drift_allow.txt', True,
     'suppression list paired with knowledge_drift.py; every entry is a '
     'dotted key name and a reason, no client data'),
    ('knowledgelint.py', True,
     'REQUIRED reading after any knowledge-layer edit; fixed in this pass '
     'to report a knowledge-less checkout as SKIP rather than FAIL (see '
     'tools/knowledgelint.py\'s own diff)'),
    ('layersplit.py', False,
     'splits long rationale comments out of a knowledge layer the OWNER '
     'authors, into catalogue-rationale.md-shaped files; a trial install '
     'with no real knowledge layers has nothing to run this on'),
    ('libindex.py', False,
     "indexes knowledge/library/, the owner's private PDF drop folder, "
     'which never ships -- named explicitly in the commission as an '
     'example exclusion'),
    ('library.py', False,
     'lists knowledge/library/ against its ledger -- same folder as '
     'libindex.py above, same reason, also named explicitly'),
    ('make_icons.py', True,
     'generic ribbon-icon generator; optional Pillow dependency is its own '
     'business, nothing else needs it'),
    ('package.py', False,
     'superseded by tools/make_dist.py for anything leaving this machine, '
     'and unconditionally zips the WHOLE knowledge/ tree (its own '
     '`INCLUDE` list names it) -- exactly what must never leave the '
     "owner's machine as real data. Stays in the owner's own working repo, "
     'not in a trial dist'),
    ('panelanalysis.py', True,
     'generic Eriksson-Wall-shaped demo report over a job.json deck, no '
     'owner-specific data; a good second thing to try after the example '
     'deck'),
    ('qna_watch.py', False,
     'watches QNA/, which is excluded from this distribution wholesale '
     '(see .gitignore\'s own comment on it) -- nothing for this to watch'),
    ('reinforce_offline.py', True,
     'generic reinforcing-suggestion CLI reading a --index the colleague '
     'builds themselves; no owner data baked in'),
    ('rvt.py', True, "the live-Revit bridge client -- generic, and it's the "
     'whole point of the GOD_MOD button'),
    ('scrub.py', True,
     'ships despite being infrastructure-flavored: CI '
     '(.github/workflows/ci.yml), the pre-commit hook, and CONTRIBUTING.md '
     'all call it, and every code path already degrades cleanly with no '
     'real .scrub-terms or CLAUDE.md present (empty term list -> '
     'structural-patterns-only scan, missing CLAUDE.md -> no placeholder '
     'exemption, both by design, see the module docstring and '
     'load_placeholders()). What does NOT ship is the real .scrub-terms '
     'list itself -- see EXCLUDE_ALWAYS below.'),
    ('setup_machine.py', True,
     'updated in this pass to make a knowledge-less install a first-class, '
     'PASS-shaped state (--no-knowledge / auto-detect) -- the doctor a '
     'colleague runs first'),
    ('snapshot_cycle.py', True, 'generic Revit session cycling, no owner paths'),
    ('snapshot_defer.py', True, 'host half of snapshot_cycle.py, same reason'),
    ('sweep.py', True, 'stays -- named explicitly in the commission'),
    ('sweep_in_revit.py', True, 'stays -- paired with sweep.py above'),
    ('ticket_delta.py', True,
     'generic PDF ticket-revision diff tool (pymupdf, already a dependency)'),
    ('vision_read.py', True,
     "Ticket QC's vision-escalation CPython half; reads ANTHROPIC_API_KEY "
     'from the environment and fails cleanly with none set'),
    ('make_dist.py', False, 'this script -- builds a dist, is not one'),
]

# ================================================================ docs/ manifest
DOCS_MANIFEST = [
    ('architecture-migration.md', True,
     'clean architecture/design rationale, no client or job data'),
    ('checklist-code-map.md', True,
     'mechanical id-to-code reference for the ticket checklist, no client data'),
    ('handoff.md', False,
     'written "for an agent picking this repo up cold" -- points at '
     'CLAUDE.md (excluded) and carries open engineering questions tied to '
     'real sweep findings, not colleague documentation'),
    ('interference.md', True, 'clean technical explanation of the clash check'),
    ('knowledge-override.md', True,
     'generic layering-precedence rule, useful once a colleague writes '
     'their own plant/job layers'),
    ('library-index.md', False,
     'documents tools/libindex.py, which is excluded from this distribution'),
    ('lift-check.md', True, 'user-facing explanation of Panel Lift Check'),
    ('panellift-ewf-alignment.md', True,
     'clean mapping doc; references a private sample-data folder a '
     'colleague will not have, but names no client or job'),
    ('plan-panellift-three-stage.md', True,
     'design doc, explicitly named in the commission as clean and worth '
     'shipping'),
    ('plan-strands-auto-layout.md', True, 'same as above'),
    ('plan-ticketqc-refactor.md', True, 'same as above'),
    ('revit-api-cheatsheet.md', True, 'generic Revit 2025 API patterns'),
    ('revit-bridge-context.md', True, 'generic explanation of the rvt.py bridge'),
    ('snapshot-cycle.md', True, 'generic explanation of the snapshot tool'),
    ('std-clash-all.md', False,
     "a raw findings table from a real job's clash run, not reference "
     'material -- job data, not documentation'),
    ('std-clash-judgement.md', False, 'same reason as std-clash-all.md'),
    ('std-nonclash.md', False, 'same reason as std-clash-all.md'),
    ('takeoff.md', True, 'user-facing explanation of the Takeoff button'),
    ('ticket-audit-wp1-12.md', False,
     'an internal audit trail over twelve real tickets, mid-revision; the '
     'durable lessons from it already live in ticket-checklist.md and '
     'ticket-knowledge-deltas.md below'),
    ('ticket-check.skill.md', True, 'user-facing skill doc for Ticket QC'),
    ('ticket-checklist.md', True,
     'durable, generic checklist reference; piece marks it cites (WP194 '
     'etc.) are design-number-shaped naming, not client information, per '
     "CLAUDE.md's own placeholder table"),
    ('ticket-knowledge-deltas.md', True,
     'generic lessons on what belongs at which knowledge layer'),
    ('ticketqc-vision.skill.md', True, 'user-facing skill doc for the vision escalation'),
    ('todo-units.md', True, 'clean roadmap note, no client or job data'),
]

# ============================================================ top-level files
# Copied verbatim from the working tree. README.md and INSTALL.md are the
# versions already rewritten in this repo's own working tree for this
# audience -- see the commission's part C.
TOP_LEVEL_FILES = [
    'requirements.txt',
    'INSTALL.md',
    'README.md',
    'CONTRIBUTING.md',
    '.setup-opts.example',
]

# Whole directories, tracked files only (via `git ls-files`), no curation.
WHOLE_DIRS = ['WillTools.extension', 'panellift']

# Files inside a whole-copied directory that still do not ship, because the
# TOOL they test does not. Paired 1:1 with a False row in TOOLS_MANIFEST.
TEST_EXCLUDES = {
    os.path.join('WillTools.extension', 'tests', 'test_libindex.py'):
        'tests tools/libindex.py, which is excluded (see TOOLS_MANIFEST)',
}

# Never copied, whatever else changes -- named here once so a reviewer can
# see the whole "never" list in one place rather than infer it from what is
# absent.
EXCLUDE_ALWAYS_NOTE = """\
Deliberately never touched by this script (no rule below CAN reach them,
because copying is include-list-driven off `git ls-files`, never a directory
walk):
  knowledge/         except README.md, catalogue.yaml.template, and a
                     catalogue.yaml written FRESH below by
                     build_catalogue_carveout() -- never copied from the
                     real file. catalogue-rationale.md, and everything
                     gitignored under it (plant/, job/, library/, MC/,
                     _archive/, building code/) all stay out. The carve-out
                     omits catalogue.inserts and connectors.products (real
                     manufacturer capacities) even though it reads every
                     other branch of the real file -- see "THE CATALOGUE
                     CARVE-OUT" above.
  QNA/               agent task inbox, gitignored except its own README.md.
  CLAUDE.md          the owner's own agent operating manual -- Chinese
                     commission quotes, real measured incidents, not
                     colleague documentation.
  PROJECT.md         owner's own project log.
  AGENTS.md          owner's own agent roster notes.
  .scrub-terms       the real client/job denylist -- .scrub-terms.example
                     ships in its place.
  .setup-opts        this machine's own declared install choices.
  sample data/, *.pdf, *.log, _scratch/, .git/, .pytest_cache/
"""


# ================================================================ scrub gate
#
# The private repo's own host and the owner's machine paths, added on top of
# tools/scrub.py's own STRUCTURAL list (which already flags any
# `C:\\Users\\<name>` path generically -- this adds the two more specific
# strings named in the commission).
EXTRA_PATTERNS = [
    (re.compile(re.escape('git.will123song.xyz'), re.I), 'private repo host'),
    (re.compile(r'\bV:\\', re.I), "owner's project drive path"),
]


def scrub_tree(dist_dir):
    """Every hit tools/scrub.py's own machinery would flag, plus
    EXTRA_PATTERNS, scanned over EVERY file under `dist_dir`. Returns a list
    of hit strings; empty means clean.

    Reuses `scrub.load_terms`, `scrub.load_placeholders`, `scrub.patterns`
    and `scrub.FILENAME_CATEGORIES` / `scrub._is_placeholder_filename`
    directly rather than reimplementing the regex or the placeholder
    exemption -- one definition of "a hit", used by the pre-commit hook, CI,
    and this build gate alike.
    """
    terms = scrubmod.load_terms(os.path.join(ROOT, '.scrub-terms'))
    placeholders = scrubmod.load_placeholders(os.path.join(ROOT, 'CLAUDE.md'))
    pats = scrubmod.patterns(terms) + EXTRA_PATTERNS

    hits = []
    for dirpath, dirnames, filenames in os.walk(dist_dir):
        dirnames[:] = [d for d in dirnames
                       if d not in ('.git', '__pycache__', '.pytest_cache')]
        for fn in sorted(filenames):
            path = os.path.join(dirpath, fn)
            rel = os.path.relpath(path, dist_dir)
            try:
                with open(path, encoding='utf-8', errors='replace') as fh:
                    lines = fh.readlines()
            except OSError:
                continue
            for n, line in enumerate(lines, 1):
                for pat, why in pats:
                    m = pat.search(line)
                    if not m:
                        continue
                    if (why in scrubmod.FILENAME_CATEGORIES and
                            scrubmod._is_placeholder_filename(m.group(0), placeholders)):
                        continue
                    hits.append('{0}:{1}: {2} -- {3}'.format(
                        rel, n, why, m.group(0)[:80]))
    return hits


# ============================================================= assembly helpers

def git(*args):
    return subprocess.run(['git'] + list(args), cwd=ROOT, check=False,
                          capture_output=True, text=True, errors='replace')


def tracked_under(rel_dir):
    """Tracked files under `rel_dir`, forward-slash relative paths, via
    `git ls-files` -- never a raw walk. See the module docstring for why."""
    out = git('ls-files', '--', rel_dir).stdout
    return [line for line in out.splitlines() if line]


def _copy(rel_src, dest_root):
    src = os.path.join(ROOT, rel_src)
    dst = os.path.join(dest_root, rel_src)
    os.makedirs(os.path.dirname(dst), exist_ok=True)
    shutil.copy2(src, dst)


def copy_top_level(dest_root):
    for name in TOP_LEVEL_FILES:
        src = os.path.join(ROOT, name)
        if not os.path.isfile(src):
            raise SystemExit('make_dist: expected top-level file missing: {0} '
                             '(part C of the commission writes this)'.format(name))
        _copy(name, dest_root)


def copy_whole_dirs(dest_root):
    for rel_dir in WHOLE_DIRS:
        for rel in tracked_under(rel_dir):
            if rel.replace('/', os.sep) in TEST_EXCLUDES:
                continue
            _copy(rel.replace('/', os.sep), dest_root)


def copy_curated(dest_root, subdir, manifest):
    tracked = set(tracked_under(subdir))
    named = set(name for name, _, _ in manifest)
    on_disk = set(os.path.basename(p) for p in tracked)
    missing_from_manifest = on_disk - named
    if missing_from_manifest:
        raise SystemExit(
            'make_dist: {0}/ has file(s) with no manifest entry, refusing '
            'to guess whether they ship: {1}'.format(
                subdir, ', '.join(sorted(missing_from_manifest))))
    for name, include, _reason in manifest:
        if not include:
            continue
        rel = '{0}/{1}'.format(subdir, name)
        if rel not in tracked:
            raise SystemExit(
                'make_dist: manifest names {0} but it is not a tracked file '
                'under {1}/ -- manifest is stale'.format(name, subdir))
        _copy(rel.replace('/', os.sep), dest_root)


def write(dest_root, rel_path, content):
    dst = os.path.join(dest_root, rel_path)
    os.makedirs(os.path.dirname(dst), exist_ok=True)
    with open(dst, 'w', encoding='utf-8', newline='\n') as fh:
        fh.write(content)


def copy_knowledge_stub(dest_root):
    for name in ('README.md', 'catalogue.yaml.template'):
        src = os.path.join(ROOT, 'knowledge', name)
        if not os.path.isfile(src):
            raise SystemExit('make_dist: knowledge/{0} missing from source '
                             'repo -- write it before building'.format(name))
        _copy(os.path.join('knowledge', name), dest_root)
    write(dest_root, os.path.join('knowledge', 'catalogue.yaml'),
          build_catalogue_carveout())


# ======================================================= the catalogue carve-out
#
# THE COMMISSION, owner, 2026-09-02 (verbatim): "你先做B，把B确认做通了以后，
# 再抽掉这些知识... 缺了知识没法跑是对的，但是还是得确认有了知识就能跑这件事
# 儿" -- prove the tool runs WITH knowledge, in full, before shipping a build
# that has none. Measured that day on this machine: stripping
# `knowledge/catalogue.yaml` entirely (the previous behaviour of this
# script) turned 171 tests red across both suites, all of them tests
# asserting SOME shape of that one file's content -- grouped, on inspection,
# into two very different kinds:
#
#   (i)  TOOL LOGIC tests that merely need *a* catalogue.yaml to exist with
#        the right STRUCTURE -- 165 of the 171. Most fail on a bare
#        FileNotFoundError, or on a checklist-row KeyError once the file
#        opens but `checklist.master` is empty. None of these 165 assert a
#        specific manufacturer number; they assert that the MECHANISM (face
#        designation resolution, the up-end rule, a checklist row existing
#        and reaching the rendered table) works given SOME catalogue.
#   (ii) KNOWLEDGE CONTENT tests -- 6 of the 171 -- that read
#        `catalogue.inserts` / `connectors.products` directly and assert
#        the OWNER'S real published numbers: a specific insert's 4:1
#        safety factor cited to "ALP catalog p53, 02/2024", a specific
#        tie's 3" edge distance. These are tests of the KNOWLEDGE, not of
#        this tool, and satisfying them here would mean shipping that real
#        data through a back door -- exactly what the commission says not
#        to do. See `append_test_exclusions()` below for how they are
#        excluded, honestly and by name, rather than faked.
#
# This function builds the fix for (i): a `knowledge/catalogue.yaml` that
# carries every (i)-relevant branch of the REAL file -- re-read and
# re-dumped from it, never hand-typed, so it cannot drift out of step with
# what the real file actually publishes -- while `catalogue.inserts` and
# `connectors.products`, the two branches (ii) depends on, ship EMPTY.
CARVEOUT_HEADER = """\
# ============================================================================
# knowledge/catalogue.yaml -- DISTRIBUTION CARVE-OUT, not the real file
# ============================================================================
# Written by tools/make_dist.py's build_catalogue_carveout(), which reads
# the owner's real knowledge/catalogue.yaml and re-emits ONLY the branches
# below -- parsed and re-dumped from the real file, never hand-typed, so
# this cannot drift out of step with what the real file actually publishes.
# The real file itself never leaves the private checkout. See
# knowledge/README.md and CONTRIBUTING.md for the full accounting.
#
# SHIPPED, AND WHY EACH PIECE IS SAFE TO:
#
#   face_designations   trade abbreviations (N.F., F.F., T.I.F., ...)
#                       printed on shop tickets everywhere. The real file's
#                       own header already calls this branch public --
#                       identifies no client, job, or plant.
#   orientation.up_end  a fact about precast assemblies IN GENERAL (which
#                       Revit axis is "up" once an assembly's origin is
#                       correctly configured), not one plant's naming
#                       convention -- the owner's own ruling for keeping it
#                       in the public layer rather than a plant file.
#   catalogue.rebar     ASTM A706 / A615 grade tables -- a published
#                       national standard, not a manufacturer's rating.
#   catalogue.safety_factors   both null, by design -- see the note inline.
#   checklist.master    the shop-ticket QC checklist's SKELETON: row ids,
#                       names, grades, and which check function answers
#                       each one. This tool's own built-in configuration,
#                       not knowledge about any job -- identifies nobody.
#
# NOT SHIPPED:
#
#   catalogue.inserts       real lifting-insert capacities (ALP catalog,
#                           02/2024), cited to a page and a date. EMPTY here.
#   catalogue.slotted_inserts   real JVI slotted-insert capacities. Omitted
#                               entirely (nothing in this distribution's
#                               tests needs its structure present at all).
#   connectors.products     real sandwich-wall tie capacities. EMPTY here.
#
# A handful of tests read those omitted tables directly and assert the
# owner's real published numbers. They are excluded from this
# distribution's test run, by name, with that reason -- see
# tools/make_dist.py's append_test_exclusions() and CONTRIBUTING.md.
# ============================================================================

"""

# The knowledge-content tests in WillTools.extension/tests/test_knowledge.py
# -- every one reads `catalogue.inserts` (via a real-file helper, several
# layers of indirection deep in a couple of cases) and pins a real ALP or HK
# Composites number. See the module docstring's "(ii)" paragraph above. This
# list is a SNAPSHOT, not a closed set -- test_knowledge.py is under active
# development on other branches of this same repo, and a new test asserting
# a real published number lands here the same way
# test_qc589_plant_override_beats_the_public_catalogue did on 2026-09-02
# (its own docstring: "this is not a stand-in catalogue, it is the one the
# tool actually ships, and it really does publish QL589SPG at 16,000 lb").
# If a dist build's test run comes back with an UNEXPECTED failure in
# test_knowledge.py rather than a clean skip, check whether it is a new
# knowledge-content test before assuming the carve-out itself regressed.
EXT_KNOWLEDGE_CONTENT_TESTS = [
    'test_the_shipped_catalogue_file_agrees_with_the_defaults',
    'test_the_factor_resolves_per_part_and_is_never_borrowed',
    'test_the_spacing_minimum_is_derived_from_the_part_s_own_edge_distance',
    'test_a_published_spacing_overrides_the_multiple',
    'test_a_part_with_no_edge_distance_to_double_stands_down_naming_it',
    'test_st100_carries_the_owners_number_and_st50_still_does_not',
    'test_qc589_plant_override_beats_the_public_catalogue',
]

# panellift/tests/test_safety_factors.py's own module docstring already
# calls this "a golden-value check against a page, not a recomputation" and
# already self-skips when knowledge/catalogue.yaml is absent -- but its
# skip condition is file EXISTENCE (`CATALOGUE.is_file()`), and this carve-
# out makes the file exist with `catalogue.inserts` empty. Only the one
# parametrized function that reads real ALP rows needs an added exclusion;
# the other 6 tests in that file pass fine against an empty `inserts`.
PANELLIFT_KNOWLEDGE_CONTENT_TESTS = [
    'test_the_catalogue_publishes_four_to_one_in_shear_for_every_qlpd_row',
]


def _load_real_catalogue():
    src = os.path.join(ROOT, 'knowledge', 'catalogue.yaml')
    if not os.path.isfile(src):
        raise SystemExit('make_dist: knowledge/catalogue.yaml missing from '
                         'the source repo -- the carve-out has nothing to '
                         'read from')
    with open(src, encoding='utf-8') as fh:
        data = yaml.safe_load(fh.read())
    return data


def build_catalogue_carveout():
    data = _load_real_catalogue()
    carveout = {
        'face_designations': data['face_designations'],
        'orientation': data['orientation'],
        'catalogue': {
            'safety_factors': data['catalogue']['safety_factors'],
            'inserts': {},
            'rebar': data['catalogue']['rebar'],
        },
        'connectors': {
            'what_it_is': data['connectors']['what_it_is'],
            'products': {},
        },
        'checklist': data['checklist'],
    }
    # Fail loudly rather than ship a carve-out that quietly stopped covering
    # what the (i) tests need -- these three are cheap, structural
    # invariants the real file has held since this script was written.
    if 'stock_by_size' in carveout['catalogue']['rebar']:
        raise SystemExit('make_dist: the real catalogue.rebar now carries '
                         'stock_by_size -- that is plant knowledge and must '
                         'not reach the carve-out; fix build_catalogue_carveout()')
    if not set(carveout['face_designations'].get('vocabularies', {})) >= {
            'near_far', 'in_form'}:
        raise SystemExit('make_dist: the real face_designations no longer '
                         'publishes near_far/in_form -- carve-out is stale')
    if len(carveout['checklist'].get('master') or []) < 30:
        raise SystemExit('make_dist: the real checklist.master looks '
                         'truncated ({0} rows) -- refusing to ship a partial '
                         'checklist'.format(len(carveout['checklist'].get('master') or [])))
    dumped = yaml.safe_dump(carveout, default_flow_style=False,
                            allow_unicode=True, sort_keys=False, width=100)
    return CARVEOUT_HEADER + dumped


# ========================================================= dist-only test exclusions
#
# Appended to the ALREADY-COPIED conftest.py inside dest_root -- never to
# this repo's own tracked WillTools.extension/tests/conftest.py or
# panellift/tests/conftest.py, which other agents may be editing at the
# same time (see this landing's own commission). The block below is plain
# text written straight into the copy in OUT_DIR; the source files are
# never opened for writing.
_EXT_CONFTEST_ADDITION = '''

# ============================================================================
# Appended by tools/make_dist.py -- knowledge-content test exclusion
# ============================================================================
# See knowledge/catalogue.yaml's own header (build_catalogue_carveout() in
# tools/make_dist.py) for the full accounting. Short version: these {n}
# tests read `catalogue.inserts` and assert the owner's real published
# manufacturer numbers, which this distribution does not ship. Skipped by
# name, not faked with invented numbers standing in for real knowledge.
#
# AUTO-DETECTING: if `catalogue.inserts` is ever non-empty in the
# knowledge/catalogue.yaml this test run actually finds -- e.g. a colleague
# who has since been handed the owner's real knowledge layers -- these run
# normally, unskipped, no edit needed here.
_KNOWLEDGE_CONTENT_TESTS = {test_list!r}


def _dist_real_inserts_present():
    import io as _io
    import os as _os
    here = _os.path.dirname(_os.path.dirname(_os.path.abspath(__file__)))
    path = _os.path.join(_os.path.dirname(here), 'knowledge', 'catalogue.yaml')
    try:
        import yaml as _yaml
        with _io.open(path, encoding='utf-8') as fh:
            data = _yaml.safe_load(fh.read()) or {{}}
    except Exception:
        return False
    return bool(((data.get('catalogue') or {{}}).get('inserts')) or None)


def pytest_collection_modifyitems(config, items):
    import pytest as _pytest
    if _dist_real_inserts_present():
        return
    skip = _pytest.mark.skip(
        reason='knowledge-content test: asserts the owner\\'s real product '
               'capacity data, which this distribution does not ship -- '
               'see knowledge/catalogue.yaml\\'s header and CONTRIBUTING.md')
    for item in items:
        if item.originalname in _KNOWLEDGE_CONTENT_TESTS or item.name in _KNOWLEDGE_CONTENT_TESTS:
            item.add_marker(skip)
'''

_PANELLIFT_CONFTEST_ADDITION = '''

# ============================================================================
# Appended by tools/make_dist.py -- knowledge-content test exclusion
# ============================================================================
# See knowledge/catalogue.yaml's own header. test_safety_factors.py already
# self-skips its whole module when knowledge/catalogue.yaml is ABSENT (see
# its own pytestmark) -- but this distribution's carve-out ships that file
# with `catalogue.inserts` EMPTY, so the file-existence check alone no
# longer catches the {n} test(s) below that need real ALP rows in it.
# AUTO-DETECTING the same way: real inserts present, and these run normally.
_KNOWLEDGE_CONTENT_TESTS = {test_list!r}


def _dist_real_inserts_present():
    import io as _io
    import os as _os
    here = _os.path.dirname(_os.path.dirname(_os.path.abspath(__file__)))
    path = _os.path.join(_os.path.dirname(here), 'knowledge', 'catalogue.yaml')
    try:
        import yaml as _yaml
        with _io.open(path, encoding='utf-8') as fh:
            data = _yaml.safe_load(fh.read()) or {{}}
    except Exception:
        return False
    return bool(((data.get('catalogue') or {{}}).get('inserts')) or None)


def pytest_collection_modifyitems(config, items):
    import pytest as _pytest
    if _dist_real_inserts_present():
        return
    skip = _pytest.mark.skip(
        reason='knowledge-content test: asserts the owner\\'s real product '
               'capacity data, which this distribution does not ship -- '
               'see knowledge/catalogue.yaml\\'s header and CONTRIBUTING.md')
    for item in items:
        if item.originalname in _KNOWLEDGE_CONTENT_TESTS or item.name in _KNOWLEDGE_CONTENT_TESTS:
            item.add_marker(skip)
'''


def append_test_exclusions(dest_root):
    """Appends the two blocks above to the COPIES of conftest.py already
    sitting in dest_root (written by copy_whole_dirs -- this runs after
    it). Refuses if either copy is missing, rather than silently shipping
    a dist where the (ii) tests fail loudly instead of skipping cleanly."""
    targets = [
        (os.path.join('WillTools.extension', 'tests', 'conftest.py'),
         _EXT_CONFTEST_ADDITION, EXT_KNOWLEDGE_CONTENT_TESTS),
        (os.path.join('panellift', 'tests', 'conftest.py'),
         _PANELLIFT_CONFTEST_ADDITION, PANELLIFT_KNOWLEDGE_CONTENT_TESTS),
    ]
    for rel, template, test_list in targets:
        path = os.path.join(dest_root, rel)
        if not os.path.isfile(path):
            raise SystemExit('make_dist: {0} not found in the assembled tree '
                             '-- append_test_exclusions() must run after '
                             'copy_whole_dirs()'.format(rel))
        addition = template.format(n=len(test_list), test_list=test_list)
        with open(path, 'a', encoding='utf-8', newline='\n') as fh:
            fh.write(addition)


# ---- examples/ -- the ASPIRE fixture, not a real job ------------------------
#
# The published ASPIRE Winter 2019 worked example, the SAME dict
# tools/setup_machine.py's own verify_panellift() uses to prove the
# installed CLI reaches real arithmetic -- built from a public journal
# article, not from any real job. A colleague's first success should be
# `python -m panellift examples/example-panel.job.json -o report.html`
# with no Revit, no model, no knowledge layer.
EXAMPLE_JOB = {
    'schema': 'panellift/1',
    'panel': {
        'mark': 'ASPIRE-EXAMPLE-1', 'length_ft': 32.0, 'width_ft': 10.0,
        'thickness_in': 8.75, 'unit_weight_pcf': 150.0,
        'fci_psi': 4000.0, 'fc_psi': 5000.0,
        'lifters': [
            {'mark': 'L1', 'family': 'EXAMPLE_LIFTER', 'x': 6.627416997969522,
             'y': 2.5, 'capacity_lb': 12000.0},
            {'mark': 'L2', 'family': 'EXAMPLE_LIFTER', 'x': 6.627416997969522,
             'y': 7.5, 'capacity_lb': 12000.0},
            {'mark': 'L3', 'family': 'EXAMPLE_LIFTER', 'x': 25.372583002030478,
             'y': 2.5, 'capacity_lb': 12000.0},
            {'mark': 'L4', 'family': 'EXAMPLE_LIFTER', 'x': 25.372583002030478,
             'y': 7.5, 'capacity_lb': 12000.0},
        ],
    },
    'rigging': {'mode': 'rolling', 'sling_angle_deg': 90.0},
    'stages': [{'name': 'stripping', 'multiplier': 1.3, 'fc_psi': 4000.0}],
}

EXAMPLES_README = """\
# examples/

`example-panel.job.json` is a `panellift` job file for a flat panel with four
lifting inserts -- the ASPIRE Winter 2019 worked example (Brown,
"Calculations for Handling and Shipping Precast Concrete Deck Panels"), the
same fixture panellift's own test suite and `tools/setup_machine.py --check`
verify the installed CLI against. It is a published example, not a real job:
nothing about it identifies anyone.

Try it with no Revit, no model, and no knowledge layer:

    python -m panellift examples/example-panel.job.json -o report.html

Expect a peak transfer-stage tension near 244 psi, matching the article. If
you see a different number, `panellift`'s own suite (`python -m pytest
panellift`) is the next place to look.
"""


def write_examples(dest_root):
    write(dest_root, os.path.join('examples', 'example-panel.job.json'),
          json.dumps(EXAMPLE_JOB, indent=2) + '\n')
    write(dest_root, os.path.join('examples', 'README.md'), EXAMPLES_README)


# ---- WillTools-Intro.pptx -- the colleague intro deck, riding with the dist ---
#
# THE COMMISSION, owner, 2026-09-02 (verbatim): "GitHub推了吧。给我PPT也放上
# 去。" -- push, and put the intro PPT up too.
#
# Lives beside this repo, not inside it (`os.path.dirname(ROOT)`, i.e. the
# same folder pyrevit-dev-git itself sits in) -- a binary with 8 embedded
# screenshots does not belong in git history the way source does, so it is
# never tracked here and this is the ONE place its path is named. Copied
# into the dist tree's ROOT (not `docs/`, which is a curated,
# one-reason-per-file manifest meant for text) so it rides in the SAME
# orphan commit as everything else, where `scrub_tree()` also walks it --
# harmlessly, since a .pptx is a zip of compressed XML and the raw bytes
# will not contain readable text for the patterns to match. That is NOT the
# real gate for this file: a .pptx's text (slide bodies, SPEAKER NOTES,
# docProps author/company fields) lives inside compressed per-slide XML
# entries, and only a zip-aware scan finds it. That scan is a manual,
# one-time pass run before each build that ships a new copy of the deck --
# see the commit that landed this comment for what it found and fixed (two
# slides used an ad hoc placeholder model filename, styled like a real one
# but not this repo's own sanctioned `sample_job` form -- deliberately not
# quoted verbatim here, so this comment does not itself read as a Revit
# filename to the structural scrub pattern) -- not a repeatable tool,
# because the deck itself changes rarely and a colleague is not expected to
# rebuild it.
INTRO_DECK_NAME = 'WillTools-Intro.pptx'
INTRO_DECK_SRC = os.path.join(os.path.dirname(ROOT), INTRO_DECK_NAME)


def copy_intro_deck(dest_root):
    if not os.path.isfile(INTRO_DECK_SRC):
        print('make_dist: no {0} found beside the repo at {1} -- skipping '
             '(not required for a build; only added when the owner asks '
             'for it to ride along)'.format(INTRO_DECK_NAME, INTRO_DECK_SRC))
        return
    shutil.copy2(INTRO_DECK_SRC, os.path.join(dest_root, INTRO_DECK_NAME))
    print('copied {0} ({1} bytes) -- scrubbed by hand before this build; '
         'see tools/make_dist.py\'s copy_intro_deck() comment'.format(
             INTRO_DECK_NAME, os.path.getsize(INTRO_DECK_SRC)))


# ---- .github/ -- CI workflow (comment corrected) + issue templates ---------

CI_YML = """\
name: ci

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install
        # requirements.txt is the single source of truth for host-side deps.
        # PyYAML fixes collection (ticketpage._read_yaml falls back to it off
        # IronPython; without it eight test files die before running). pymupdf
        # is required for tools/ticket_delta.py's tests.
        run: pip install -r requirements.txt

      - name: panellift tests
        run: python -m pytest panellift -q -p no:cacheprovider

      - name: vibe tests (the Revit-free half of the extension)
        run: python -m pytest WillTools.extension -q -p no:cacheprovider

      - name: Syntax smoke (extension + tools + panellift)
        run: python -m compileall -q WillTools.extension tools panellift/src

      - name: Scrub worktree with the committed generic sample terms
        # Real client names never leave the owner's machine: this
        # distribution ships no .scrub-terms at all, so CI runs the generic
        # .scrub-terms.example instead -- the same fallback a colleague's own
        # clone uses locally with no real term list.
        run: python tools/scrub.py --worktree --terms .scrub-terms.example
"""

ISSUE_CONFIG_YML = """\
blank_issues_enabled: false
contact_links:
  - name: Ask the owner directly
    url: https://github.com/will123song/pyrevit-dev/discussions
    about: Not sure it's a bug, or want to talk through knowledge layers first?
"""

BUG_REPORT_MD = """\
---
name: Bug report
about: Something ran and gave a wrong or crashing answer
title: ''
labels: bug
assignees: ''
---

<!--
Before filing: is this a WRONG answer, or a REFUSAL to answer?

WillTools refuses rather than guesses, by design -- a "NOT JUDGED" result
with a stated reason is often the tool working CORRECTLY (an unmapped
parameter, a missing knowledge layer, a check that legitimately has nothing
to say about this piece). If the tool told you clearly why it would not
answer, that is very likely not this template -- it's the tool doing what it
is supposed to. File that as a "refused, and I think it shouldn't have"
report instead (see the Question template), because that is knowledge-layer
work, not a code fix.

This template is for: it crashed, it gave two different answers to the same
input, or it silently produced a wrong number with no stated reason.
-->

## What happened

<!-- Paste the exact error, verbatim, in a code block. Do not summarize it or
retype it from memory -- the exact text (including any traceback) is what
gets this fixed fast. -->

```
paste it here
```

## What you expected instead

## Steps to reproduce

1.
2.
3.

## Environment

Run this and paste its ENTIRE output below, unedited -- it is the single
most diagnostic thing you can attach. It is plain text with no color codes,
so it pastes cleanly into this box:

```
python tools/setup_machine.py --check
```

<details>
<summary>setup_machine.py --check output</summary>

```
paste it here
```

</details>

- Windows version:
- `python --version`:
- Revit version (if this involves a ribbon button):
- pyRevit version (if this involves a ribbon button):

## Which button or command

<!-- e.g. "Ticket QC" on the ribbon, or `python -m panellift ...` -->

## No client or job data, please

Screenshots, pasted output, and job files should carry no real client name,
job name, or job number. If a screenshot or a job file shows one, use a
placeholder (`sample_client`, `sample_job`) or ask the owner how to share it
privately instead of pasting it here.
"""

QUESTION_MD = """\
---
name: Question
about: Something was unclear, or you're not sure what a result means
title: ''
labels: question
assignees: ''
---

## What you were trying to do

## What the report or the ribbon said

<!-- Paste the relevant text, or attach a screenshot with no client/job
names visible (see the note in the bug report template). -->

## What was unclear about it

<!-- Was it a refusal you think should have been an answer? A wording you
could not parse? A row you did not expect to be blank? Say which. -->
"""


def write_github(dest_root):
    write(dest_root, os.path.join('.github', 'workflows', 'ci.yml'), CI_YML)
    write(dest_root, os.path.join('.github', 'ISSUE_TEMPLATE', 'config.yml'),
          ISSUE_CONFIG_YML)
    write(dest_root, os.path.join('.github', 'ISSUE_TEMPLATE', 'bug_report.md'),
          BUG_REPORT_MD)
    write(dest_root, os.path.join('.github', 'ISSUE_TEMPLATE', 'question.md'),
          QUESTION_MD)


# =================================================================== orphan git

def git_init_and_commit(dest_root, source_hash):
    stamp = datetime.date.today().isoformat()
    message = (
        'WillTools distribution {0}\n\n'
        'Clean colleague-trial snapshot, built by tools/make_dist.py.\n'
        'Source commit (owner repo, not published): {1}\n'
        'No plant/job layers, no catalogue-rationale.md, no product-capacity '
        'data (catalogue.inserts / connectors.products ship empty). '
        'knowledge/catalogue.yaml DOES ship a structural carve-out -- see '
        'its own header and knowledge/README.md.\n'
    ).format(stamp, source_hash)

    def run(*args):
        r = subprocess.run(['git'] + list(args), cwd=dest_root, check=False,
                            capture_output=True, text=True, errors='replace')
        if r.returncode != 0:
            raise SystemExit('make_dist: git {0} failed:\n{1}'.format(
                ' '.join(args), r.stderr))
        return r

    run('init', '-q')
    run('add', '-A')
    run('commit', '-q', '-m', message)
    rev = run('rev-parse', 'HEAD').stdout.strip()
    return rev, message


# ======================================================================= main

def _clear_readonly_and_retry(func, path, exc_info):
    """`shutil.rmtree`'s onerror/onexc hook. A previous build's `.git`
    directory carries read-only loose-object files on Windows -- git sets
    that mode deliberately, and plain `shutil.rmtree` does not clear it
    before unlinking, so `--force` over a tree that already has an orphan
    commit in it raises PermissionError instead of overwriting. Clear the
    bit and retry once; if that still fails, let it raise for real."""
    os.chmod(path, 0o666)
    func(path)


def build(dest_root, force=False):
    if os.path.exists(dest_root):
        if not force:
            raise SystemExit('make_dist: {0} already exists -- pass --force '
                             'to overwrite'.format(dest_root))
        try:
            shutil.rmtree(dest_root, onexc=lambda f, p, e: _clear_readonly_and_retry(f, p, e))
        except TypeError:
            # Python < 3.12: shutil.rmtree has onerror, not onexc, and the
            # callback signature drops the exception-info-vs-exception
            # difference that does not matter to _clear_readonly_and_retry.
            shutil.rmtree(dest_root, onerror=_clear_readonly_and_retry)
    os.makedirs(dest_root)

    copy_top_level(dest_root)
    copy_whole_dirs(dest_root)
    copy_curated(dest_root, 'tools', TOOLS_MANIFEST)
    copy_curated(dest_root, 'docs', DOCS_MANIFEST)
    copy_knowledge_stub(dest_root)
    append_test_exclusions(dest_root)
    write_examples(dest_root)
    write_github(dest_root)
    copy_intro_deck(dest_root)

    print('assembled {0}'.format(dest_root))

    hits = scrub_tree(dest_root)
    if hits:
        print('\nSCRUB GATE FAILED -- {0} hit(s), build aborted, no commit made:'
              .format(len(hits)))
        for hit in hits:
            print('  ' + hit)
        raise SystemExit(1)
    print('scrub gate: clean -- 0 hits against the real term list + '
          'structural patterns + private-host/machine-path patterns')

    source_hash = git('rev-parse', 'HEAD').stdout.strip() or '(no commits yet)'
    rev, message = git_init_and_commit(dest_root, source_hash)
    print('\norphan commit {0} written in {1}'.format(rev[:12], dest_root))
    print('-' * 72)
    print(message)
    print('-' * 72)

    print("""
DOES NOT PUSH. Review the tree at {0} first. Once reviewed, from inside it:

    git remote add github https://github.com/will123song/pyrevit-dev.git
    git push --force-with-lease github HEAD:dist

Pushed to a `dist` branch, NOT main -- deliberate, so nothing already on the
GitHub repo's main branch is clobbered by this. Point colleagues at the
`dist` branch (or open a PR from dist -> main once reviewed) rather than
main directly.
""".format(dest_root))
    return dest_root


def main(argv=None):
    ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0])
    ap.add_argument('out_dir', nargs='?', help='where to build the tree')
    ap.add_argument('--force', action='store_true',
                    help='overwrite out_dir if it exists')
    ap.add_argument('--scan-only', metavar='DIR',
                    help='run only the scrub gate over an existing tree '
                         '(e.g. to prove it aborts on a planted term) and exit')
    args = ap.parse_args(argv)

    if args.scan_only:
        hits = scrub_tree(args.scan_only)
        if hits:
            print('{0} hit(s):'.format(len(hits)))
            for hit in hits:
                print('  ' + hit)
            return 1
        print('clean: 0 hits')
        return 0

    if not args.out_dir:
        print('usage: make_dist.py OUT_DIR [--force]', file=sys.stderr)
        return 2

    build(os.path.abspath(args.out_dir), force=args.force)
    return 0


if __name__ == '__main__':
    raise SystemExit(main())