# -*- coding: utf-8 -*- """The takeoff: what a job actually contains, per piece and in total. No Revit. WHAT THIS IS BUILT ON, AND WHY IT IS NOT ANOTHER MODEL SWEEP ------------------------------------------------------------- `vibe.reinforceindex` already walks every assembly, resolves its members and records a `families` dict -- family name -> instance count -- on every piece record. That is a takeoff's raw material and nothing was reading it. This module takes the index cache as its INPUT and adds no model access of its own, so a whole-job takeoff costs whatever `reinforceindex.read()` costs and nothing more. The consequence is a limit, stated once here because every honest line of the output depends on it: THIS MODULE CAN ONLY SEE WHAT THE INDEX RECORDED. Three limits follow and each is reported rather than papered over. 1. The index builds only ELIGIBLE pieces -- reinforced, not on hold. A job has more assemblies than that. The caller passes `population` so the report can say what fraction of the job it covered instead of implying the whole of it. 2. The index keys `families` by FAMILY NAME, and at this plant a reinforcing family is a BEND SHAPE, not a bar mark: one shape family carries one bar mark on one piece and a different one on the next. So hardware resolves to a part number off the family name and REINFORCING LARGELY DOES NOT. Those instances are counted and reported as `bar_shape` with the mark unresolved -- never dropped, and never silently rolled into a bar mark that was guessed. The fix is one field on the index record and it is written up in `docs/takeoff.md`: a `marks` counter built from `ticketqc._member_mark(member, know)`, which already returns the mark an engineer says out loud on a bar and on an anchor alike. `piece_lines()` uses it the moment it appears -- see `_bar_lines_from_marks` -- and needs no other change here. 3. `features is None` on an index record means the piece could not be MEASURED. It does not mean the piece is empty, and it does not stop the takeoff: the `families` dict is recorded on nearly every failure path. A record with no families dict at all is the real could-not-judge, and it is listed as one with its reason rather than counted as zero. THE COUNTING RULES THIS JOB HAS ALREADY PAID TO LEARN ------------------------------------------------------ **Control numbers.** An assembly is a MARK. A mark is a drawing. The pieces on the floor are CONTROL NUMBERS. In one sampled ticket set 151 tickets carried only 71 distinct marks, and one mark alone accounted for eighteen of them. So a takeoff that counts marks under-orders every material in the job, and this one multiplies -- but it prints both figures side by side, with the multiplier and where the multiplier came from, because per-mark and per-job answer different questions and a reader will want to check one against the other. A multiplier the caller could not source is 1 with basis `assumed_one`, and that is reported as an UNDERCOUNT, loudly, not as a number. This module never decides that multiplier and never reads a knowledge key for it: `pieces` arrives per mark, already sourced and already ranked. What one exported ticket covers -- `nomenclature.ticket_unit` -- is declared per job, read in `takeoffgather.run`, and acted on by `takeoffcount.counting_plan`. Both files carry the reasoning. **Nesting.** A member whose `SuperComponent` is set is a child and still appears in `GetMemberIds()`, so a family nested inside another is counted twice: once as itself, once inside its parent's instance. This module does not guess. A nesting DECLARED in `takeoff.nested_in` is honoured -- the child leaves the order list for its own section. A nesting nobody declared is DETECTED and reported as a suspicion for a person to rule on: `suspected_nesting()` finds family pairs whose counts are equal on every piece where both occur. It never subtracts on a suspicion. An automatic correction that is wrong is invisible in the total; a suspicion that is wrong costs one sentence to dismiss. **Attached bars.** `drawings.annotation_covered_by_parent` already records which bar marks arrive welded to which part. Those bars are real steel and are always COUNTED. Whether they are ORDERED is a purchasing question, not a geometry one, so it is a switch -- `takeoff.order_attached_bars_separately`, default false -- and either way they appear in their own section with the parent named. Silently either way is the one thing not on offer. WHAT IT REFUSES TO COUNT ------------------------- A family the rules cannot classify goes to `unclassified` with its name, its count and how many pieces carry it. It is never bucketed into 'other'. Every classification carries a `basis`: `declared` when a knowledge key named the family outright, `rule:` when a pattern decided it, and the report groups by basis so every inference this module made is on the page. """ # pylint: disable=invalid-name import re # Every class this takeoff knows. `unclassified` is a real answer, not a # failure mode, and it is the only one whose contents are a work list. BAR = 'bar' # reinforcing, mark resolved BAR_SHAPE = 'bar_shape' # reinforcing, family is a bend shape, mark unknown STRAND = 'strand' MESH = 'mesh' HARDWARE = 'hardware' # an orderable part with a part number CAST = 'cast' # separately modelled cast concrete, not an insert NESTED = 'nested' # declared as a child of another family NOT_ORDERED = 'not_ordered' # the piece itself, or a modelling construct UNCLASSIFIED = 'unclassified' ORDERABLE = (BAR, BAR_SHAPE, STRAND, MESH, HARDWARE, CAST) # Defaults live here rather than in `knowledge.DEFAULTS` so that this module # owns them and no other file has to change to add a takeoff. Any knowledge # layer overrides them under `takeoff:`; the plant or job layer is where the # real prefixes belong, and `docs/takeoff.md` carries the block to paste. # # `not_ordered` DEFAULTS TO EMPTY on purpose. A default list of things to # ignore is a default under-count, and the families it would name -- the # panel itself, a modelling zone -- are plant nomenclature that this module # has no business knowing. Left empty they surface as `unclassified`, which # is the loud version of the same fact and the one a person can act on. DEFAULTS = { 'bar_prefixes': ['SR'], 'part_prefixes': ['SP', 'P'], 'shape_markers': ['(SCHEDULED)'], 'not_ordered': [], 'family_class': {}, 'nested_in': {}, 'attached_to': {}, 'order_attached_bars_separately': False, } def settings(know, get=None): """The `takeoff:` block, defaults filled in. `get` is `knowledge.get`. `get` is passed in rather than imported so this module needs nothing but the standard library, which is what keeps it testable off a model. """ out = dict(DEFAULTS) for key in DEFAULTS: if get is None: value = ((know or {}).get('takeoff') or {}).get(key) else: value = get(know, 'takeoff.' + key) if value is not None: out[key] = value return out # ------------------------------------------------------------ classification def _up(text): return (text or '').upper() def _prefixed(name, prefixes): """The longest `` token in `name`, or None. Longest match wins so a part in an `SP` series is never read as the `P` series inside it, and the lookbehind is what stops an embed family whose name wraps a part number from resolving to anything but that number. """ up = _up(name) best = None for prefix in sorted([_up(p) for p in (prefixes or [])], key=len, reverse=True): if not prefix: continue found = re.search(r'(? len(best): best = token return best def classify(family, cfg, is_strand=None, is_mesh=None, is_cast=None): """(class, key, basis) for one family name. `key` is what a person orders by: a part number, a bar mark, a bend shape, or -- when nothing resolved -- the family name itself, so that an unclassified row still names the thing it could not classify. Order matters and is deliberate: 1. a family named outright in knowledge beats every pattern; 2. a declared nesting, so a child never reaches the order list; 3. a declared not-ordered family; 4. strand, mesh and cast concrete, which the existing `families:` rules already decide and which this module must not re-decide; 5. a bend-shape marker, BEFORE the bar-prefix rule -- a shape family has no mark in its name and must not borrow one; 6. bar prefixes, then part prefixes; 7. unclassified. """ up = _up(family) declared = cfg.get('family_class') or {} for key in declared: if _up(key) == up: return (str(declared[key]), family, 'declared') for child in (cfg.get('nested_in') or {}): if _up(child) == up: return (NESTED, family, 'declared') for token in (cfg.get('not_ordered') or []): if _up(token) and _up(token) in up: return (NOT_ORDERED, family, 'declared') if is_strand is not None and is_strand(family): return (STRAND, family, 'rule:strand') if is_mesh is not None and is_mesh(family): return (MESH, family, 'rule:mesh') if is_cast is not None and is_cast(family): return (CAST, family, 'rule:cast') for marker in (cfg.get('shape_markers') or []): if _up(marker) and _up(marker) in up: return (BAR_SHAPE, family, 'rule:shape_marker') mark = _prefixed(family, cfg.get('bar_prefixes')) if mark: return (BAR, mark, 'rule:bar_prefix') part = _prefixed(family, cfg.get('part_prefixes')) if part: return (HARDWARE, part, 'rule:part_prefix') return (UNCLASSIFIED, family, 'none') # ------------------------------------------------------------- attached bars def attached_parents(cfg, know, get=None): """{BAR MARK: [parent marks]} -- bars that ship welded to a part. Read from `drawings.annotation_covered_by_parent`, which the drawing checks already maintain, so a bar cannot be attached for the purpose of one check and loose for another. `takeoff.attached_to` overrides it for a job where the two genuinely differ. """ if get is None: source = ((know or {}).get('drawings') or {}).get( 'annotation_covered_by_parent') or {} else: source = get(know, 'drawings.annotation_covered_by_parent', {}) or {} override = cfg.get('attached_to') if override: source = override out = {} for child in source: parents = source[child] if isinstance(parents, (list, tuple)): out[_up(child)] = [str(p) for p in parents] elif parents: out[_up(child)] = [str(parents)] return out # ----------------------------------------------------------------- one piece def piece_lines(record, cfg, attached=None, is_strand=None, is_mesh=None, pieces=1, is_cast=None): """Every takeoff line for one index record, per-mark and per-job. `pieces` is the control-number multiplier: how many physical pieces this one mark issues. `per_mark` is what one of them contains; `per_job` is what the job needs. """ attached = attached or {} lines = [] families = (record or {}).get('families') or {} marks = (record or {}).get('marks') skip = set() if marks: lines.extend(_bar_lines_from_marks( record, marks, families, cfg, attached, pieces)) skip = set(_up(f) for f in _shape_families(families, cfg)) for family in sorted(families): if _up(family) in skip: continue klass, key, basis = classify(family, cfg, is_strand, is_mesh, is_cast) count = int(families[family] or 0) lines.append(_line(record, klass, key, family, basis, count, pieces, attached)) return sorted(lines, key=lambda r: (r['class'], r['key'])) def _shape_families(families, cfg): """Families that are bend shapes -- the ones a `marks` counter replaces.""" out = [] for family in families: for marker in (cfg.get('shape_markers') or []): if _up(marker) and _up(marker) in _up(family): out.append(family) break return out def _bar_lines_from_marks(record, marks, families, cfg, attached, pieces): """Bar lines built from a per-instance mark counter, when one exists. THIS IS THE PATH THAT DOES NOT EXIST YET. `reinforceindex.extract` does not record `marks`; the moment it does, every `bar_shape` row in the output turns into a real bar mark and nothing else in this module changes. Written now so that the index change is a one-field change and not a rewrite, and exercised by the tests with a synthetic record. A mark counter carries hardware marks too, because a hardware family and its mark are the same string. Only marks that read as a BAR and are not already a family we counted are taken from here; anything else would be counted once as a family and once again as a mark. """ known = set(_up(f) for f in families) lines = [] for mark in sorted(marks): bar = _prefixed(mark, cfg.get('bar_prefixes')) if not bar or _up(bar) in known: continue lines.append(_line(record, BAR, bar, None, 'index:marks', int(marks[mark] or 0), pieces, attached)) return lines def _line(record, klass, key, family, basis, count, pieces, attached): parents = attached.get(_up(key)) if klass in (BAR, BAR_SHAPE) else None return { 'piece': (record or {}).get('name'), 'design_number': (record or {}).get('design_number'), 'class': klass, 'key': key, 'family': family, 'basis': basis, 'per_mark': count, 'pieces': int(pieces), 'per_job': count * int(pieces), 'attached_to': list(parents) if parents else None, } # ------------------------------------------------------------------- the job def build(cache, cfg, multipliers=None, attached=None, is_strand=None, is_mesh=None, population=None, is_cast=None): """The whole takeoff. `cache` is a `reinforcecore` cache, read, not touched. `multipliers` is {piece name: (count, basis)}. A piece missing from it is 1 with basis `assumed_one` and lands in `undercounted`, because a multiplier nobody could source is not a multiplier of one. """ multipliers = multipliers or {} attached = attached or {} pieces = (cache or {}).get('pieces') or {} by_piece = {} unjudged = [] undercounted = [] all_lines = [] for name in sorted(pieces): record = pieces[name] or {} count, basis = multipliers.get(name, (1, 'assumed_one')) count = int(count or 1) if basis == 'assumed_one': undercounted.append(name) families = record.get('families') or {} if not families: unjudged.append({ 'piece': name, 'reason': record.get('error') or 'no families recorded', }) by_piece[name] = {'piece': name, 'pieces': count, 'multiplier_basis': basis, 'lines': [], 'judged': False, 'design_number': record.get('design_number')} continue lines = piece_lines(record, cfg, attached, is_strand, is_mesh, pieces=count, is_cast=is_cast) by_piece[name] = { 'piece': name, 'design_number': record.get('design_number'), 'pieces': count, 'multiplier_basis': basis, 'lines': lines, 'judged': True, } all_lines.extend(lines) return { 'by_piece': by_piece, 'totals': roll_up(all_lines), 'unclassified': unclassified_rows(all_lines), 'unjudged': unjudged, 'undercounted': sorted(undercounted), 'nesting_suspects': suspected_nesting(pieces), 'attached': attached, 'order_attached_separately': bool( cfg.get('order_attached_bars_separately')), 'coverage': coverage(cache, population, unjudged), 'bases': bases(all_lines), 'pieces_on_the_floor': sum( by_piece[n]['pieces'] for n in by_piece), } def roll_up(lines): """Job totals, keyed (class, key). Both figures, always both.""" out = {} for line in lines: slot = (line['class'], line['key']) row = out.get(slot) if row is None: row = {'class': line['class'], 'key': line['key'], 'per_mark_total': 0, 'per_job_total': 0, 'marks': 0, 'families': [], 'bases': [], 'attached_to': line.get('attached_to')} out[slot] = row row['per_mark_total'] += line['per_mark'] row['per_job_total'] += line['per_job'] row['marks'] += 1 if line.get('family') and line['family'] not in row['families']: row['families'].append(line['family']) if line['basis'] not in row['bases']: row['bases'].append(line['basis']) rows = list(out.values()) for row in rows: row['families'].sort() row['bases'].sort() row['multiplier'] = ( round(row['per_job_total'] / float(row['per_mark_total']), 3) if row['per_mark_total'] else None) rows.sort(key=lambda r: (r['class'], -r['per_job_total'], r['key'])) return rows def order_rows(totals, order_attached_separately=False): """(order, ships_with_parent, not_ordered) -- the split a buyer needs. An attached bar is always counted. Whether it appears on the ORDER is the switch; either way it appears, with its parent named. """ order, ships, other = [], [], [] for row in totals: if row['class'] not in ORDERABLE: other.append(row) elif row.get('attached_to') and not order_attached_separately: ships.append(row) else: order.append(row) return order, ships, other def unclassified_rows(lines): """Every family no rule could place, with what it would have cost. Never merged into an 'other' row. An under-count that looks tidy is worse than a messy honest one, so this is the section that has to be read. """ out = {} for line in lines: if line['class'] != UNCLASSIFIED: continue row = out.get(line['key']) if row is None: row = {'family': line['key'], 'per_mark_total': 0, 'per_job_total': 0, 'pieces': 0, 'examples': []} out[line['key']] = row row['per_mark_total'] += line['per_mark'] row['per_job_total'] += line['per_job'] row['pieces'] += 1 if line.get('piece') and len(row['examples']) < 5: row['examples'].append(line['piece']) rows = list(out.values()) rows.sort(key=lambda r: (-r['per_job_total'], r['family'])) return rows def bases(lines): """How many rows each basis decided. The inference budget, on the page.""" out = {} for line in lines: out[line['basis']] = out.get(line['basis'], 0) + 1 return out def coverage(cache, population, unjudged): """What the takeoff saw, against what the job has. Never implied. `assemblies_in_model` is the caller's honest denominator -- the index only ever builds reinforced, not-on-hold pieces, so `marks_not_indexed` is a real hole in the job and not a rounding. """ pieces = (cache or {}).get('pieces') or {} population = population or {} indexed = len(pieces) assemblies = population.get('assemblies') return { 'indexed_marks': indexed, 'judged_marks': indexed - len(unjudged), 'unjudged_marks': len(unjudged), 'assemblies_in_model': assemblies, 'marks_not_indexed': (None if assemblies is None else max(0, assemblies - indexed)), 'built_at': (cache or {}).get('built_at'), 'complete': bool(assemblies is not None and assemblies == indexed and not unjudged), } # ---------------------------------------------------------- nesting suspects def suspected_nesting(pieces, min_pieces=3, min_agreement=0.9, min_distinct_counts=2): """Family pairs that look like child-inside-parent. A SUSPICION only. Two families whose counts are EQUAL on the pieces where the first occurs is the signature of a nested family: the child is counted once on its own and once inside its parent's instance. WHY A LOCKSTEP OF ONES IS NOT EVIDENCE. Two families that each appear exactly once on the same forty pieces agree perfectly and mean nothing -- that is what a pair of unrelated parts on one panel type looks like. What a nested family does is FOLLOW its parent's count as that count VARIES: four of one, four of the other; eight and eight. So a pair only qualifies when the child's count takes at least `min_distinct_counts` different values across the pieces. Without that filter this list came back with forty rows of coincidence and the three real pairs buried in them. WHY THE AGREEMENT IS NOT REQUIRED TO BE PERFECT. The strongest declared nesting on this job -- a bar that lives inside an embed whose family name says "with bar" -- has 938 instances of the bar against 943 of the embed. Five pieces disagree, for reasons that are somebody's to explain, and an exact-equality test finds nothing at all. So agreement is a RATIO, it is reported on every row, and `1.0` is visibly different from `0.95`. DIRECTION IS OFTEN UNDECIDABLE FROM COUNTS ALONE, and saying so is the honest part. When A and B occur on the same pieces in the same numbers, the arithmetic cannot tell which sits inside which -- only that one of the two is very probably being counted twice. Such a pair is reported ONCE with `direction: undecided` rather than twice with a direction each way, because two rows reads like two findings. Nothing is subtracted on this evidence. It is printed so that a person can move the pair into `takeoff.nested_in`, which IS acted on. An automatic correction that is wrong is invisible in the total; a suspicion that is wrong costs one sentence to dismiss. """ where = {} for name in pieces: record = pieces[name] or {} for family in (record.get('families') or {}): where.setdefault(family, {})[name] = int( record['families'][family] or 0) def agreement(child, parent): child_on, parent_on = where[child], where[parent] if not child_on: return 0.0 same = 0 for piece in child_on: if parent_on.get(piece) == child_on[piece]: same += 1 return same / float(len(child_on)) names = sorted(where) found = {} for child in names: if len(where[child]) < min_pieces: continue if len(set(where[child].values())) < min_distinct_counts: continue for parent in names: if parent == child: continue if len(where[parent]) < len(where[child]) * min_agreement: continue forward = agreement(child, parent) if forward < min_agreement: continue back = (len(where[parent]) >= min_pieces and agreement(parent, child) >= min_agreement) key = tuple(sorted((child, parent))) if back else (child, parent) if key in found: continue first, second = (key if back else (child, parent)) found[key] = { 'child': first, 'parent': second, 'direction': 'undecided' if back else 'child_in_parent', 'agreement': round(forward if not back else min(forward, agreement(parent, child)), 3), 'pieces': len(where[child]), 'instances': sum(where[child].values()), 'parent_pieces': len(where[parent]), } out = list(found.values()) out.sort(key=lambda r: (-r['agreement'], -r['instances'], r['child'], r['parent'])) return out