will
25592fa6c2
ci / test (push) Successful in 6m14s
History-free snapshot of the private development repository at 77a82eb. Code, docs and the public catalogue only: no plant or job knowledge layers, no client, job or producer names (tools/scrub.py clean against the widened term list; a second word-level grep clean). The intro deck is left out because its slide images cannot be checked for job drawings.
471 lines
22 KiB
Python
471 lines
22 KiB
Python
# -*- coding: utf-8 -*-
|
|
"""How many physical pieces one mark is. The control-number multiplier. No Revit.
|
|
|
|
WHY THIS IS ITS OWN MODULE
|
|
---------------------------
|
|
It is the single number that decides whether the takeoff is right, and there
|
|
is more than one place it can come from. An assembly is a MARK, a mark is one
|
|
drawing, and the pieces on the floor are the pieces the mark issues.
|
|
Identical pieces share a mark, so counting marks under-orders every material
|
|
in the job.
|
|
|
|
How badly is measurable off an exported ticket set without opening Revit:
|
|
151 tickets, 71 distinct marks. The heaviest mark in that sample issued
|
|
eighteen tickets. A per-mark takeoff of that sample would have ordered 71
|
|
pieces' worth of hardware for 151 pieces -- IF one ticket is one piece there,
|
|
which is a declared fact and not this module's to assume. See the ticket-unit
|
|
section below; it is the reason a ticket set can be read to two different
|
|
numbers.
|
|
|
|
THREE SOURCES, RANKED, AND THE RANKING IS THE POINT
|
|
----------------------------------------------------
|
|
control_numbers the control numbers carried by the model's own
|
|
elements, grouped by mark. AUTHORITATIVE: a control
|
|
number is one physical piece, and this is where they
|
|
actually live. Measured on this job by the job
|
|
reconciliation: 2480 control-numbered elements ->
|
|
1776 pieces, against 755 assemblies. They sit on
|
|
STRUCTURAL FRAMING; walls, floors, columns,
|
|
foundations and generic models were swept -- 17,239
|
|
elements -- and carried none.
|
|
ticket_files the exported tickets, named <job>-<control>_<mark>.
|
|
Only covers marks that have been ticketed, so it is a
|
|
cross-check rather than a source -- and a very good
|
|
one, because it is ground truth for the marks it does
|
|
cover.
|
|
assembly_instances how many assembly instances share the mark's name.
|
|
MEASURED AND FOUND USELESS ON THIS JOB, and kept only
|
|
because it is useless in a way worth recording: 755
|
|
assembly instances, 755 distinct names, not one
|
|
duplicate. It therefore returned 1 for every mark,
|
|
which made every per-job figure silently equal its
|
|
per-mark figure -- the exact undercount this module
|
|
exists to prevent, wearing the costume of a sourced
|
|
number. It now REFUSES to emit a count of one (see
|
|
`from_assembly_names`), so on a job like this one it
|
|
contributes nothing at all and the marks it cannot
|
|
speak for stay visibly `assumed_one`.
|
|
|
|
Anything none of them covers is 1 with basis `assumed_one`, which the report
|
|
prints as an UNDERCOUNT rather than as a quantity. A multiplier nobody could
|
|
source is not a multiplier of one -- and the reason that rule was worth
|
|
writing is that it is what caught `assembly_instances` being wrong.
|
|
|
|
WHAT ONE TICKET IS, AND WHY THIS MODULE MAY NOT DECIDE IT
|
|
----------------------------------------------------------
|
|
`ticket_files` is a piece count only where ONE TICKET IS ONE PIECE. That is
|
|
not universal and it is not this module's to assume: some plants issue one
|
|
ticket per control number, and some issue one ticket covering a whole mark --
|
|
one drawing, one sheet, a QUANITY field and several control numbers printed
|
|
on it. The same export folder read under the second arrangement gives one
|
|
filename per mark, and counting those filenames as pieces would return a
|
|
confident 1 for a mark that is really seven. That is exactly the
|
|
`assembly_instances` failure in different clothes, and it is why the ranking
|
|
above stops being a constant here.
|
|
|
|
So the fact is DECLARED, in `nomenclature.ticket_unit`, and read out of the
|
|
knowledge layers rather than compiled in. `takeoffgather.run` does the read,
|
|
with the literal path, and hands the value to `counting_plan()` below, which
|
|
turns it into the ranking this run is allowed to use and one sentence saying
|
|
why. Three states, and all three are printed:
|
|
|
|
control_number one ticket per control number. A ticket IS a piece, so
|
|
all three sources rank exactly as `ORDER` says.
|
|
piece_mark one ticket covers the mark. A ticket filename is not a
|
|
piece count, so `ticket_files` is WITHDRAWN and the
|
|
ticket-against-model cross-check stands down with a
|
|
sentence rather than reporting every mark as a
|
|
disagreement.
|
|
not declared nobody has said, so this run does not know which of the
|
|
two it is looking at. `ticket_files` is withdrawn HERE
|
|
TOO, and the refusal names the key to declare.
|
|
|
|
That last one is the only judgement call in this file and it goes the way
|
|
this codebase's other unset switches go -- `rails.left_rail_is_fixed`,
|
|
`takeoff.not_ordered`, `notes.instruction_views`. Falling back to
|
|
"one ticket is one piece" would be a guess that can only ever UNDER-count,
|
|
and it would arrive wearing a source's name (`ticket_files`) rather than
|
|
`assumed_one`. Withdrawing the source instead costs nothing that was known:
|
|
the marks it covered fall to `assumed_one`, which the report already prints
|
|
as an undercount in those words. A number that says "could not judge" is a
|
|
different thing from a number that says "judged, found one", and the whole
|
|
point of this module is that the two must never look alike.
|
|
|
|
ONE PARSER FOR ONE RULE
|
|
------------------------
|
|
A control field may hold a range: `50120-50123` is four pieces, not one
|
|
token. `jobrecon.expand_control_field` already implements that expansion,
|
|
with its dash normalisation and its 500-wide sanity cap, and this module
|
|
CALLS IT rather than reimplementing it. Three implementations of one parsing
|
|
rule is how they come to disagree.
|
|
"""
|
|
# pylint: disable=invalid-name
|
|
import re
|
|
|
|
from vibe import jobrecon
|
|
|
|
# <job>-<control>_<mark>.pdf, with an optional revision suffix on the mark.
|
|
# The older export was named by mark alone, which is exactly how one file per
|
|
# mark came to be mistaken for one piece per mark; a name with no control
|
|
# number in it is therefore IGNORED rather than counted as one piece.
|
|
TICKET_RE = re.compile(r'^(?P<job>[^-_]+)-(?P<control>[^_]+)_(?P<mark>.+)$')
|
|
|
|
# Precedence, best first. A mark found by two sources takes the better one
|
|
# and the report can still show what the other said.
|
|
#
|
|
# It is the ranking FOR A JOB WHERE ONE TICKET IS ONE PIECE. Which sources a
|
|
# given job may actually use is `counting_plan()`, off the declared
|
|
# `nomenclature.ticket_unit`; this tuple is the full ranking those plans are
|
|
# drawn from, and no caller should reach for it instead of a plan.
|
|
ORDER = ('control_numbers', 'ticket_files', 'assembly_instances')
|
|
|
|
# `nomenclature.ticket_unit` -- what one exported ticket covers. A JOB fact:
|
|
# the same plant can do it either way and the manual shows both, so it is
|
|
# declared in the job layer and never inferred here.
|
|
TICKET_UNIT_KEY = 'nomenclature.ticket_unit'
|
|
CONTROL_NUMBER = 'control_number'
|
|
PIECE_MARK = 'piece_mark'
|
|
|
|
# Which sources are a PIECE COUNT under each declared unit. Only
|
|
# `ticket_files` turns on this: a control number off a model element is one
|
|
# physical piece however the plant tickets it, and a duplicated assembly name
|
|
# is a duplicated piece either way.
|
|
NO_TICKETS = tuple(b for b in ORDER if b != 'ticket_files')
|
|
UNIT_SOURCES = {
|
|
CONTROL_NUMBER: ORDER,
|
|
PIECE_MARK: NO_TICKETS,
|
|
}
|
|
|
|
|
|
def _stem(filename):
|
|
name = (filename or '').replace('\\', '/').rsplit('/', 1)[-1]
|
|
if '.' in name:
|
|
name = name.rsplit('.', 1)[0]
|
|
return name
|
|
|
|
|
|
def parse_ticket_name(filename):
|
|
"""(mark, control) from an exported ticket filename, or (None, None).
|
|
|
|
A revision suffix belongs to the ticket, not to the piece: two tickets
|
|
for the same control number are one piece, so the control number is what
|
|
is counted and the mark keeps its revision stripped.
|
|
|
|
A CONTROL NUMBER CONTAINS AT LEAST ONE DIGIT, and that one rule is the
|
|
difference between reading an export folder and reading a ticket set.
|
|
The folder also holds the detail sheets, whose names take the identical
|
|
three-part shape -- `<job>-SHEET_X4-1.pdf` parses out a "control number"
|
|
of SHEET and a "mark" of X4-1 -- and twelve of those were being counted
|
|
as marks. They cost nothing in quantity, because no real mark is named
|
|
after a detail sheet, but they inflated what the source claimed to know
|
|
and they were caught by `join_quality` rather than by anyone reading the
|
|
code. Letters in a control number are still fine; letters ALONE are not.
|
|
"""
|
|
found = TICKET_RE.match(_stem(filename))
|
|
if found is None:
|
|
return (None, None)
|
|
control = found.group('control').strip()
|
|
if not any(ch.isdigit() for ch in control):
|
|
return (None, None)
|
|
mark = found.group('mark').strip()
|
|
mark = re.sub(r'[_-]R\d+$', '', mark, flags=re.IGNORECASE)
|
|
return (mark or None, control or None)
|
|
|
|
|
|
def unparsed_ticket_files(filenames):
|
|
"""The filenames `from_ticket_files` could not read. Never silent.
|
|
|
|
A folder of exports is somebody's directory and holds whatever they put
|
|
in it. What was skipped is a fact about the source's coverage, so it is
|
|
returned rather than assumed to be rubbish.
|
|
"""
|
|
return sorted(name for name in (filenames or [])
|
|
if parse_ticket_name(name) == (None, None))
|
|
|
|
|
|
def from_ticket_files(filenames):
|
|
"""{mark: (distinct control numbers, 'ticket_files')}.
|
|
|
|
Distinct, not total: a mark reissued after a revision has two files and
|
|
one piece.
|
|
"""
|
|
seen = {}
|
|
for filename in filenames or []:
|
|
mark, control = parse_ticket_name(filename)
|
|
if mark and control:
|
|
seen.setdefault(mark, set()).add(control)
|
|
return dict((mark, (len(seen[mark]), 'ticket_files')) for mark in seen)
|
|
|
|
|
|
def from_assembly_names(names, min_instances=2):
|
|
"""{mark: (instances, 'assembly_instances')} -- DUPLICATED NAMES ONLY.
|
|
|
|
Wants the names WITH duplicates, because the duplication is the whole
|
|
signal. A name that appears once is NOT evidence that the mark is one
|
|
piece; it is the absence of evidence, and the two must not produce the
|
|
same output.
|
|
|
|
That distinction is not academic. Measured on this job through the
|
|
bridge: 755 assembly instances, 755 distinct names, zero duplicates. An
|
|
earlier version of this function returned 1 for each of them, every mark
|
|
got a confident `assembly_instances` basis, and every per_job figure
|
|
silently equalled its per_mark figure -- a whole-job undercount that
|
|
looked sourced. `min_instances=2` is what makes that impossible: with no
|
|
duplicates this returns {} and every mark falls through to
|
|
`assumed_one`, which the report prints as an undercount in those words.
|
|
"""
|
|
out = {}
|
|
for name in names or []:
|
|
if not name:
|
|
continue
|
|
out[name] = out.get(name, 0) + 1
|
|
return dict((name, (out[name], 'assembly_instances')) for name in out
|
|
if out[name] >= min_instances)
|
|
|
|
|
|
def from_piece_records(pieces):
|
|
"""({mark: (pieces, 'control_numbers')}, unreadable) from model elements.
|
|
|
|
`pieces` is the list `jobreconscan.gather` produces -- the same data
|
|
contract the job reconciliation eats, deliberately, so that the takeoff
|
|
and the reconciliation cannot end up disagreeing about how many pieces
|
|
the job has. Each record is {'control', 'mark', 'id', ...}.
|
|
|
|
Control numbers are counted DISTINCT per mark, because a control number
|
|
shared by several elements is still one piece: the reconciliation reports
|
|
2480 control-numbered elements resolving to 1776 pieces, and the
|
|
difference between those two numbers is exactly that sharing.
|
|
|
|
A control field is expanded through `jobrecon.expand_control_field`, so a
|
|
range is four pieces rather than one token and an unreadable slot is
|
|
RETURNED rather than dropped. An element whose mark is missing cannot be
|
|
attributed to a mark and is returned as unreadable too -- it is a real
|
|
piece this multiplier cannot place, and losing it silently would
|
|
under-count in exactly the way this module exists to prevent.
|
|
"""
|
|
by_mark = {}
|
|
unreadable = []
|
|
for piece in pieces or []:
|
|
mark = (piece.get('mark') or '').strip()
|
|
controls, bad = jobrecon.expand_control_field(piece.get('control'))
|
|
for token in bad:
|
|
unreadable.append({'id': piece.get('id'), 'mark': mark or None,
|
|
'token': token,
|
|
'why': 'not readable as a control number'})
|
|
if not controls:
|
|
continue
|
|
if not mark:
|
|
unreadable.append({'id': piece.get('id'), 'mark': None,
|
|
'token': ', '.join(controls),
|
|
'why': 'the element carries no mark, so its '
|
|
'pieces cannot be attributed'})
|
|
continue
|
|
by_mark.setdefault(mark, set()).update(controls)
|
|
return (dict((mark, (len(by_mark[mark]), 'control_numbers'))
|
|
for mark in by_mark),
|
|
unreadable)
|
|
|
|
|
|
def from_control_numbers(by_mark):
|
|
"""{mark: (distinct control numbers, 'control_numbers')}.
|
|
|
|
The already-grouped form, for a caller that has the mapping in hand.
|
|
`from_piece_records` is the one that takes the reconciliation's own
|
|
element records and does the grouping and the range expansion.
|
|
|
|
`by_mark` is {mark: [control numbers]} read off the model. Distinct
|
|
again, and an empty list is NOT a zero: a mark whose control numbers
|
|
could not be read is left out entirely so it falls through to the next
|
|
source rather than being multiplied by nothing.
|
|
"""
|
|
out = {}
|
|
for mark in by_mark or {}:
|
|
controls = set(str(c).strip() for c in (by_mark[mark] or [])
|
|
if str(c).strip())
|
|
if controls:
|
|
out[mark] = (len(controls), 'control_numbers')
|
|
return out
|
|
|
|
|
|
def counting_plan(unit, sources=None):
|
|
"""Which sources may count pieces, given the DECLARED `ticket_unit`.
|
|
|
|
`unit` is the value of `nomenclature.ticket_unit` as the knowledge layers
|
|
hold it -- read by the caller, so this module needs no knowledge access
|
|
and stays testable with a string. `sources` is {basis: {mark: (n, basis)}}
|
|
and is optional: it is used only to say what withdrawing a source
|
|
actually COST on this job, which is the difference between a rule and a
|
|
finding.
|
|
|
|
Returns a plan the report prints whole:
|
|
|
|
unit what was declared, or None
|
|
declared somebody said
|
|
recognised and this module knows what they said
|
|
order the bases that may count pieces, best first
|
|
withdrawn the bases that may not, and why is in `reason`
|
|
withdrawn_marks marks a withdrawn source knew
|
|
cost marks that lose their ONLY source to the withdrawal
|
|
reason one sentence, ALWAYS, including when nothing changed
|
|
refusal the same sentence when the run COULD NOT JUDGE --
|
|
undeclared or unrecognised. None when somebody
|
|
declared it, whichever way they declared it.
|
|
|
|
An unrecognised value is a refusal and not a fallback. A key set to
|
|
something nobody implemented is somebody's intention, misspelt; reading
|
|
it as the default would silently do the opposite of what they asked for
|
|
on exactly the jobs where it matters.
|
|
"""
|
|
unit = (str(unit).strip() or None) if unit is not None else None
|
|
order = UNIT_SOURCES.get(unit)
|
|
plan = {
|
|
'key': TICKET_UNIT_KEY,
|
|
'unit': unit,
|
|
'declared': unit is not None,
|
|
'recognised': order is not None,
|
|
'order': tuple(order if order is not None else NO_TICKETS),
|
|
'reason': None,
|
|
'refusal': None,
|
|
}
|
|
plan['withdrawn'] = tuple(b for b in ORDER if b not in plan['order'])
|
|
|
|
if unit == CONTROL_NUMBER:
|
|
plan['reason'] = (
|
|
'`{0}: {1}` -- this job issues one ticket per control number, so '
|
|
'one exported ticket is one piece and the ticket files count as '
|
|
'pieces, ranked behind the model\'s own control '
|
|
'numbers.'.format(TICKET_UNIT_KEY, CONTROL_NUMBER))
|
|
elif unit == PIECE_MARK:
|
|
plan['reason'] = (
|
|
'`{0}: {1}` -- one ticket covers a whole mark, so a ticket '
|
|
'filename names one control number and NOT one piece. The '
|
|
'exported tickets were therefore NOT counted as pieces, and the '
|
|
'ticket-against-model cross-check stands down rather than '
|
|
'reporting every mark as a disagreement. Marks that only the '
|
|
'tickets covered are counted once and shown as an '
|
|
'undercount.'.format(TICKET_UNIT_KEY, PIECE_MARK))
|
|
elif unit is None:
|
|
plan['refusal'] = (
|
|
'`{0}` is not declared, so this run does not know whether one '
|
|
'exported ticket is one piece or a whole mark of them. The '
|
|
'exported tickets were NOT used as a piece count: taking one '
|
|
'ticket as one piece would under-count a job ticketed the other '
|
|
'way, and it would do it wearing a source\'s name instead of '
|
|
'`assumed_one`. Declare `{0}: {1}` (one ticket per control '
|
|
'number) or `{0}: {2}` (one ticket covering the mark) in the job '
|
|
'knowledge file and press again.'.format(
|
|
TICKET_UNIT_KEY, CONTROL_NUMBER, PIECE_MARK))
|
|
else:
|
|
plan['refusal'] = (
|
|
'`{0}` is set to "{1}", which this takeoff does not recognise -- '
|
|
'it knows `{2}` and `{3}`. A value nobody can read is not a '
|
|
'declaration, so the exported tickets were NOT used as a piece '
|
|
'count and nothing was assumed on your behalf.'.format(
|
|
TICKET_UNIT_KEY, unit, CONTROL_NUMBER, PIECE_MARK))
|
|
if plan['reason'] is None:
|
|
plan['reason'] = plan['refusal']
|
|
|
|
known = {}
|
|
for basis in (sources or {}):
|
|
for mark in (sources[basis] or {}):
|
|
known.setdefault(mark, set()).add(basis)
|
|
plan['withdrawn_marks'] = len(
|
|
[m for m in known if known[m] & set(plan['withdrawn'])])
|
|
plan['cost'] = len([m for m in known
|
|
if not known[m] & set(plan['order'])])
|
|
return plan
|
|
|
|
|
|
def merge(*sources, **kwargs):
|
|
"""Fold the sources together, best basis first. Later never overrides.
|
|
|
|
`order` (keyword) is the ranking this job is allowed to use --
|
|
`counting_plan()['order']`. It defaults to `ORDER`, all three sources, so
|
|
a caller that has not asked the question gets exactly the behaviour this
|
|
function has always had.
|
|
|
|
A basis NOT in `order` is DROPPED, never ranked last: a source that is not
|
|
a piece count on this job must not become one merely because nothing
|
|
better covered the mark. Ranking it 99th would let it through on exactly
|
|
the marks where nothing can check it, which is where a wrong multiplier
|
|
does its damage.
|
|
"""
|
|
order = kwargs.pop('order', None) or ORDER
|
|
if kwargs:
|
|
raise TypeError('merge() got an unexpected keyword argument {0}'
|
|
.format(sorted(kwargs)[0]))
|
|
rank = dict((basis, i) for i, basis in enumerate(order))
|
|
out = {}
|
|
for source in sources:
|
|
for mark in source or {}:
|
|
count, basis = source[mark]
|
|
if basis not in rank:
|
|
continue
|
|
have = out.get(mark)
|
|
if have is None or rank[basis] < rank.get(have[1], 99):
|
|
out[mark] = (int(count), basis)
|
|
return out
|
|
|
|
|
|
def disagreements(a, b):
|
|
"""Marks where two sources give different counts. Worth printing.
|
|
|
|
A mismatch between the exported tickets and whatever the model says is
|
|
the cheapest possible audit of the multiplier, and it is the one number
|
|
in this takeoff that nothing else can check.
|
|
"""
|
|
out = []
|
|
for mark in sorted(set(a or {}) & set(b or {})):
|
|
if a[mark][0] != b[mark][0]:
|
|
out.append({'mark': mark,
|
|
'left': a[mark][0], 'left_basis': a[mark][1],
|
|
'right': b[mark][0], 'right_basis': b[mark][1]})
|
|
return out
|
|
|
|
|
|
def join_quality(multipliers, marks):
|
|
"""Do the multiplier's marks and the takeoff's marks name the same things?
|
|
|
|
The takeoff keys on the ASSEMBLY NAME; the multiplier keys on the piece
|
|
MARK read off a framing element. Measured, those are the same string --
|
|
an assembly named for its mark -- but "measured" is not "guaranteed", and
|
|
the failure if they ever diverge is silent and total: every mark falls
|
|
through to `assumed_one` and the job is under-ordered wholesale while the
|
|
report shows a source that found plenty.
|
|
|
|
So the join is counted, both ways, and printed. `unmatched_source` is the
|
|
tell: a source that found hundreds of marks the takeoff has never heard
|
|
of is not a source, it is a different naming scheme.
|
|
"""
|
|
marks = set(marks or [])
|
|
source = set(multipliers or {})
|
|
return {
|
|
'takeoff_marks': len(marks),
|
|
'source_marks': len(source),
|
|
'matched': len(marks & source),
|
|
'unsourced': sorted(marks - source),
|
|
'unmatched_source': sorted(source - marks),
|
|
}
|
|
|
|
|
|
def summarise(multipliers, marks=None):
|
|
"""The multiplier, as a reader has to be able to check it.
|
|
|
|
`marks` is every mark the takeoff covered, so that a mark with no
|
|
multiplier is counted into `assumed_one` rather than vanishing.
|
|
"""
|
|
marks = list(marks) if marks is not None else list(multipliers or {})
|
|
by_basis = {}
|
|
total = 0
|
|
for mark in marks:
|
|
count, basis = (multipliers or {}).get(mark, (1, 'assumed_one'))
|
|
by_basis[basis] = by_basis.get(basis, 0) + 1
|
|
total += int(count)
|
|
return {
|
|
'marks': len(marks),
|
|
'pieces': total,
|
|
'multiplier': (round(total / float(len(marks)), 3) if marks else None),
|
|
'by_basis': by_basis,
|
|
'assumed_one': by_basis.get('assumed_one', 0),
|
|
}
|