will
25592fa6c2
ci / test (push) Successful in 6m14s
History-free snapshot of the private development repository at 77a82eb. Code, docs and the public catalogue only: no plant or job knowledge layers, no client, job or producer names (tools/scrub.py clean against the widened term list; a second word-level grep clean). The intro deck is left out because its slide images cannot be checked for job drawings.
598 lines
24 KiB
Python
598 lines
24 KiB
Python
# -*- coding: utf-8 -*-
|
|
"""The takeoff: what a job actually contains, per piece and in total. No Revit.
|
|
|
|
WHAT THIS IS BUILT ON, AND WHY IT IS NOT ANOTHER MODEL SWEEP
|
|
-------------------------------------------------------------
|
|
`vibe.reinforceindex` already walks every assembly, resolves its members and
|
|
records a `families` dict -- family name -> instance count -- on every piece
|
|
record. That is a takeoff's raw material and nothing was reading it. This
|
|
module takes the index cache as its INPUT and adds no model access of its
|
|
own, so a whole-job takeoff costs whatever `reinforceindex.read()` costs and
|
|
nothing more.
|
|
|
|
The consequence is a limit, stated once here because every honest line of the
|
|
output depends on it: THIS MODULE CAN ONLY SEE WHAT THE INDEX RECORDED.
|
|
Three limits follow and each is reported rather than papered over.
|
|
|
|
1. The index builds only ELIGIBLE pieces -- reinforced, not on hold. A job
|
|
has more assemblies than that. The caller passes `population` so the
|
|
report can say what fraction of the job it covered instead of implying
|
|
the whole of it.
|
|
|
|
2. The index keys `families` by FAMILY NAME, and at this plant a
|
|
reinforcing family is a BEND SHAPE, not a bar mark: one shape family
|
|
carries one bar mark on one piece and a different one on the next. So
|
|
hardware resolves to a part number off the family name and REINFORCING
|
|
LARGELY DOES NOT. Those instances are counted and reported as
|
|
`bar_shape` with the mark unresolved -- never dropped, and never
|
|
silently rolled into a bar mark that was guessed.
|
|
|
|
The fix is one field on the index record and it is written up in
|
|
`docs/takeoff.md`: a `marks` counter built from
|
|
`ticketqc._member_mark(member, know)`, which already returns the mark an
|
|
engineer says out loud on a bar and on an anchor alike. `piece_lines()`
|
|
uses it the moment it appears -- see `_bar_lines_from_marks` -- and
|
|
needs no other change here.
|
|
|
|
3. `features is None` on an index record means the piece could not be
|
|
MEASURED. It does not mean the piece is empty, and it does not stop the
|
|
takeoff: the `families` dict is recorded on nearly every failure path.
|
|
A record with no families dict at all is the real could-not-judge, and
|
|
it is listed as one with its reason rather than counted as zero.
|
|
|
|
THE COUNTING RULES THIS JOB HAS ALREADY PAID TO LEARN
|
|
------------------------------------------------------
|
|
**Control numbers.** An assembly is a MARK. A mark is a drawing. The pieces
|
|
on the floor are CONTROL NUMBERS. In one sampled ticket set 151 tickets
|
|
carried only 71 distinct marks, and one mark alone accounted for eighteen of
|
|
them. So a takeoff that counts marks under-orders every material in the job,
|
|
and this one multiplies -- but it prints both figures side by side, with the
|
|
multiplier and where the multiplier came from, because per-mark and per-job
|
|
answer different questions and a reader will want to check one against the
|
|
other. A multiplier the caller could not source is 1 with basis
|
|
`assumed_one`, and that is reported as an UNDERCOUNT, loudly, not as a
|
|
number.
|
|
|
|
This module never decides that multiplier and never reads a knowledge key for
|
|
it: `pieces` arrives per mark, already sourced and already ranked. What one
|
|
exported ticket covers -- `nomenclature.ticket_unit` -- is declared per job,
|
|
read in `takeoffgather.run`, and acted on by `takeoffcount.counting_plan`.
|
|
Both files carry the reasoning.
|
|
|
|
**Nesting.** A member whose `SuperComponent` is set is a child and still
|
|
appears in `GetMemberIds()`, so a family nested inside another is counted
|
|
twice: once as itself, once inside its parent's instance. This module does
|
|
not guess. A nesting DECLARED in `takeoff.nested_in` is honoured -- the child
|
|
leaves the order list for its own section. A nesting nobody declared is
|
|
DETECTED and reported as a suspicion for a person to rule on:
|
|
`suspected_nesting()` finds family pairs whose counts are equal on every
|
|
piece where both occur. It never subtracts on a suspicion. An automatic
|
|
correction that is wrong is invisible in the total; a suspicion that is wrong
|
|
costs one sentence to dismiss.
|
|
|
|
**Attached bars.** `drawings.annotation_covered_by_parent` already records
|
|
which bar marks arrive welded to which part. Those bars are real steel and
|
|
are always COUNTED. Whether they are ORDERED is a purchasing question, not a
|
|
geometry one, so it is a switch -- `takeoff.order_attached_bars_separately`,
|
|
default false -- and either way they appear in their own section with the
|
|
parent named. Silently either way is the one thing not on offer.
|
|
|
|
WHAT IT REFUSES TO COUNT
|
|
-------------------------
|
|
A family the rules cannot classify goes to `unclassified` with its name, its
|
|
count and how many pieces carry it. It is never bucketed into 'other'. Every
|
|
classification carries a `basis`: `declared` when a knowledge key named the
|
|
family outright, `rule:<name>` when a pattern decided it, and the report
|
|
groups by basis so every inference this module made is on the page.
|
|
"""
|
|
# pylint: disable=invalid-name
|
|
import re
|
|
|
|
# Every class this takeoff knows. `unclassified` is a real answer, not a
|
|
# failure mode, and it is the only one whose contents are a work list.
|
|
BAR = 'bar' # reinforcing, mark resolved
|
|
BAR_SHAPE = 'bar_shape' # reinforcing, family is a bend shape, mark unknown
|
|
STRAND = 'strand'
|
|
MESH = 'mesh'
|
|
HARDWARE = 'hardware' # an orderable part with a part number
|
|
CAST = 'cast' # separately modelled cast concrete, not an insert
|
|
NESTED = 'nested' # declared as a child of another family
|
|
NOT_ORDERED = 'not_ordered' # the piece itself, or a modelling construct
|
|
UNCLASSIFIED = 'unclassified'
|
|
|
|
ORDERABLE = (BAR, BAR_SHAPE, STRAND, MESH, HARDWARE, CAST)
|
|
|
|
# Defaults live here rather than in `knowledge.DEFAULTS` so that this module
|
|
# owns them and no other file has to change to add a takeoff. Any knowledge
|
|
# layer overrides them under `takeoff:`; the plant or job layer is where the
|
|
# real prefixes belong, and `docs/takeoff.md` carries the block to paste.
|
|
#
|
|
# `not_ordered` DEFAULTS TO EMPTY on purpose. A default list of things to
|
|
# ignore is a default under-count, and the families it would name -- the
|
|
# panel itself, a modelling zone -- are plant nomenclature that this module
|
|
# has no business knowing. Left empty they surface as `unclassified`, which
|
|
# is the loud version of the same fact and the one a person can act on.
|
|
DEFAULTS = {
|
|
'bar_prefixes': ['SR'],
|
|
'part_prefixes': ['SP', 'P'],
|
|
'shape_markers': ['(SCHEDULED)'],
|
|
'not_ordered': [],
|
|
'family_class': {},
|
|
'nested_in': {},
|
|
'attached_to': {},
|
|
'order_attached_bars_separately': False,
|
|
}
|
|
|
|
|
|
def settings(know, get=None):
|
|
"""The `takeoff:` block, defaults filled in. `get` is `knowledge.get`.
|
|
|
|
`get` is passed in rather than imported so this module needs nothing but
|
|
the standard library, which is what keeps it testable off a model.
|
|
"""
|
|
out = dict(DEFAULTS)
|
|
for key in DEFAULTS:
|
|
if get is None:
|
|
value = ((know or {}).get('takeoff') or {}).get(key)
|
|
else:
|
|
value = get(know, 'takeoff.' + key)
|
|
if value is not None:
|
|
out[key] = value
|
|
return out
|
|
|
|
|
|
# ------------------------------------------------------------ classification
|
|
|
|
def _up(text):
|
|
return (text or '').upper()
|
|
|
|
|
|
def _prefixed(name, prefixes):
|
|
"""The longest `<PREFIX><digits>` token in `name`, or None.
|
|
|
|
Longest match wins so a part in an `SP` series is never read as the `P`
|
|
series inside it, and the lookbehind is what stops an embed family whose
|
|
name wraps a part number from resolving to anything but that number.
|
|
"""
|
|
up = _up(name)
|
|
best = None
|
|
for prefix in sorted([_up(p) for p in (prefixes or [])],
|
|
key=len, reverse=True):
|
|
if not prefix:
|
|
continue
|
|
found = re.search(r'(?<![A-Z0-9])' + re.escape(prefix) + r'(\d+)', up)
|
|
if found is None:
|
|
continue
|
|
token = prefix + found.group(1)
|
|
if best is None or len(token) > len(best):
|
|
best = token
|
|
return best
|
|
|
|
|
|
def classify(family, cfg, is_strand=None, is_mesh=None, is_cast=None):
|
|
"""(class, key, basis) for one family name.
|
|
|
|
`key` is what a person orders by: a part number, a bar mark, a bend
|
|
shape, or -- when nothing resolved -- the family name itself, so that an
|
|
unclassified row still names the thing it could not classify.
|
|
|
|
Order matters and is deliberate:
|
|
1. a family named outright in knowledge beats every pattern;
|
|
2. a declared nesting, so a child never reaches the order list;
|
|
3. a declared not-ordered family;
|
|
4. strand, mesh and cast concrete, which the existing `families:`
|
|
rules already decide and which this module must not re-decide;
|
|
5. a bend-shape marker, BEFORE the bar-prefix rule -- a shape family
|
|
has no mark in its name and must not borrow one;
|
|
6. bar prefixes, then part prefixes;
|
|
7. unclassified.
|
|
"""
|
|
up = _up(family)
|
|
|
|
declared = cfg.get('family_class') or {}
|
|
for key in declared:
|
|
if _up(key) == up:
|
|
return (str(declared[key]), family, 'declared')
|
|
|
|
for child in (cfg.get('nested_in') or {}):
|
|
if _up(child) == up:
|
|
return (NESTED, family, 'declared')
|
|
|
|
for token in (cfg.get('not_ordered') or []):
|
|
if _up(token) and _up(token) in up:
|
|
return (NOT_ORDERED, family, 'declared')
|
|
|
|
if is_strand is not None and is_strand(family):
|
|
return (STRAND, family, 'rule:strand')
|
|
if is_mesh is not None and is_mesh(family):
|
|
return (MESH, family, 'rule:mesh')
|
|
if is_cast is not None and is_cast(family):
|
|
return (CAST, family, 'rule:cast')
|
|
|
|
for marker in (cfg.get('shape_markers') or []):
|
|
if _up(marker) and _up(marker) in up:
|
|
return (BAR_SHAPE, family, 'rule:shape_marker')
|
|
|
|
mark = _prefixed(family, cfg.get('bar_prefixes'))
|
|
if mark:
|
|
return (BAR, mark, 'rule:bar_prefix')
|
|
|
|
part = _prefixed(family, cfg.get('part_prefixes'))
|
|
if part:
|
|
return (HARDWARE, part, 'rule:part_prefix')
|
|
|
|
return (UNCLASSIFIED, family, 'none')
|
|
|
|
|
|
# ------------------------------------------------------------- attached bars
|
|
|
|
def attached_parents(cfg, know, get=None):
|
|
"""{BAR MARK: [parent marks]} -- bars that ship welded to a part.
|
|
|
|
Read from `drawings.annotation_covered_by_parent`, which the drawing
|
|
checks already maintain, so a bar cannot be attached for the purpose of
|
|
one check and loose for another. `takeoff.attached_to` overrides it for a
|
|
job where the two genuinely differ.
|
|
"""
|
|
if get is None:
|
|
source = ((know or {}).get('drawings') or {}).get(
|
|
'annotation_covered_by_parent') or {}
|
|
else:
|
|
source = get(know, 'drawings.annotation_covered_by_parent', {}) or {}
|
|
override = cfg.get('attached_to')
|
|
if override:
|
|
source = override
|
|
out = {}
|
|
for child in source:
|
|
parents = source[child]
|
|
if isinstance(parents, (list, tuple)):
|
|
out[_up(child)] = [str(p) for p in parents]
|
|
elif parents:
|
|
out[_up(child)] = [str(parents)]
|
|
return out
|
|
|
|
|
|
# ----------------------------------------------------------------- one piece
|
|
|
|
def piece_lines(record, cfg, attached=None, is_strand=None, is_mesh=None,
|
|
pieces=1, is_cast=None):
|
|
"""Every takeoff line for one index record, per-mark and per-job.
|
|
|
|
`pieces` is the control-number multiplier: how many physical pieces this
|
|
one mark issues. `per_mark` is what one of them contains; `per_job` is
|
|
what the job needs.
|
|
"""
|
|
attached = attached or {}
|
|
lines = []
|
|
families = (record or {}).get('families') or {}
|
|
marks = (record or {}).get('marks')
|
|
|
|
skip = set()
|
|
if marks:
|
|
lines.extend(_bar_lines_from_marks(
|
|
record, marks, families, cfg, attached, pieces))
|
|
skip = set(_up(f) for f in _shape_families(families, cfg))
|
|
|
|
for family in sorted(families):
|
|
if _up(family) in skip:
|
|
continue
|
|
klass, key, basis = classify(family, cfg, is_strand, is_mesh, is_cast)
|
|
count = int(families[family] or 0)
|
|
lines.append(_line(record, klass, key, family, basis, count, pieces,
|
|
attached))
|
|
return sorted(lines, key=lambda r: (r['class'], r['key']))
|
|
|
|
|
|
def _shape_families(families, cfg):
|
|
"""Families that are bend shapes -- the ones a `marks` counter replaces."""
|
|
out = []
|
|
for family in families:
|
|
for marker in (cfg.get('shape_markers') or []):
|
|
if _up(marker) and _up(marker) in _up(family):
|
|
out.append(family)
|
|
break
|
|
return out
|
|
|
|
|
|
def _bar_lines_from_marks(record, marks, families, cfg, attached, pieces):
|
|
"""Bar lines built from a per-instance mark counter, when one exists.
|
|
|
|
THIS IS THE PATH THAT DOES NOT EXIST YET. `reinforceindex.extract` does
|
|
not record `marks`; the moment it does, every `bar_shape` row in the
|
|
output turns into a real bar mark and nothing else in this module
|
|
changes. Written now so that the index change is a one-field change and
|
|
not a rewrite, and exercised by the tests with a synthetic record.
|
|
|
|
A mark counter carries hardware marks too, because a hardware family and
|
|
its mark are the same string. Only marks that read as a BAR and are not
|
|
already a family we counted are taken from here; anything else would be
|
|
counted once as a family and once again as a mark.
|
|
"""
|
|
known = set(_up(f) for f in families)
|
|
lines = []
|
|
for mark in sorted(marks):
|
|
bar = _prefixed(mark, cfg.get('bar_prefixes'))
|
|
if not bar or _up(bar) in known:
|
|
continue
|
|
lines.append(_line(record, BAR, bar, None, 'index:marks',
|
|
int(marks[mark] or 0), pieces, attached))
|
|
return lines
|
|
|
|
|
|
def _line(record, klass, key, family, basis, count, pieces, attached):
|
|
parents = attached.get(_up(key)) if klass in (BAR, BAR_SHAPE) else None
|
|
return {
|
|
'piece': (record or {}).get('name'),
|
|
'design_number': (record or {}).get('design_number'),
|
|
'class': klass,
|
|
'key': key,
|
|
'family': family,
|
|
'basis': basis,
|
|
'per_mark': count,
|
|
'pieces': int(pieces),
|
|
'per_job': count * int(pieces),
|
|
'attached_to': list(parents) if parents else None,
|
|
}
|
|
|
|
|
|
# ------------------------------------------------------------------- the job
|
|
|
|
def build(cache, cfg, multipliers=None, attached=None, is_strand=None,
|
|
is_mesh=None, population=None, is_cast=None):
|
|
"""The whole takeoff. `cache` is a `reinforcecore` cache, read, not touched.
|
|
|
|
`multipliers` is {piece name: (count, basis)}. A piece missing from it is
|
|
1 with basis `assumed_one` and lands in `undercounted`, because a
|
|
multiplier nobody could source is not a multiplier of one.
|
|
"""
|
|
multipliers = multipliers or {}
|
|
attached = attached or {}
|
|
pieces = (cache or {}).get('pieces') or {}
|
|
|
|
by_piece = {}
|
|
unjudged = []
|
|
undercounted = []
|
|
all_lines = []
|
|
|
|
for name in sorted(pieces):
|
|
record = pieces[name] or {}
|
|
count, basis = multipliers.get(name, (1, 'assumed_one'))
|
|
count = int(count or 1)
|
|
if basis == 'assumed_one':
|
|
undercounted.append(name)
|
|
families = record.get('families') or {}
|
|
if not families:
|
|
unjudged.append({
|
|
'piece': name,
|
|
'reason': record.get('error') or 'no families recorded',
|
|
})
|
|
by_piece[name] = {'piece': name, 'pieces': count,
|
|
'multiplier_basis': basis, 'lines': [],
|
|
'judged': False,
|
|
'design_number': record.get('design_number')}
|
|
continue
|
|
lines = piece_lines(record, cfg, attached, is_strand, is_mesh,
|
|
pieces=count, is_cast=is_cast)
|
|
by_piece[name] = {
|
|
'piece': name,
|
|
'design_number': record.get('design_number'),
|
|
'pieces': count,
|
|
'multiplier_basis': basis,
|
|
'lines': lines,
|
|
'judged': True,
|
|
}
|
|
all_lines.extend(lines)
|
|
|
|
return {
|
|
'by_piece': by_piece,
|
|
'totals': roll_up(all_lines),
|
|
'unclassified': unclassified_rows(all_lines),
|
|
'unjudged': unjudged,
|
|
'undercounted': sorted(undercounted),
|
|
'nesting_suspects': suspected_nesting(pieces),
|
|
'attached': attached,
|
|
'order_attached_separately': bool(
|
|
cfg.get('order_attached_bars_separately')),
|
|
'coverage': coverage(cache, population, unjudged),
|
|
'bases': bases(all_lines),
|
|
'pieces_on_the_floor': sum(
|
|
by_piece[n]['pieces'] for n in by_piece),
|
|
}
|
|
|
|
|
|
def roll_up(lines):
|
|
"""Job totals, keyed (class, key). Both figures, always both."""
|
|
out = {}
|
|
for line in lines:
|
|
slot = (line['class'], line['key'])
|
|
row = out.get(slot)
|
|
if row is None:
|
|
row = {'class': line['class'], 'key': line['key'],
|
|
'per_mark_total': 0, 'per_job_total': 0,
|
|
'marks': 0, 'families': [], 'bases': [],
|
|
'attached_to': line.get('attached_to')}
|
|
out[slot] = row
|
|
row['per_mark_total'] += line['per_mark']
|
|
row['per_job_total'] += line['per_job']
|
|
row['marks'] += 1
|
|
if line.get('family') and line['family'] not in row['families']:
|
|
row['families'].append(line['family'])
|
|
if line['basis'] not in row['bases']:
|
|
row['bases'].append(line['basis'])
|
|
rows = list(out.values())
|
|
for row in rows:
|
|
row['families'].sort()
|
|
row['bases'].sort()
|
|
row['multiplier'] = (
|
|
round(row['per_job_total'] / float(row['per_mark_total']), 3)
|
|
if row['per_mark_total'] else None)
|
|
rows.sort(key=lambda r: (r['class'], -r['per_job_total'], r['key']))
|
|
return rows
|
|
|
|
|
|
def order_rows(totals, order_attached_separately=False):
|
|
"""(order, ships_with_parent, not_ordered) -- the split a buyer needs.
|
|
|
|
An attached bar is always counted. Whether it appears on the ORDER is the
|
|
switch; either way it appears, with its parent named.
|
|
"""
|
|
order, ships, other = [], [], []
|
|
for row in totals:
|
|
if row['class'] not in ORDERABLE:
|
|
other.append(row)
|
|
elif row.get('attached_to') and not order_attached_separately:
|
|
ships.append(row)
|
|
else:
|
|
order.append(row)
|
|
return order, ships, other
|
|
|
|
|
|
def unclassified_rows(lines):
|
|
"""Every family no rule could place, with what it would have cost.
|
|
|
|
Never merged into an 'other' row. An under-count that looks tidy is worse
|
|
than a messy honest one, so this is the section that has to be read.
|
|
"""
|
|
out = {}
|
|
for line in lines:
|
|
if line['class'] != UNCLASSIFIED:
|
|
continue
|
|
row = out.get(line['key'])
|
|
if row is None:
|
|
row = {'family': line['key'], 'per_mark_total': 0,
|
|
'per_job_total': 0, 'pieces': 0, 'examples': []}
|
|
out[line['key']] = row
|
|
row['per_mark_total'] += line['per_mark']
|
|
row['per_job_total'] += line['per_job']
|
|
row['pieces'] += 1
|
|
if line.get('piece') and len(row['examples']) < 5:
|
|
row['examples'].append(line['piece'])
|
|
rows = list(out.values())
|
|
rows.sort(key=lambda r: (-r['per_job_total'], r['family']))
|
|
return rows
|
|
|
|
|
|
def bases(lines):
|
|
"""How many rows each basis decided. The inference budget, on the page."""
|
|
out = {}
|
|
for line in lines:
|
|
out[line['basis']] = out.get(line['basis'], 0) + 1
|
|
return out
|
|
|
|
|
|
def coverage(cache, population, unjudged):
|
|
"""What the takeoff saw, against what the job has. Never implied.
|
|
|
|
`assemblies_in_model` is the caller's honest denominator -- the index
|
|
only ever builds reinforced, not-on-hold pieces, so `marks_not_indexed`
|
|
is a real hole in the job and not a rounding.
|
|
"""
|
|
pieces = (cache or {}).get('pieces') or {}
|
|
population = population or {}
|
|
indexed = len(pieces)
|
|
assemblies = population.get('assemblies')
|
|
return {
|
|
'indexed_marks': indexed,
|
|
'judged_marks': indexed - len(unjudged),
|
|
'unjudged_marks': len(unjudged),
|
|
'assemblies_in_model': assemblies,
|
|
'marks_not_indexed': (None if assemblies is None
|
|
else max(0, assemblies - indexed)),
|
|
'built_at': (cache or {}).get('built_at'),
|
|
'complete': bool(assemblies is not None and assemblies == indexed
|
|
and not unjudged),
|
|
}
|
|
|
|
|
|
# ---------------------------------------------------------- nesting suspects
|
|
|
|
def suspected_nesting(pieces, min_pieces=3, min_agreement=0.9,
|
|
min_distinct_counts=2):
|
|
"""Family pairs that look like child-inside-parent. A SUSPICION only.
|
|
|
|
Two families whose counts are EQUAL on the pieces where the first occurs
|
|
is the signature of a nested family: the child is counted once on its own
|
|
and once inside its parent's instance.
|
|
|
|
WHY A LOCKSTEP OF ONES IS NOT EVIDENCE. Two families that each appear
|
|
exactly once on the same forty pieces agree perfectly and mean nothing --
|
|
that is what a pair of unrelated parts on one panel type looks like. What
|
|
a nested family does is FOLLOW its parent's count as that count VARIES:
|
|
four of one, four of the other; eight and eight. So a pair only qualifies
|
|
when the child's count takes at least `min_distinct_counts` different
|
|
values across the pieces. Without that filter this list came back with
|
|
forty rows of coincidence and the three real pairs buried in them.
|
|
|
|
WHY THE AGREEMENT IS NOT REQUIRED TO BE PERFECT. The strongest declared
|
|
nesting on this job -- a bar that lives inside an embed whose family name
|
|
says "with bar" -- has 938 instances of the bar against 943 of the embed.
|
|
Five pieces disagree, for reasons that are somebody's to explain, and an
|
|
exact-equality test finds nothing at all. So agreement is a RATIO, it is
|
|
reported on every row, and `1.0` is visibly different from `0.95`.
|
|
|
|
DIRECTION IS OFTEN UNDECIDABLE FROM COUNTS ALONE, and saying so is the
|
|
honest part. When A and B occur on the same pieces in the same numbers,
|
|
the arithmetic cannot tell which sits inside which -- only that one of
|
|
the two is very probably being counted twice. Such a pair is reported
|
|
ONCE with `direction: undecided` rather than twice with a direction each
|
|
way, because two rows reads like two findings.
|
|
|
|
Nothing is subtracted on this evidence. It is printed so that a person
|
|
can move the pair into `takeoff.nested_in`, which IS acted on. An
|
|
automatic correction that is wrong is invisible in the total; a suspicion
|
|
that is wrong costs one sentence to dismiss.
|
|
"""
|
|
where = {}
|
|
for name in pieces:
|
|
record = pieces[name] or {}
|
|
for family in (record.get('families') or {}):
|
|
where.setdefault(family, {})[name] = int(
|
|
record['families'][family] or 0)
|
|
|
|
def agreement(child, parent):
|
|
child_on, parent_on = where[child], where[parent]
|
|
if not child_on:
|
|
return 0.0
|
|
same = 0
|
|
for piece in child_on:
|
|
if parent_on.get(piece) == child_on[piece]:
|
|
same += 1
|
|
return same / float(len(child_on))
|
|
|
|
names = sorted(where)
|
|
found = {}
|
|
for child in names:
|
|
if len(where[child]) < min_pieces:
|
|
continue
|
|
if len(set(where[child].values())) < min_distinct_counts:
|
|
continue
|
|
for parent in names:
|
|
if parent == child:
|
|
continue
|
|
if len(where[parent]) < len(where[child]) * min_agreement:
|
|
continue
|
|
forward = agreement(child, parent)
|
|
if forward < min_agreement:
|
|
continue
|
|
back = (len(where[parent]) >= min_pieces
|
|
and agreement(parent, child) >= min_agreement)
|
|
key = tuple(sorted((child, parent))) if back else (child, parent)
|
|
if key in found:
|
|
continue
|
|
first, second = (key if back else (child, parent))
|
|
found[key] = {
|
|
'child': first,
|
|
'parent': second,
|
|
'direction': 'undecided' if back else 'child_in_parent',
|
|
'agreement': round(forward if not back
|
|
else min(forward, agreement(parent, child)),
|
|
3),
|
|
'pieces': len(where[child]),
|
|
'instances': sum(where[child].values()),
|
|
'parent_pieces': len(where[parent]),
|
|
}
|
|
out = list(found.values())
|
|
out.sort(key=lambda r: (-r['agreement'], -r['instances'], r['child'],
|
|
r['parent']))
|
|
return out
|