#!/usr/bin/env python3
"""aiextract_run - the DSV4F-vision half of Design Extract, run out of process.

    python tools/aiextract_run.py --job <job.json>

Design Extract now chains two extraction passes over one design summary
(owner, Discord, 2026-09-01, approving the proposal: "可以，不过别用opus，用
dsv4f带vision吧" -- yes, but not opus, use DSV4F with vision):

    1. the mechanical reader (`vibe.designspec.build_spec`) -- regex and
       geometry only, never guesses, leaves everything it cannot place as
       `null` or in `unresolved`. It never even attempts the strand matrix:
       its own docstring says a wrong grid is worse than an honest blank.
    2. THIS SCRIPT -- reads whatever the mechanical pass left behind, plus
       the strand matrix (drawn geometry, not text, which is why it needs
       vision), by putting a rendered PNG of the summary view in front of a
       vision-capable model and asking it to fill in the gaps.

It exists as a separate CPython process, run detached from Revit, for the
same reason `vibe.aiextract`'s old opus-CLI path did: the extension runs on
IronPython 2.7 inside Revit's UI thread, a model call can take a couple of
minutes, and this script touches no Revit API at all -- it reads a job file,
an evidence JSON file and (optionally) some PNGs, talks to a local HTTP
server, and writes a spec file. `vibe.aiextract.launch()` is the IronPython
side of this boundary: it gathers the paths, writes the job file, and starts
this script in a visible detached window so its reasoning stays readable.

THE RULE THAT MAKES THIS SAFE TO RUN UNSUPERVISED: NEVER OVERWRITE, NEVER
HALF-MERGE
--------------------------------------------------------------------------
The mechanical reader outranks this script on anything it actually matched.
`apply_ai_fill()` below is the enforcement, and it does not trust the model
to have followed the prompt's own copy of this rule -- it re-checks every
field against the spec on disk before accepting a proposed value:

  * a scalar field is filled only if it is currently `None` (or an empty
    string);
  * a "structured" list -- `strand_groups`, `strand_layout.columns_in`,
    `strand_layout.rows` -- is a single coherent value, and is filled only
    if it is currently empty; a spec with even one entry already there is
    left exactly as it was;
  * `notes` and `unresolved` are collections of independent lines, so new,
    distinct entries are appended rather than gating on emptiness -- but
    nothing already in either list is ever changed or dropped;
  * `confidence` entries are accepted only for paths this pass actually
    just filled, never for a path the model merely commented on.

A run that fails for any reason -- dsh unreachable, the session times out,
the reply cannot be parsed as the agreed JSON shape -- leaves the spec's
FIELD VALUES exactly as the mechanical reader wrote them and appends exactly
one line to `unresolved` saying the AI pass did not run and why. There is no
partial-merge outcome: either every field this pass proposes is validated
and applied, or none of them are.

DSV4F PRICING
-------------
DSV4F has an off-peak pricing window the owner otherwise reserves for his own
manual daytime use of the harness. This script only ever runs from a button
press inside Revit, which is inherently a daytime, on-peak action -- it is
never scheduled to run at all, let alone into the off-peak window, so it
never competes with that reservation.

THE DSH WIRE CONTRACT
----------------------
dsh serves a local HTTP RPC at `POST http://127.0.0.1:3080/api/<method>`,
body `{"type": "client-request", "rpcId": <uuid>, "method": "session.create"
| "session.prompt" | "session.history" | "session.list", "payload": {...}}`,
with the rpcId echoed back. CONFIRMED against the documented contract
(owner, 2026-09-01).

The RESPONSE envelope's first two levels are CONFIRMED against a REAL dsh
reply: `session.create` on a live WPA5/sample_job run (owner, 2026-09-01)
answered `{'type': 'server-response', 'rpcId': ..., 'result': {'ok': True,
'value': {'sessionId': 'session-5969e01e-...', 'agentPreset': 'standard'}}}`
-- TWO levels of nesting below `result` (`result.ok`, `result.value.sessionId`),
not one. The first version of this script read the wrong level
(`result.sessionId`) and declared `DshError: session.create did not return
a session id` on a call that had, in fact, fully succeeded server-side --
the session was live on dsh with nobody in this process able to see it. See
`_rpc_value()`, which enforces and centralises this exact shape, and
`cancel_session()` / `run()`'s failure path, added the same day to limit how
long a session like that stays orphaned once ordinary in-flight failures
(not this specific parsing bug, which is now fixed) leave one behind.

MEASURED LIVE against a real dsh at http://127.0.0.1:3080, 2026-09-03
(supersedes every earlier guess about `session.prompt` / `session.history`,
below -- those guesses are what produced the `invalid payload for
session.prompt` failure on every prior run):

1. `session.prompt` payload must be
   `{"sessionId": <id>, "mode": "queue", "content": [{"type": "text",
   "text": <prompt>}]}`. `send_prompt()` sends exactly this now.

2. `session.history` payload is `{"sessionId": <id>, "maxMessages": N}` --
   N is an EVENT count, not a message count: the server returns the last N
   raw events, and one turn produces on the order of 20,000 of them, almost
   all `assistant/chunk` streaming deltas. `result.value` is
   `{"events": [{"event": {"type": <str>, "data": {...}}}, ...],
   "hasMore": bool, "projections": {...}}` -- there is NO `messages` key and
   NO `status`/`state` key anywhere in it. Event types seen: `user/message`
   (`data.content` is a list of `{"type": "text", "text": ...}` blocks),
   `assistant/chunk` (streaming deltas -- IGNORED entirely, see
   `_history_messages()`), `assistant/message` (`data.message` is
   `{"role": "assistant", "content": [...]}`, one per completed step of the
   turn -- the LAST one of a turn holds the final answer), `tool/call`,
   `tool/result`, `step/start`, `step/end`, `turn/start`, plus a few config
   events. `_history_messages()` projects the events shape into
   `{"role", "text"}` pairs; it still tolerates the old guessed
   `{"messages": [...]}` shape too, for the same reason `_rpc_value` stays
   permissive elsewhere -- but the events shape is what a live server
   actually sends and takes precedence when both are present.

3. `session.list` payload `{}` -> `result.value = {"items": [{"sessionId":
   <id>, "running": bool, "agentPreset": ..., "cwd": ..., "updatedAt": ...,
   "projections": ..., "blank": ...}, ...]}`. `running` is true while a turn
   is in progress and false once it has finished -- THIS is the done signal
   `wait_for_reply()` polls (`session_running()`), not a status field on
   history, because `session.history` carries no such field at all.

Every RPC call is funnelled through `_rpc()` / `_rpc_value()` /
`_http_post_json()` so any further correction stays in one place.

`resolve_model()`'s settings.yaml scan WAS confirmed against the real file
(owner, 2026-09-01): `~/.dsh/settings.yaml` nests a model's `id:` and its
`input:` list several keys apart inside provider/model list structure --
`_iter_mappings()` walks the whole parsed tree rather than assuming any
particular shape above the two keys. See its docstring.

4. THE ACTUAL BUG (found 2026-09-03, reading dsh's own source at
   `%APPDATA%/npm/node_modules/@deepseek-ai/dsh/node_modules/@deepseek-ai/
   dsh-host-apiproxy/lib/types/api/sessions.schema.js`, and confirmed on a
   live throwaway session): `session.create`'s payload zod schema
   (`sessionCreateRequestSchema`) is `{workspaceId?, cwd?, sessionId?,
   agentPreset?}` -- there is NO `model` field at all. The `{"model": ...}`
   key this file used to send there was silently stripped by zod's default
   unknown-key handling, so every session inherited whatever
   `agent-default-model` names in settings.yaml (on this machine:
   `commandcode` / `deepseek/deepseek-v4-flash`, no image input) regardless
   of what was asked for -- exactly the failure the task that added this
   section was opened to chase down. A session's model is set by a
   SEPARATE, later RPC: `session.selectModel`, payload `{"sessionId",
   "provider", "model", "reasoningEffort"?}` -> `result.value = {"selected":
   {"provider", "model", "reasoningEffort"?}}` (`sessionSelectModelRequestSchema`
   / `sessionSelectModelValueSchema`, same file). `provider` is REQUIRED --
   the model catalog's grouping key (`~/.dsh/settings.yaml`'s `providers:`
   mapping, keyed by provider id -- `commandcode` hosts both
   `deepseek/deepseek-v4-flash` and `deepseek/deepseek-v4-flash-vision-exp`
   on this machine), never inferable from the model id string alone. Live
   proof, 2026-09-03: `session.create` -> `session.selectModel({provider:
   "commandcode", model: "deepseek/deepseek-v4-flash-vision-exp"})` ->
   `session.models` immediately echoed that pair back as `current`, and the
   turn's own `request/context` history event (`{"type": "request/context",
   "data": {"provider": "commandcode", "model":
   "deepseek/deepseek-v4-flash-vision-exp", "contextWindow": 1000000}}`) --
   the event that says what the turn ACTUALLY ran on, not merely what was
   asked for -- named that exact pair. `select_model()` sends the RPC;
   `verify_session_model()` re-reads history afterwards and checks that
   event before this file ever trusts a reply, because a request that dsh
   accepted is not proof the turn ran on it -- a silent fallback to the
   account default looks identical at every earlier layer.

IronPython does not run this file -- it is plain CPython 3, same rule as
`tools/rvt.py` and `tools/vision_read.py`.
"""
from __future__ import annotations

import argparse
import copy
import io
import json
import os
import re
import sys
import time
import types
import urllib.error
import urllib.request
import uuid
from pathlib import Path


# --------------------------------------------------------------- vibe import
#
# `specformat.py` and `specsave.py` are the Revit-free half of the design
# spec pipeline (see their own docstrings, and CLAUDE.md's `vibe` layout
# section) -- pure data in, text out -- but `vibe/__init__.py` imports the
# Revit API at module scope the instant ANY submodule of `vibe` is imported
# the ordinary way. `WillTools.extension/tests/conftest.py` solves this for
# pytest by registering a bare package stub before the first import; the
# same trick, unchanged, lets this standalone script reach `specformat`
# without a Revit licence or an IronPython interpreter.
_REPO_ROOT = Path(__file__).resolve().parent.parent
_VIBE_DIR = _REPO_ROOT / 'WillTools.extension' / 'lib' / 'vibe'
if 'vibe' not in sys.modules:
    _stub = types.ModuleType('vibe')
    _stub.__path__ = [str(_VIBE_DIR)]
    _stub.__file__ = str(_VIBE_DIR / '__init__.py')
    sys.modules['vibe'] = _stub

from vibe import specformat  # noqa: E402  (see the stub above)


# ------------------------------------------------------------------- config

DEFAULT_DSH_BASE_URL = 'http://127.0.0.1:3080'
# Vision model on a long prompt + drawing PNG measured well over 180s to
# produce even a first assistant message on 2026-09-03 -- the old 180s
# default cancelled a session that was still working, never one that had
# failed. Keep in lockstep with the identically-named constant in
# WillTools.extension/lib/vibe/aiextract.py.
DEFAULT_TIMEOUT_S = 900
POLL_INTERVAL_S = 3
MODEL_ENV = 'WILLTOOLS_DSV4F_MODEL'
# `session.selectModel` requires an explicit provider id alongside the model
# id (see the module docstring's point 4) -- there is no fallback constant
# for this the way `FALLBACK_VISION_MODEL` is one for the model, because a
# wrong provider guess is exactly the silent-mismatch failure this file was
# rewritten to stop; `resolve_model()` fails closed (`provider is None`)
# rather than invent one.
PROVIDER_ENV = 'WILLTOOLS_DSV4F_PROVIDER'
# Named because the owner named it ("dsv4f带vision") -- not because this
# script has ever confirmed it against a real settings.yaml. See
# `resolve_model()` and the module docstring's owner-must-verify note.
FALLBACK_VISION_MODEL = 'dsv4f-vision'


# ---------------------------------------------------------------- the prompt
#
# Every field name below is copied from `specformat.SPEC_TEMPLATE` (via
# AI_REPLY_TEMPLATE, further down) rather than retyped, for the same reason
# rule 0 gives: a renamed field is read as absent by every check downstream,
# and the check then silently does not run. The OLD prompt in
# `vibe.aiextract` (the opus-CLI path this replaces) had drifted from that
# template -- it still asked for `spacings_in` / `positions_in` /
# `strands_per_row`, which stopped existing the day the spec grew the
# `columns_in` / `rows` matrix shape (commit `73fdd9a`, "spec 改成人类可读
# YAML + 矩阵 + 去掉 AI") -- because nothing called that prompt any more to
# notice. This one is built from the live template so it cannot drift the
# same way without a test noticing (see `test_aiextract_run.py`).

_PROMPT_HEADER = u"""You are the SECOND pass of a two-stage precast concrete design-summary
extractor, running as DSV4F with vision. A mechanical regex/geometry reader
already made a first pass over this design and wrote everything it could
match with certainty into the spec below. Your only job is the leftovers:
whatever that reader left null, empty, or listed under "unresolved" -- and,
because you have vision and it deliberately does not, the strand matrix,
which is drawn geometry, not text, and was never something the mechanical
reader should have guessed at.

THE RULE THAT MATTERS MOST: A VALUE ALREADY IN THE SPEC BELOW IS NOT YOURS
TO TOUCH. The mechanical reader outranks you on anything it actually
matched. If a field already carries a non-null value, or a list already has
entries, leave it out of your reply entirely -- even if you would have read
it differently. Only propose a value for a field that is null, an empty
list, or named in "unresolved" below. A host-side merge enforces this
regardless of what you send back, so there is no advantage to repeating or
"correcting" an already-filled field: your proposal for it is simply
discarded, and repeating values only wastes your own context.
"""

_PROMPT_RULES = u"""Rules that matter:

1. NEVER invent a value. If you cannot place a piece of text confidently,
   add an entry to "unresolved" of the form {{"text": "<verbatim>", "reason":
   "<why you would not encode it>"}}. A missing value becomes a visible
   warning downstream; a wrong value silently passes and could reach the
   plant.

2. The min/max envelope table is written as separate text notes: a label
   such as "MAX. LENGTH" and, beside it at nearly the same y, its value such
   as "49'- 3\\"". Pair them by position. Record both the decimal feet
   (min_ft, max_ft) and the original string (min_text, max_text). Only
   propose these if the corresponding field above is still null -- the
   mechanical reader usually gets this one, so most of the time you will
   have nothing to add here.

3. There can be MORE THAN ONE strand group. Shielded or de-tensioned
   strands are called out separately, e.g. "(14) 1/2\\" ... @ 31,600 # I.T."
   together with "(4) ... @ 5,000 # I.T.". Emit one object per group and set
   "role" to what the text says it is (e.g. "main", "shielded"), or null if
   unstated. dia_in is a decimal: 1/2 -> 0.5, 9/16 -> 0.5625. Propose the
   WHOLE "strand_groups" list only if it is currently empty -- it is one
   coherent value, not something to patch entry by entry.

4. Units: psi for strengths, pcf for unit weight, pounds for I.T., inches
   for diameters and thickness, decimal FEET for the envelope.

4a. THE STRAND MATRIX -- the whole point of your having vision. Fill
   reinforcing.strand_layout the same way a detailer would read it off the
   drawing:

     columns_in  -- strand positions ACROSS the piece, as cumulative
                    distance in inches from the datum (below), same numbers
                    a detailer would read off the dimension chain.
     rows        -- one entry per layer through the piece's thickness:
                    {{"from_bottom_in": <number>, "strands": [true/false, ...]}}
                    -- "strands" has exactly len(columns_in) entries, true
                    where that row actually has a strand at that column,
                    false where it does not. A design where every column has
                    a strand at every row -- the ordinary case -- has
                    "strands" all true; a design that is not a full grid
                    says so column by column instead of forcing a human to
                    reverse-engineer it from a total count.
     total_in    -- the chain total; should equal the piece width. Say so in
                    "unresolved" if it plainly does not.
     note        -- anything about the chain a human should double check.

   How to read the chain:

     a. Dimensions at (nearly) the same y are pieces of ONE chain --
        concatenate them left to right in increasing x, even when the
        evidence hands them to you as several separate dimension elements.
        Quote the concatenated chain, not the separate pieces.
     b. One chain per row. A section cut through two strand rows normally
        carries two chains -- the top row's drawn ABOVE the section, the
        bottom row's BELOW it -- while a full grid (every row the same)
        carries a single chain shared by every row. A real chain either
        closes to the far edge (total == piece width) or stops at the last
        strand (total == that strand's position); it does not fall short of
        both.
     c. A horizontal chain whose total is neither the width nor consistent
        with a strand row is NOT a strand chain -- mild bars, blocking
        diagrams and similar callouts can draw their own dimension chain
        nearby. Ignore it rather than forcing it to be a row.
     d. Row depths come from the VERTICAL chain, cumulative from the bottom
        face -- its segments sum to the piece thickness.
     e. Count check before you answer: the strands you place across every
        row MUST equal the stated quantity in the strand group text, and,
        when "symbols" are present in the evidence, the number of strand
        symbols too. If they do not agree, leave the matrix EMPTY and say in
        "unresolved" which count disagreed with which -- do not force a
        placement just to make the arithmetic close.
     f. "symbols" (when the evidence carries them) give the centre of every
        drawn detail component, in view feet -- the drawn strand marks are
        among them. Cluster their y to find the rows, and their x-gaps to
        match against a chain's segments; drafting views are true scale, so
        a 1.0 ft gap between symbols reads as a `1'-0"` segment.
     g. If "unresolved" already carries an entry for "STRAND PATTERN (matrix)",
        read its "reason" FIRST -- it is the mechanical reader's
        own geometric attempt telling you exactly what did not add up (a
        symbol drawn as the wrong kind, a chain that would not close, ...).
        Your job is to resolve THAT disagreement, not to re-derive the
        matrix from scratch and hope it happens to agree.

   Only propose strand_layout.columns_in / rows / total_in if they are
   CURRENTLY EMPTY -- never propose reinforcing.strand_layout.datum: that is
   plant knowledge (which rail the chain is measured from), not something
   you can read off one drawing, and it is filled in already. The datum in
   force for this design:

       {datum}

   Do not change your reading to make the numbers look nicer from the other
   end; if the chain plainly does not start where the datum says, record
   that in "unresolved" and leave the datum alone.

   QUOTE THE DIMENSION CHAIN VERBATIM in your prose summary below the JSON
   block -- the exact numbers you read off the drawing, in order -- so a
   person can hold your quote next to the drawing and check it without
   redoing your reading.

5. Prose instructions that are not a number -- "USE MC STANDARD
   REINFORCEMENT FOR CUT DOWN PANELS", "STRIPPING & BLOCKING DETAIL < 25'"
   -- belong in "notes" as written. Do not try to encode them. Only add
   entries here for lines nothing above already covers.

6. Fill "confidence" with an entry for every field you propose a value for
   (and ONLY for fields you propose a value for), mapping the dotted field
   path to "high", "medium" or "low". Use "high" only when the drawing
   states the value unambiguously.

7. Never set anything under "source" -- extractor, reviewed_by and the rest
   are the host's to stamp, not yours. Your reply carries none of it.
"""

_PROMPT_OUTPUT = u"""Reply with exactly one fenced ```json code block matching the shape given
above (omit or leave null anything you are not proposing -- do not resend a
value already present in the spec you were given), followed by a short
prose summary: what you filled and why, the strand matrix's dimension chain
quoted verbatim so a person can check it against the drawing, and anything
you are adding to "unresolved" and why. Do not ask questions -- if something
is ambiguous, leave it out of the JSON and say so in the summary and in
"unresolved".
"""


def _json_block(value):
    return json.dumps(value, indent=2, sort_keys=True, ensure_ascii=False)


def build_prompt(evidence, regex_spec, datum_clause, image_paths=None):
    """The exact text sent to the model. Pure -- no Revit, no dsh, no I/O.

    Kept as one function so the prompt actually sent can be captured and
    reviewed (see `--print-prompt`) rather than only existing implicitly
    inside `run()`.
    """
    image_paths = list(image_paths or [])
    parts = [_PROMPT_HEADER]

    parts.append(
        u"\nEVIDENCE (a deterministic scrape of this design's summary view "
        u"and its sheet -- every text note with its X/Y position, every "
        u"dimension with its segments, and one real assembly of this "
        u"design so you can sanity-check magnitudes; nothing in it has "
        u"been interpreted):\n\n" + _json_block(evidence))

    parts.append(
        u"\nTHE SPEC AS THE MECHANICAL READER LEFT IT (this is what you "
        u"may not overwrite -- see the rule above):\n\n" +
        _json_block(regex_spec))

    parts.append(
        u"\nTHE SHAPE YOUR REPLY MUST FIT. Reply with a JSON object using "
        u"EXACTLY these field names -- do not rename, do not add fields, "
        u"do not drop fields:\n\n" + _json_block(AI_REPLY_TEMPLATE))

    parts.append(u"\n" + (_PROMPT_RULES.format(datum=datum_clause)))

    if image_paths:
        lines = u"\n".join(u"  - {0}".format(p) for p in image_paths)
        parts.append(
            u"\nDRAWING IMAGES -- read each with your read_image tool. "
            u"These are file paths on this machine, not embedded bytes:\n\n"
            + lines +
            u"\n\nUse these for anything the deterministic evidence above "
            u"cannot give you -- above all the strand matrix, which is a "
            u"drawn grid, not text.\n")
    else:
        parts.append(
            u"\nNo drawing image was available for this pass -- text and "
            u"the evidence above only. Do not attempt the strand matrix "
            u"without a drawing to read it from; leave strand_layout empty "
            u"and say why in \"unresolved\" if it matters.\n")

    parts.append(u"\n" + _PROMPT_OUTPUT)
    return u"\n".join(parts)


# ------------------------------------------------------------ the AI's shape
#
# What the model is allowed to propose values for. Deliberately narrower
# than SPEC_TEMPLATE: no "source", no "design_number", no "schema" -- those
# are the host's to stamp (see CONTENT_PATHS below, which this mirrors
# exactly) and never travel over the wire to the model at all.

AI_REPLY_TEMPLATE = {
    'piece_description': None,
    'concrete': {'fc_psi': None, 'fci_psi': None,
                'unit_weight_pcf': None, 'weight_class': None},
    'reinforcing': {
        'mild_grade_ksi': None,
        'wwf': {'mesh': None, 'layers': None, 'per_face': None},
        'strand_groups': [],          # [{qty, dia_in, grade_ksi, it_lb,
                                       #   role, note}]
        'strand_layout': {
            'columns_in': [],         # cumulative in from datum
            'rows': [],               # [{from_bottom_in, strands: [bool,..]}]
            'total_in': None,
            'note': None,
        },
    },
    'envelope': {'length': {'min_ft': None, 'max_ft': None,
                            'min_text': None, 'max_text': None},
                 'width': {'min_ft': None, 'max_ft': None,
                           'min_text': None, 'max_text': None}},
    'thickness_in': None,
    'notes': [],
    'unresolved': [],
    'confidence': {},
}

# The scalar / single-structured-value paths the AI may fill -- one path per
# leaf in AI_REPLY_TEMPLATE except the two list-of-independent-entries
# fields, which get their own merge rule (see ATOMIC_LIST_PATHS) and
# "confidence", which is merged separately because it depends on what else
# was just filled.
CONTENT_PATHS = (
    'piece_description',
    'concrete.fc_psi', 'concrete.fci_psi',
    'concrete.unit_weight_pcf', 'concrete.weight_class',
    'reinforcing.mild_grade_ksi',
    'reinforcing.wwf.mesh', 'reinforcing.wwf.layers',
    'reinforcing.wwf.per_face',
    'reinforcing.strand_groups',
    'reinforcing.strand_layout.columns_in',
    'reinforcing.strand_layout.rows',
    'reinforcing.strand_layout.total_in',
    'reinforcing.strand_layout.note',
    'envelope.length.min_ft', 'envelope.length.max_ft',
    'envelope.length.min_text', 'envelope.length.max_text',
    'envelope.width.min_ft', 'envelope.width.max_ft',
    'envelope.width.min_text', 'envelope.width.max_text',
    'thickness_in',
)

# These three are each ONE coherent value (a grid, not independent lines):
# filled only if currently empty, and replaced whole -- never patched
# entry-by-entry, and never touched at all once anything is there.
STRUCTURED_LIST_PATHS = (
    'reinforcing.strand_groups',
    'reinforcing.strand_layout.columns_in',
    'reinforcing.strand_layout.rows',
)

# These two are collections of INDEPENDENT lines: new, distinct entries are
# appended; nothing already present is ever changed or dropped.
ATOMIC_LIST_PATHS = ('notes', 'unresolved')


# ------------------------------------------------------------- merge logic
#
# The part of this file that is actually tested (see test_aiextract_run.py)
# and the part that makes an otherwise-unsupervised process safe to run:
# regardless of what the model sends back, nothing here can overwrite a
# value the mechanical reader placed, and nothing here can produce a
# half-merged file -- `apply_ai_fill` either fills a field or it does not,
# there is no partial write of one field.

def _is_empty(value):
    if value is None:
        return True
    if isinstance(value, str) and value.strip() == '':
        return True
    if isinstance(value, (list, dict)) and len(value) == 0:
        return True
    return False


def _set_path(tree, dotted, value):
    parts = dotted.split('.')
    node = tree
    for part in parts[:-1]:
        nxt = node.get(part)
        if not isinstance(nxt, dict):
            nxt = {}
            node[part] = nxt
        node = nxt
    node[parts[-1]] = value


def _entry_key(entry):
    """A dedupe key for one 'unresolved' / 'notes' entry, tolerant of shape.

    Mirrors `vibe.specsave.unresolved_entries`'s tolerance: an entry may be
    a bare string (an older writer) or a {'text':, 'reason':} dict.
    """
    if isinstance(entry, dict):
        text = entry.get('text')
    else:
        text = entry
    return re.sub(r'\s+', ' ', (u'{0}'.format(text or ''))).strip().lower()


def _merge_atomic_entries(current, proposed):
    """New entries from `proposed` not already present in `current`, in order."""
    seen = set(_entry_key(e) for e in (current or []))
    added = []
    for entry in (proposed or []):
        key = _entry_key(entry)
        if key and key not in seen:
            added.append(entry)
            seen.add(key)
    return added


def apply_ai_fill(spec, ai_payload):
    """Merge one AI proposal into a regex-written spec.

    Returns (merged_spec, filled_paths) -- a NEW dict; `spec` is untouched,
    which is what lets a failed or bad reply be handed the original spec
    back unmodified. `filled_paths` is the honest record of what actually
    changed (`source.ai_filled` downstream), and it is used again inside
    this function to gate which "confidence" entries are accepted -- a
    confidence rating for a field the model did not just fill is noise, not
    provenance, and is dropped.
    """
    merged = copy.deepcopy(spec)
    ai_payload = ai_payload if isinstance(ai_payload, dict) else {}
    filled = []

    for path in CONTENT_PATHS:
        proposed = specformat.get_path(ai_payload, path)
        if proposed is None:
            continue
        if path in STRUCTURED_LIST_PATHS:
            current = specformat.get_path(merged, path, []) or []
            if current:
                continue                       # already placed -- untouchable
            if not isinstance(proposed, list) or not proposed:
                continue
        else:
            current = specformat.get_path(merged, path)
            if not _is_empty(current):
                continue                       # already placed -- untouchable
        _set_path(merged, path, proposed)
        filled.append(path)

    for path in ATOMIC_LIST_PATHS:
        proposed = specformat.get_path(ai_payload, path, [])
        if not proposed:
            continue
        current = specformat.get_path(merged, path, []) or []
        added = _merge_atomic_entries(current, proposed)
        if added:
            _set_path(merged, path, list(current) + added)
            filled.append(path)

    proposed_conf = ai_payload.get('confidence')
    if isinstance(proposed_conf, dict):
        current_conf = dict(specformat.get_path(merged, 'confidence', {}) or {})
        changed = False
        for field_path, level in proposed_conf.items():
            if field_path in filled and field_path not in current_conf:
                current_conf[field_path] = level
                changed = True
        if changed:
            merged['confidence'] = current_conf

    return merged, filled


FAILURE_REASON_PREFIX = 'AI pass (dsv4f-vision) did not run'


def apply_failure_note(spec, reason):
    """The failure outcome: field values untouched, one line in unresolved.

    Never raises, never drops an existing note, and never adds a second
    line for the exact same reason on a retry (`_entry_key` dedupes by
    normalised text, same as the ordinary unresolved-entry merge).
    """
    merged = copy.deepcopy(spec)
    note = {'text': FAILURE_REASON_PREFIX, 'reason': reason}
    current = specformat.get_path(merged, 'unresolved', []) or []
    added = _merge_atomic_entries(current, [note])
    if added:
        merged['unresolved'] = list(current) + added
    return merged


# ----------------------------------------------------------- dsh RPC client
#
# See the module docstring's "THE DSH WIRE CONTRACT ASSUMED HERE" section --
# every assumption about the wire format is contained to this section, and
# every call goes through `_rpc()` / `_http_post_json()` so it can be fixed
# in one place. Tests fake `transport` (a plain callable), never the
# network, which is the HTTP boundary the task asked to fake at.

class DshError(Exception):
    """dsh reached, but the RPC failed or answered something unusable."""


class DshTimeout(DshError):
    """dsh did not answer (connect/read timeout, or no reply within budget)."""


def _http_post_json(url, body, timeout):
    """The one real network call in this file. Swappable via `transport=`."""
    data = json.dumps(body).encode('utf-8')
    req = urllib.request.Request(
        url, data=data, headers={'Content-Type': 'application/json'},
        method='POST')
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            raw = resp.read().decode('utf-8', 'replace')
    except urllib.error.HTTPError as exc:
        raw = exc.read().decode('utf-8', 'replace')
    except urllib.error.URLError as exc:
        if isinstance(getattr(exc, 'reason', None), TimeoutError):
            raise DshTimeout('{0}: {1}'.format(url, exc.reason))
        raise DshError('could not reach dsh at {0}: {1}'.format(url, exc.reason))
    except TimeoutError as exc:
        raise DshTimeout('{0}: {1}'.format(url, exc))
    try:
        return json.loads(raw)
    except ValueError:
        raise DshError('dsh returned non-JSON from {0}: {1!r}'
                       .format(url, raw[:200]))


def _rpc_value(res, method):
    """Unwrap a dsh response envelope down to its `result.value` payload,
    raising a DshError that names EXACTLY which layer was missing or the
    wrong shape -- never a silent `None` a caller then has to re-check.

    CONFIRMED against a REAL dsh reply (owner, WPA5/sample_job run,
    2026-09-01)::

        {'type': 'server-response', 'rpcId': '3d7ad...',
         'result': {'ok': True,
                    'value': {'sessionId': 'session-5969e01e-...',
                              'agentPreset': 'standard'}}}

    That is `session.create`'s envelope, confirmed to this exact depth --
    it is what `create_session` failed to reach before this fix, because the
    code that used to live here (and in `_history_messages` /
    `_history_status`, which each re-derived their own partial version of
    this same unwrap) stopped one level too shallow, at `result.sessionId`
    rather than `result.value.sessionId`. Centralising it here means the
    next shape drift gets fixed in one place instead of three, and every
    caller now sees the same failure mode instead of three different ones.

    `session.prompt`, `session.history` and `session.list` share this exact
    two-level `result.ok` / `result.value` envelope -- MEASURED against a
    live dsh, 2026-09-03 (see the module docstring's wire-contract section),
    not merely assumed any more.
    """
    if not isinstance(res, dict):
        raise DshError('dsh {0} response was not a JSON object: {1!r}'
                       .format(method, res))
    result = res.get('result')
    if not isinstance(result, dict):
        raise DshError('dsh {0} response had no usable "result" object '
                       '(got {1}): {2!r}'
                       .format(method, type(result).__name__, res))
    if not result.get('ok'):
        raise DshError('dsh {0} response result.ok was not truthy ({1!r}): '
                       '{2!r}'.format(method, result.get('ok'), res))
    value = result.get('value')
    if not isinstance(value, dict):
        raise DshError('dsh {0} response result.value was not an object '
                       '(got {1}): {2!r}'
                       .format(method, type(value).__name__, res))
    return value


def _rpc(base_url, method, payload, timeout, transport=None):
    """One `{"type": "client-request", ...}` round trip -> the method's
    `result.value`, ALREADY UNWRAPPED (see `_rpc_value`). Raises DshError on
    any failure: an rpcId mismatch, a top-level `error`, or a `result.value`
    that is missing or the wrong shape.

    Checks the echoed `rpcId` the same way `tools/rvt.py` checks its own
    correlation id against the bridge -- cheap insurance against exactly the
    class of bug that cost real time there (REQUEST_CORRELATION in that
    file), and free to add here since dsh's contract already carries an id.

    Every caller gets the unwrapped payload from this ONE place -- see
    `_rpc_value`'s docstring for why that centralisation is the actual fix,
    not just this function's own change.
    """
    transport = transport or _http_post_json
    rpc_id = uuid.uuid4().hex
    body = {'type': 'client-request', 'rpcId': rpc_id, 'method': method,
            'payload': payload}
    url = base_url.rstrip('/') + '/api/' + method
    res = transport(url, body, timeout)
    if isinstance(res, dict):
        got = res.get('rpcId')
        if got is not None and got != rpc_id:
            raise DshError('dsh answered a different request on {0} '
                           '(asked {1}, answered {2})'
                           .format(method, rpc_id, got))
        if res.get('error'):
            raise DshError('dsh {0} failed: {1}'.format(method, res['error']))
    return _rpc_value(res, method)


def create_session(base_url, timeout, transport=None):
    """Ask dsh for a new session; return its id, or raise DshError.

    Sends an EMPTY payload -- `session.create`'s real schema
    (`sessionCreateRequestSchema`, see the module docstring's point 4) is
    `{workspaceId?, cwd?, sessionId?, agentPreset?}`. There used to be a
    `model` argument here and a `{"model": model}` payload built from it;
    that key has never been part of the schema, so zod silently stripped it
    and every session came up on whatever `agent-default-model` names
    instead. Setting a session's model is `select_model()`'s job now, as a
    separate RPC after this one returns -- see `run()`.

    `_rpc` has already confirmed `result.ok` and `result.value` by the time
    this runs -- the ONLY thing left to validate here is that `value` itself
    actually carries a usable `sessionId` (a non-empty string), because that
    is the one part of the shape `_rpc_value` cannot check on session.create's
    behalf: it does not know which key inside `value` matters to this
    particular method.
    """
    value = _rpc(base_url, 'session.create', {}, timeout, transport)
    session_id = value.get('sessionId')
    if not isinstance(session_id, str) or not session_id.strip():
        raise DshError('session.create response value had no usable '
                       '"sessionId" (got {0!r}): {1!r}'
                       .format(session_id, value))
    return session_id


def select_model(base_url, session_id, provider, model, timeout,
                 transport=None):
    """`session.selectModel` -- the RPC that actually pins a session's
    model. See the module docstring's point 4 for how this was found:
    `session.create` accepts no `model` field at all, so a freshly created
    session keeps running on the account's `agent-default-model` until this
    is called.

    Payload `{"sessionId", "provider", "model"}` -> `result.value =
    {"selected": {"provider", "model", "reasoningEffort"?}}`
    (`sessionSelectModelRequestSchema` / `sessionSelectModelValueSchema`,
    `dsh-host-apiproxy`). `provider` is required by that schema -- there is
    no default and no way to omit it, which is why `resolve_model()` fails
    closed rather than calling this with a guessed one.

    Accepting this RPC is NOT proof the turn actually ran on the requested
    model -- see `verify_session_model()`, which `run()` calls afterwards
    to check the session's own history rather than trusting this echo.
    """
    payload = {'sessionId': session_id, 'provider': provider, 'model': model}
    return _rpc(base_url, 'session.selectModel', payload, timeout, transport)


def cancel_session(base_url, session_id, timeout, transport=None):
    """Best-effort `session.cancel` -- swallows every error, never raises.

    ORPHAN-SESSION RISK: `session.create` succeeding server-side is not the
    same event as this script noticing it succeeded -- anything that fails
    AFTER a valid `session_id` comes back (the prompt call, the history
    poll, an unparsable reply) still leaves that session open on the dsh
    server with nothing left in this process to close it, because the whole
    point of `run()`'s failure path is that it gives up and writes the
    documented "left untouched" outcome rather than retrying. `run()` calls
    this from that failure path whenever it already has a trusted
    `session_id` in hand (never a guessed one -- there is nothing to cancel
    when `create_session` itself never returned).

    This is deliberately the only cleanup attempted. A `session.create` that
    fails before returning an id (the bug this file was fixed for) is not a
    case this function can help with -- the id was never something the
    caller could see, only something visible with hindsight once
    `_rpc_value`'s error message is read; there is nothing here to guess it
    from safely, and guessing wrong would risk cancelling an unrelated
    session. See the module docstring / owner report for that residual risk.

    Returns True if the cancel RPC came back without raising, False
    otherwise -- purely informational for the log; neither outcome changes
    `run()`'s documented failure result.
    """
    try:
        _rpc(base_url, 'session.cancel', {'sessionId': session_id}, timeout,
             transport)
        return True
    except Exception:                                  # noqa: BLE001
        return False


def send_prompt(base_url, session_id, text, timeout, transport=None):
    payload = {'sessionId': session_id, 'mode': 'queue',
              'content': [{'type': 'text', 'text': text}]}
    return _rpc(base_url, 'session.prompt', payload, timeout, transport)


def _text_from_content_blocks(blocks, kinds=('text',)):
    """Join the `text` of every block in `blocks` whose `type` is in `kinds`.

    Used for both `user/message`'s `data.content` (kinds defaults to just
    "text") and `assistant/message`'s `data.message.content`, where
    "reasoning" blocks sit beside "text" ones and must be dropped -- the
    reasoning is the model's scratch work, never part of the settled reply.
    """
    parts = []
    for block in (blocks or []):
        if isinstance(block, dict) and block.get('type') in kinds \
                and isinstance(block.get('text'), str):
            parts.append(block['text'])
    return ''.join(parts)


def _history_messages(value):
    """`value` is what `_rpc(..., 'session.history', ...)` returns -- already
    unwrapped to `result.value` by `_rpc_value`.

    MEASURED against a live dsh, 2026-09-03 (see the module docstring): the
    real shape is `{"events": [{"event": {"type": ..., "data": {...}}}, ...],
    "hasMore": bool, "projections": {...}}`, not `{"messages": [...]}`. This
    projects that events list down to the two event types that matter --
    `user/message` and `assistant/message` -- as an ordered list of
    `{"role": "user"|"assistant", "text": str}` dicts. Everything else
    (`assistant/chunk` streaming deltas above all -- there are ~20,000 of
    them in one turn's history) is skipped entirely, not just ignored by a
    later filter, so callers never have to know these event types exist.

    Still tolerates the OLD guessed `{"messages": [...]}` shape (a bare list
    of `{role, content|text}` dicts) so tests written against that guess
    keep passing -- but the events shape takes precedence when both keys
    are present, since that is what a live server actually sends.
    """
    if isinstance(value, dict) and isinstance(value.get('events'), list):
        messages = []
        for entry in value['events']:
            event = entry.get('event') if isinstance(entry, dict) else None
            if not isinstance(event, dict):
                continue
            etype = event.get('type')
            data = event.get('data') or {}
            if etype == 'user/message':
                text = _text_from_content_blocks(data.get('content'))
                messages.append({'role': 'user', 'text': text})
            elif etype == 'assistant/message':
                message = data.get('message') or {}
                text = _text_from_content_blocks(message.get('content'),
                                                  kinds=('text',))
                messages.append({'role': 'assistant', 'text': text})
            # every other event type (assistant/chunk, tool/call,
            # tool/result, step/*, turn/start, config, ...) is skipped.
        return messages
    if isinstance(value, dict) and isinstance(value.get('messages'), list):
        return [{'role': _message_role(m), 'text': _message_text(m)}
               for m in value['messages']]
    if isinstance(value, list):
        return [{'role': _message_role(m), 'text': _message_text(m)}
               for m in value]
    return []


def _message_role(message):
    return (message.get('role') or message.get('type') or '').lower()


def _message_text(message):
    if isinstance(message.get('text'), str):
        return message['text']
    content = message.get('content')
    if isinstance(content, str):
        return content
    if isinstance(content, list):
        parts = []
        for chunk in content:
            if isinstance(chunk, dict) and isinstance(chunk.get('text'), str):
                parts.append(chunk['text'])
            elif isinstance(chunk, str):
                parts.append(chunk)
        return ''.join(parts)
    return ''


def session_running(base_url, session_id, timeout, transport=None):
    """Whether `session_id` currently has a turn in progress, per
    `session.list` -- the done-signal `wait_for_reply()` polls.

    MEASURED against a live dsh, 2026-09-03: `session.list` payload `{}` ->
    `result.value = {"items": [{"sessionId": ..., "running": bool, ...}]}`.
    Returns True/False from the matching item's `running` field, or None if
    the session is not listed at all, or the shape is not what is expected
    -- callers must treat None as "cannot tell", never as False, the same
    "could not judge vs judged and found nothing" discipline as everywhere
    else in this tool.
    """
    try:
        value = _rpc(base_url, 'session.list', {}, timeout, transport)
    except DshError:
        return None
    items = value.get('items') if isinstance(value, dict) else None
    if not isinstance(items, list):
        return None
    for item in items:
        if isinstance(item, dict) and item.get('sessionId') == session_id:
            running = item.get('running')
            return running if isinstance(running, bool) else None
    return None


def wait_for_reply(base_url, session_id, baseline_count, timeout_s,
                   transport=None, sleep=time.sleep, clock=time.time,
                   poll_interval=POLL_INTERVAL_S):
    """The model's final reply text, or raises DshTimeout.

    Polls `session.history` (asking for enough events -- 50,000 -- to cover
    a whole turn, per the module docstring's measured event-count-not-
    message-count contract) and projects it with `_history_messages`. The
    candidate reply is the text of the LAST assistant message after
    `baseline_count` -- a turn that calls tools produces several
    `assistant/message` events, one per step, and only the final one holds
    the settled answer.

    Done when `session_running()` says the turn has finished (`running` is
    False) and a non-empty candidate exists. If `session_running()` cannot
    tell (returns None -- session not listed, or an unexpected shape), falls
    back to the old stability check: the same non-empty text on two
    consecutive polls, which tolerates a reply arriving mid-update without
    declaring victory on a still-streaming partial answer.
    """
    deadline = clock() + timeout_s
    last_text = None
    while clock() < deadline:
        value = _rpc(base_url, 'session.history',
                     {'sessionId': session_id, 'maxMessages': 50000},
                     min(30, timeout_s), transport)
        messages = _history_messages(value)
        assistant_all = [m for m in messages if m['role'] == 'assistant']
        assistant_msgs = assistant_all[baseline_count:]
        if assistant_msgs:
            text = assistant_msgs[-1]['text']
            running = session_running(base_url, session_id,
                                      min(30, timeout_s), transport)
            if running is not None:
                if running is False and text:
                    return text
            elif text and text == last_text:
                return text
            last_text = text
        sleep(poll_interval)
    raise DshTimeout('no settled assistant reply on session {0} within {1}s'
                     .format(session_id, timeout_s))


# ------------------------------------------------------ model verification
#
# THE CHECK THIS TASK WAS OPENED TO ADD. See the module docstring's point 4:
# neither `session.create`'s response nor `session.selectModel`'s own
# response is proof a turn actually ran on the requested model -- both only
# echo what was asked for (or, for `session.create`, ignore it silently).
# The one event that says what a turn ACTUALLY used is `request/context`,
# emitted into the session's own history once the turn starts. `run()` calls
# `verify_session_model()` after `wait_for_reply()` settles and BEFORE it
# ever parses the reply as data -- a wrong-model reply can look exactly like
# a right-model one (confident, well-formatted, plausible numbers), so this
# has to gate on the model, never on whether the reply merely parsed.

def _extract_context_events(history_value):
    """Every `request/context` event's `{"provider", "model"}` pair, oldest
    first. `history_value` is already unwrapped to `result.value` (same
    input shape as `_history_messages`) -- MEASURED live, 2026-09-03: the
    event is `{"type": "request/context", "data": {"provider": ...,
    "model": ..., "contextWindow": ...}}`, one per LLM call the turn makes
    (a compaction or a retry can add more than one -- the caller wants the
    LAST).
    """
    events = history_value.get('events') if isinstance(history_value, dict) \
        else None
    if not isinstance(events, list):
        return []
    out = []
    for entry in events:
        event = entry.get('event') if isinstance(entry, dict) else None
        if not isinstance(event, dict) or event.get('type') != 'request/context':
            continue
        data = event.get('data') or {}
        provider, model = data.get('provider'), data.get('model')
        if isinstance(provider, str) and isinstance(model, str):
            out.append({'provider': provider, 'model': model})
    return out


def verify_session_model(base_url, session_id, expected_provider,
                         expected_model, timeout, transport=None):
    """Raise `DshError` -- naming BOTH the requested and the actual pair --
    unless `session_id` really ran on `expected_provider` / `expected_model`,
    per its own `request/context` history event.

    Re-fetches `session.history` rather than trusting anything already in
    hand from `wait_for_reply()`'s polling, so this stays correct regardless
    of how that loop settled. Also raises when NO `request/context` event
    exists yet at all -- a session with no such event has not verifiably run
    on anything, and this refuses to treat "cannot tell" as "must be fine"
    the same way the rest of this codebase refuses to collapse "could not
    judge" into a pass (see CLAUDE.md's `knowledge.same_face()` example).

    Returns `(actual_provider, actual_model)` on success, for the caller to
    log.
    """
    value = _rpc(base_url, 'session.history',
                {'sessionId': session_id, 'maxMessages': 50000},
                timeout, transport)
    seen = _extract_context_events(value)
    if not seen:
        raise DshError(
            'could not verify session {0}\'s model -- no "request/context" '
            'event in its history yet (requested {1}/{2}); refusing to '
            'trust an unverified session rather than risk a confident '
            'text-only fill from the wrong model'.format(
                session_id, expected_provider, expected_model))
    actual = seen[-1]
    if actual['provider'] != expected_provider or actual['model'] != expected_model:
        raise DshError(
            'session {0} actually ran on {1}/{2}, not the requested {3}/{4} '
            '-- refusing to proceed: a non-vision model filling this prompt '
            'with a confident-looking text-only reply is exactly the silent '
            'failure this check exists to catch'.format(
                session_id, actual['provider'], actual['model'],
                expected_provider, expected_model))
    return actual['provider'], actual['model']


# ------------------------------------------------------- extracting the JSON

_JSON_FENCE_RE = re.compile(r'```(?:json)?\s*(\{.*?\})\s*```', re.DOTALL)


def _first_balanced_object(text, start):
    depth = 0
    for i in range(start, len(text)):
        ch = text[i]
        if ch == '{':
            depth += 1
        elif ch == '}':
            depth -= 1
            if depth == 0:
                return text[start:i + 1]
    raise ValueError('unbalanced JSON object in reply')


def extract_json_payload(reply_text):
    """The model's structured proposal, pulled out of its prose reply.

    Looks for a fenced ```json block first (the prompt asks for exactly
    one); falls back to the first balanced {...} in the text. Raises
    ValueError -- never returns {} -- when nothing parses, because a reply
    that cannot be read as data must never be silently treated as an empty,
    harmless one: that is exactly the "half-merged" outcome this whole
    module exists to avoid, so it must surface as a failure, not a no-op
    success.
    """
    reply_text = reply_text or ''
    match = _JSON_FENCE_RE.search(reply_text)
    if match:
        candidate = match.group(1)
    else:
        start = reply_text.find('{')
        if start == -1:
            raise ValueError('no JSON object found in the reply')
        candidate = _first_balanced_object(reply_text, start)
    payload = json.loads(candidate)
    if not isinstance(payload, dict):
        raise ValueError('the reply\'s JSON was not an object: {0!r}'
                         .format(type(payload).__name__))
    return payload


# --------------------------------------------------------------- model name

def resolve_model(job):
    """Which model AND which dsh provider to ask for. Fails closed on
    either rather than guessing wrong.

    Returns `(model, provider, why)`. `session.selectModel` needs BOTH --
    see the module docstring's point 4: `provider` is a required field of
    its own, never inferable from the model id string, and this refuses to
    invent one the way it will (loudly) invent a fallback MODEL -- a wrong
    provider guess is exactly the silent-session-runs-on-something-else
    failure this file was rewritten to stop, and `run()`'s
    `verify_session_model()` check downstream cannot save a session that
    never legitimately requested the right model to begin with.

    Model order: an explicit job['model'] (aiextract.py can set this once
    the owner has confirmed the real name); the WILLTOOLS_DSV4F_MODEL env
    var; a real parse of ~/.dsh/settings.yaml for a model id whose `input`
    list names "image"; the FALLBACK_VISION_MODEL constant, which is a guess
    and is flagged as one in the returned `why` string every time it is
    used, so it shows up in the log rather than passing quietly.

    Provider order: an explicit job['provider']; the WILLTOOLS_DSV4F_PROVIDER
    env var; whichever provider the settings.yaml scan above already
    associated with the chosen model (free, when the model came from that
    same scan); failing that, a SEPARATE settings.yaml lookup
    (`_find_provider_for_model_id`) for whatever provider block actually
    lists the resolved model id -- covers an explicit/env model whose
    provider was never looked up by the vision scan. `provider` comes back
    `None`, never a guess, when none of those found one.
    """
    explicit_model = job.get('model')
    env_model = os.environ.get(MODEL_ENV)
    scanned_provider = None
    if explicit_model:
        model, model_confirmed = explicit_model, True
    elif env_model:
        model, model_confirmed = env_model, True
    else:
        scanned_model, scanned_provider, scan_why = \
            _scan_settings_for_vision_model()
        if scanned_model:
            model, model_confirmed = scanned_model, True
        else:
            model, model_confirmed = FALLBACK_VISION_MODEL, False

    explicit_provider = job.get('provider')
    env_provider = os.environ.get(PROVIDER_ENV)
    if explicit_provider:
        provider, provider_why = explicit_provider, None
    elif env_provider:
        provider, provider_why = env_provider, None
    elif scanned_provider:
        provider, provider_why = scanned_provider, None
    else:
        provider, provider_why = _find_provider_for_model_id(model)

    why_parts = []
    if not model_confirmed:
        why_parts.append(
            'could not confirm the vision model name from {0} or {1} ({2}) '
            '-- using the unconfirmed fallback {3!r}. See '
            'aiextract_run.py\'s module docstring.'.format(
                MODEL_ENV, '~/.dsh/settings.yaml', scan_why,
                FALLBACK_VISION_MODEL))
    if not provider:
        why_parts.append(
            'could not determine which dsh provider serves model {0!r} '
            '({1}) -- session.selectModel requires an explicit provider id '
            'and this refuses to guess one. Set {2} explicitly, or {3} to '
            'name a different model whose provider IS in settings.yaml.'
            .format(model, provider_why, PROVIDER_ENV, MODEL_ENV))
    why = ' '.join(why_parts) if why_parts else None
    return model, provider, why


def _iter_mappings(node):
    """Every dict anywhere in a parsed YAML tree, at any depth.

    dsh's real settings.yaml nests a model's `id:` and its `input:` list
    several keys apart inside deep, provider-specific structure (measured on
    the owner's machine: `providers: [...] -> models: [...] -> - id: ... /
    (other keys) / input: [text, image]`). Walking every mapping rather than
    assuming a fixed shape above it is what survives that -- the two keys
    only need to be IN THE SAME MAPPING, never adjacent lines.
    """
    if isinstance(node, dict):
        yield node
        for value in node.values():
            for found in _iter_mappings(value):
                yield found
    elif isinstance(node, list):
        for item in node:
            for found in _iter_mappings(item):
                yield found


def _iter_model_entries(node, provider_id=None):
    """Every dict that looks like ONE model catalog entry (carries an `id`),
    paired with the provider id it sits under -- `None` when it sits under
    no recognisable provider block at all.

    A `providers` key names a provider block in the two shapes this file has
    had reason to handle: a MAPPING keyed by provider id (the real
    `~/.dsh/settings.yaml` on this machine, confirmed 2026-09-03:
    `llm-pi-ai: providers: {commandcode: {models: [...]}, ...}` -- the dict
    key IS the provider id `session.selectModel` wants), or a LIST of
    provider objects that name themselves via their own `id` or `name` field
    (an earlier guessed shape some fixtures here still use). Either way,
    once a provider block is found everything nested under it -- at any
    depth, same `_iter_mappings` tolerance -- is tagged with that provider id
    until a closer `providers` block overrides it.
    """
    if isinstance(node, dict):
        providers = node.get('providers')
        if isinstance(providers, dict):
            for pid, pconf in providers.items():
                for found in _iter_model_entries(pconf, provider_id=pid):
                    yield found
        elif isinstance(providers, list):
            for item in providers:
                if isinstance(item, dict):
                    pid = item.get('id') or item.get('name')
                    for found in _iter_model_entries(item, provider_id=pid):
                        yield found
        for key, value in node.items():
            if key == 'providers':
                continue                    # already walked, above
            for found in _iter_model_entries(value, provider_id=provider_id):
                yield found
        if isinstance(node.get('id'), str) and node.get('id').strip():
            yield provider_id, node
    elif isinstance(node, list):
        for item in node:
            for found in _iter_model_entries(item, provider_id=provider_id):
                yield found


def _declares_image_input(mapping):
    value = mapping.get('input')
    if not isinstance(value, list):
        return False
    return any(isinstance(v, str) and v.strip().lower() == 'image'
              for v in value)


def _load_settings_yaml(settings_path=None):
    """Parse ~/.dsh/settings.yaml (or `settings_path`); `(data, None)` or
    `(None, why)` -- shared by both settings.yaml lookups below so a missing
    file, a missing PyYAML, or a YAML syntax error says the same thing in
    both places rather than being spelled out twice.
    """
    path = settings_path or os.path.join(
        os.path.expanduser('~'), '.dsh', 'settings.yaml')
    if not os.path.isfile(path):
        return None, 'no settings file at {0}'.format(path)
    try:
        import yaml
    except ImportError as exc:
        return None, ('PyYAML is not installed ({0}) -- see requirements.txt'
                      .format(exc))
    try:
        with io.open(path, 'r', encoding='utf-8') as fh:
            return yaml.safe_load(fh), None
    except Exception as exc:                          # noqa: BLE001
        return None, 'could not parse {0}: {1}'.format(path, exc)


def _scan_settings_for_vision_model(settings_path=None):
    """The (model id, provider id) in ~/.dsh/settings.yaml whose `input`
    list names "image" -- a REAL `yaml.safe_load`, not a line-scan.

    PyYAML is a host-CPython dependency of this repo already (see
    requirements.txt), so there is no reason to approximate parsing a YAML
    file this script can just parse. `_iter_model_entries` finds every
    mapping in the tree that carries BOTH an `id` and an `input`, tagged
    with whichever provider block it sits under; nothing here assumes which
    key of the surrounding structure holds the model list, only that `id`
    and `input` live in the same mapping, which is the one thing dsh's
    settings.yaml is stable about across its otherwise-nested shape.

    When more than one entry declares image input, one whose id contains
    "vision" wins without asking -- the owner's own naming convention
    ("dsv4f带vision", `deepseek-v4-flash-vision-exp`) makes that safe.
    Otherwise this refuses to guess and lists every candidate id, the same
    discipline `vibe.joblocate` uses when more than one job folder qualifies
    and nothing breaks the tie -- `provider` comes back `None` alongside the
    refusal, same as the model does.

    Returns `(model_id, provider_id, why)` -- `why` is `None` only when
    `model_id` was found unambiguously; `provider_id` may still be `None`
    even then, if that particular entry sat outside any recognisable
    `providers` block (see `_iter_model_entries`).
    """
    path = settings_path or os.path.join(
        os.path.expanduser('~'), '.dsh', 'settings.yaml')
    data, load_why = _load_settings_yaml(settings_path)
    if load_why:
        return None, None, load_why

    candidates = []
    providers_by_id = {}
    seen = set()
    for provider_id, mapping in _iter_model_entries(data):
        model_id = mapping.get('id')
        if not isinstance(model_id, str) or not model_id.strip():
            continue
        if not _declares_image_input(mapping):
            continue
        if model_id not in seen:
            candidates.append(model_id)
            providers_by_id[model_id] = provider_id
            seen.add(model_id)

    if not candidates:
        return None, None, ('no entry in {0} has an id with an input: list '
                            'naming "image"'.format(path))
    if len(candidates) == 1:
        return candidates[0], providers_by_id[candidates[0]], None

    vision_named = [c for c in candidates if 'vision' in c.lower()]
    if len(vision_named) == 1:
        return vision_named[0], providers_by_id[vision_named[0]], None

    pool = vision_named or candidates
    return None, None, (
        '{0} image-capable model id(s) in {1} and none is distinguishable '
        'by name ({2}) -- nothing here guesses. Name one explicitly via {3}.'
        .format(len(pool), path, ', '.join(pool), MODEL_ENV))


def _find_provider_for_model_id(model_id, settings_path=None):
    """Which provider block in ~/.dsh/settings.yaml lists `model_id` --
    for a model that came from an explicit job/env override rather than the
    vision scan above, which is the one case `resolve_model()` has a model
    id in hand but no provider tag that came free with it.

    Deliberately does not care whether the entry declares image input --
    unlike `_scan_settings_for_vision_model`, the caller already knows which
    model it wants; this only answers "who serves it". Returns
    `(provider_id, None)` on the first match, or `(None, why)` -- never
    guesses between two providers that happen to list the same id, the
    same refuse-the-tie discipline as everywhere else in this file; `why`
    names the id it could not place.
    """
    path = settings_path or os.path.join(
        os.path.expanduser('~'), '.dsh', 'settings.yaml')
    data, load_why = _load_settings_yaml(settings_path)
    if load_why:
        return None, load_why

    found = set()
    for provider_id, mapping in _iter_model_entries(data):
        if mapping.get('id') == model_id and provider_id:
            found.add(provider_id)

    if not found:
        return None, 'no entry for model id {0!r} in {1}'.format(
            model_id, path)
    if len(found) > 1:
        return None, ('model id {0!r} is listed under more than one '
                      'provider in {1} ({2}) -- nothing here guesses'
                      .format(model_id, path, ', '.join(sorted(found))))
    return next(iter(found)), None


# ------------------------------------------------------------------ file IO

def _read_text(path):
    with io.open(path, 'r', encoding='utf-8') as fh:
        return fh.read()


def _read_json(path):
    with io.open(path, 'r', encoding='utf-8') as fh:
        return json.load(fh)


def _write_spec(spec_path, spec):
    text = specformat.spec_text(spec)
    with io.open(spec_path, 'w', encoding='utf-8') as fh:
        fh.write(text)


# --------------------------------------------------------------------- run

def run(job, transport=None, sleep=time.sleep, clock=time.time, log=None):
    """The whole AI pass for one design. Never raises for an ordinary
    failure (unreachable dsh, timeout, unparsable reply) -- those become
    the documented "left untouched, note added" outcome. Only a bad job
    file (unreadable spec/evidence path) escapes, because no dsh behaviour
    could produce that and it means this script was invoked wrong.

    Returns a small dict describing what happened, for the caller's log.
    """
    log = log if log is not None else (lambda *_a: None)
    spec_path = job['spec_path']
    original_text = _read_text(spec_path)
    spec = specformat.normalize_spec(specformat.parse_spec_text(original_text))

    # Read up front, outside the try below -- these are plain `job.get()`
    # calls that cannot themselves fail, and the except block needs them
    # (for the orphan-session cleanup call) regardless of which line inside
    # the try raised, including one that runs before this point used to.
    base_url = job.get('dsh_base_url') or DEFAULT_DSH_BASE_URL
    timeout_s = int(job.get('timeout_s') or DEFAULT_TIMEOUT_S)

    session_id = None
    try:
        evidence = _read_json(job['evidence_path'])
        model, provider, model_why = resolve_model(job)
        if model_why:
            log(model_why)
        if not provider:
            # Fail BEFORE spending a session on it -- see the module
            # docstring's point 4: `session.selectModel` requires an
            # explicit provider id and this refuses to guess one, so there
            # is nothing a session could correctly be pinned to yet.
            raise DshError(
                'no dsh provider resolved for model {0!r} -- '
                'session.selectModel requires one and this pass will not '
                'guess (see the note above)'.format(model))
        image_paths = job.get('image_paths') or []

        prompt = build_prompt(evidence=evidence, regex_spec=spec,
                              datum_clause=job.get('datum_clause') or
                              '(no datum recorded -- treat any strand '
                              'reading as unresolved rather than guessing '
                              'which rail it is measured from)',
                              image_paths=image_paths)
        log('--- prompt ---\n' + prompt + '\n--- end prompt ---')

        session_id = create_session(base_url, timeout_s, transport)
        select_model(base_url, session_id, provider, model, timeout_s,
                    transport)
        log('session: {0} (requested model {1}/{2})'.format(
            session_id, provider, model))
        before = _rpc(base_url, 'session.history',
                     {'sessionId': session_id, 'maxMessages': 50000},
                     timeout_s, transport)
        baseline = len([m for m in _history_messages(before)
                       if m['role'] == 'assistant'])
        send_prompt(base_url, session_id, prompt, timeout_s, transport)
        reply_text = wait_for_reply(base_url, session_id, baseline, timeout_s,
                                    transport, sleep=sleep, clock=clock)
        log('--- reply ---\n' + reply_text + '\n--- end reply ---')
        actual_provider, actual_model = verify_session_model(
            base_url, session_id, provider, model, timeout_s, transport)
        log('confirmed session {0} ran on {1}/{2}'.format(
            session_id, actual_provider, actual_model))
        payload = extract_json_payload(reply_text)
    except Exception as exc:                          # noqa: BLE001
        reason = '{0}: {1}'.format(type(exc).__name__, exc)
        log('AI pass failed: ' + reason)
        if session_id:
            # A session was created successfully before something LATER in
            # this same pass failed -- see `cancel_session`'s docstring for
            # why this is the one orphan-cleanup case this script can
            # attempt safely (a trusted id already in hand, not a guessed
            # one). Best-effort: its own failure never masks `reason` above.
            cancelled = cancel_session(base_url, session_id, timeout_s,
                                       transport)
            log('session {0}: {1} after the failure above'.format(
                session_id, 'cancelled' if cancelled else
                'could not be cancelled -- may be an orphan on the dsh '
                'server'))
        _write_spec(spec_path, apply_failure_note(spec, reason))
        return {'ok': False, 'error': reason}

    merged, filled = apply_ai_fill(spec, payload)
    merged['source']['extractor'] = 'dsv4f-vision'
    merged['source']['ai_filled'] = filled
    _write_spec(spec_path, merged)
    log('filled: {0}'.format(', '.join(filled) if filled else '(nothing new)'))
    return {'ok': True, 'filled': filled}


# --------------------------------------------------------------------- CLI

def main(argv=None):
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument('--job', required=True,
                    help='path to the job JSON written by vibe.aiextract')
    ap.add_argument('--print-prompt', action='store_true',
                    help='print the prompt that would be sent, and exit '
                         '-- no dsh call, no file written')
    args = ap.parse_args(argv)

    with io.open(args.job, 'r', encoding='utf-8') as fh:
        job = json.load(fh)

    log_lines = []

    def log(msg):
        log_lines.append(msg)
        print(msg)

    if args.print_prompt:
        spec = specformat.normalize_spec(
            specformat.parse_spec_text(_read_text(job['spec_path'])))
        evidence = _read_json(job['evidence_path'])
        print(build_prompt(evidence=evidence, regex_spec=spec,
                           datum_clause=job.get('datum_clause') or '',
                           image_paths=job.get('image_paths') or []))
        return 0

    result = run(job, log=log)
    log_path = job.get('log_path')
    if log_path:
        try:
            with io.open(log_path, 'a', encoding='utf-8') as fh:
                fh.write(u'\n'.join(log_lines) + u'\n')
        except OSError:
            pass
    print(json.dumps(result, indent=2))
    return 0 if result.get('ok') else 1


if __name__ == '__main__':
    raise SystemExit(main())