AML auto-modeling (agent-driven fitting & shadow twins)

Module codename aml (Auto Modeling Lab): on top of MAS mirroring DAQ and the time-series store, a built-in deep-learning expert team drives the whole chain on its own — pull data → clean → train → platform evaluation → gate iteration → HITL promotion → shadow reference.

pull data (isolated triple) → clean → write code & train → authoritative evaluation
  → gate iteration → HITL promotion → shadow reference

Before tuning parameters, a control agent queries the production model through aml_model_reference to get a shadow reference of "proposed parameters → predicted output trajectory" (the front end of the digital twin / MPC); on the REST side that is POST /api/workshop/aml/models/:id/predict. The full manual (with literature mapping and ADRs) lives in the repository at docs/aml.md.

Asset root

AML artifacts (dataset arrays, job workspaces, ONNX artifacts, the uv virtual environment) converge under the aml/ subdirectory inside the config root — assets share the root with runtime state, so aw update or switching checkouts never loses models or the venv; the whole tree can be copied, deleted and rebuilt:

Order	Condition	Asset root
①	AW_AML_DIR is set	that directory (explicit override, highest priority)
②	running inside a checkout (AW_MODE is not home)	<checkout config root>/.AgentWorkShop/aml
③	anything else (home mode / global install)	<awHome>/aml — AW_HOME redirects it, it is not hard-coded to ~/.AgentWorkShop

Directory skeleton:

aml/
  .venv/            Python environment created/managed by uv (deps pinned in aml/python/requirements.txt)
  datasets/<id>/    spec.json fetch spec · manifest.json column defs · report.json quality report · arrays/*.f32
  jobs/<id>/        job.json submission contract · workspace/(train.py+amlkit.py) · run.log · artifacts/
  models/<id>/      model.onnx + io_spec.json (I/O contract, immutable)
  twins/            hybrid-twin artifacts (frozen scenes / compiled PhysicsSpec / trials & certificates)
  runtime/          uv.json (one-click install record) / env.json
  tools/            uv binary installed by the platform (no admin, no system PATH changes)

GET /api/workshop/aml/env returns the effective path and its source, which the "Runtime environment" panel in the UI displays directly.

Settings (the aml group, 16 keys)

The aml group on the settings page ("Runtime config") holds 16 descriptors: thresholds and quotas are live (effective as soon as saved), Python path keys are restart.

config key	Default	Effect	Notes
aml.python.pythonBin	empty	restart	pin the Python interpreter (empty = auto-detect)
aml.python.uvBin	empty	restart	pin the uv executable (takes priority over ./aml/tools)
aml.python.indexUrl	empty	restart	pip index mirror (speeds up dependency provisioning)
aml.job.timeoutMs	1800000	live	wall-clock limit per job (30 minutes)
aml.job.maxConcurrent	2	live	concurrent training jobs
aml.job.diskQuotaMb	2048	live	job disk quota
aml.job.stallMs	600000	live	stall watchdog (no progress ⇒ treated as stalled)
aml.dataset.maxRows	200000	live	row cap per dataset
aml.dataset.retentionDays	30	live	retention for unreferenced datasets
aml.dataset.allowCrossRecipe	false	live	allow fetching across recipes (off by default; the isolated triple is mandatory)
aml.model.retiredKeepDays	90	live	retention for retired artifacts
aml.gates.nrmse	0.10	live	G1 one-step NRMSE ceiling
aml.gates.rolloutNrmse	0.25	live	G2 multi-step rollout NRMSE ceiling
aml.gates.valTestGap	0.2 (20%)	live	G3 generalization-gap ceiling
aml.gates.minRows	500	live	G4 minimum training rows
aml.gates.minRuns	3	live	G4 minimum batch count

CLI examples: aw config set aml.gates.nrmse 0.12, aw config get aml.job.maxConcurrent. Besides the standard AW_AML_* mapping there are 7 declared env aliases: AML_PYTHON_BIN, AML_UV_BIN, AML_PYTHON_INDEX_URL, AML_JOB_TIMEOUT_MS, AML_JOB_MAX_CONCURRENT, AML_DISK_QUOTA_MB, AML_JOB_STALL_MS.

REST tree (/api/workshop/aml/**)

Surface	Endpoints	Notes
Overview	GET /api/workshop/aml	runtime status + environment self-check + recent dataset/model/job counts (one request paints the first screen)
Runtime env	GET .../aml/env, POST .../aml/env/{uv,venv,recheck}	uv/Python detection, one-click uv install, venv create/rebuild, re-detect (async tasks + progress polling)
Entity reconciliation	GET .../aml/entities, POST .../aml/entities/prune	SQLite metadata ↔ ./aml entities reconciled id by id (missing / orphan / contract incomplete); prune orphan directories
Datasets	GET/POST .../aml/datasets, GET/PATCH/DELETE .../aml/datasets/:id, GET .../aml/datasets/:id/preview	the isolated triple (line/product/recipe) is mandatory; snapshots are immutable (sha256 is the reproduction anchor); preview sampling
Jobs	GET/POST .../aml/jobs, GET/DELETE .../aml/jobs/:id, POST .../aml/jobs/:id/{cancel,retry}, GET .../aml/jobs/:id/logs	FIFO queue + concurrency cap; ##AML NDJSON progress protocol; incremental log fetch
Experiments	GET .../aml/experiments?datasetId=	lineage tree + gate details + primary metrics
Models	GET .../aml/models, PATCH/DELETE .../aml/models/:id, POST .../aml/models/:id/{predict,promote}	registry stage transitions; predict = what-if prediction; promote = stage promotion (ONNX deep check before production)

Permissions align with line grants: reading datasets and predicting need readonly, promotion needs operate. Deletion is reference-protected (a dataset referenced by an experiment or model, a production-stage model, a running job, or a job that produced a model is always refused).

Agent tools (14)

Tool	Purpose
aml_node_catalog	inventory of modelable assets (node semantics × last-24h data volume)
aml_dataset_build	build a dataset snapshot (isolated triple + cleaning + beatMs alignment + windowing)
aml_dataset_stats	dataset statistics report (control→target lag cross-correlation / correlation / per-run profile)
aml_job_submit	submit a training job (train.py code may be inlined; a hybrid_residual job can omit code and use the built-in reference trainer)
aml_job_status	job status snapshot
aml_job_logs	job logs (troubleshooting)
aml_job_cancel	cancel a job
aml_leaderboard	leaderboard (lineage + gate details + primary metrics)
aml_model_find	pick a model by scene semantics (line / objective / keyword filters over the registry)
aml_model_promote	request a stage promotion (lead-only; goes through human approval)
aml_model_reference	query the shadow reference of a production model (tuning what-if; proposed parameters via controls)
aml_training_plan_list	list modeling plans (auto policies retrain as new data lands)
aml_training_plan_create	create a modeling plan (scene + recipe + objective + auto policy)
aml_training_plan_train	trigger one training run of a modeling plan immediately

These 14 are host tools defined in .AgentWorkShop/prompts/host-tools.json (82 entries on the host tool surface); they are a different surface from MCP (server/mcp/workshop-server.ts, 25 in-process tools). Models carry a human-readable identity: label (line · recipe · objective), description and line_id/objective_id.

Hybrid twin × MPC (channel profile hybrid_twin)

AML also owns the hybrid twin plane: a grey-box physics core with a bounded PyTorch residual, plus the SceneContract / PhysicsModelManifest / TwinSnapshot / ObjectiveProfile / VirtualTrial / RecommendationCertificate contracts. A channel opts in through its profile (legacy by default, hybrid_twin to enable), which injects twelve profile tools (twin_calibration_request is a general tool, visible to both profiles):

Tool	Purpose
twin_provider_catalog	read registered providers / scene packs / solver adapters with generation and health (read-only)
twin_scene_discover	read the Agent's bound DAQ/DCW nodes and infer control/state/disturbance/target/guard semantics with evidence and confidence
twin_scene_compile	build a SceneContract draft from bound nodes and the scene prompt (draft by default; user confirmation precedes go-live)
twin_scene_freeze	freeze the scene contract after user confirmation (a given scene_id/version can never be overwritten; changes require a new version)
twin_physics_spec_draft	draft a skeleton PhysicsSpec from the frozen scene (one first-order relaxation equation per state; τ/gain as calibratable prior-boxed parameters)
twin_physics_spec_validate	validate a declarative physics model's whitelisted AST / variable and parameter references / units / bounds / stability / monotonicity; no arbitrary code execution
twin_physics_spec_compile	compile a frozen scene's PhysicsSpec into an isolated Generic Declarative Provider candidate and save the artifact (no production registration, no PLC write)
twin_scene_read	read the scene contract (devices / signals / constraints / objectives)
twin_snapshot_create	create a TwinSnapshot (freshness enforced; stale → SNAPSHOT_STALE)
twin_trial_run	run a virtual trial: all-trajectory hard constraints reject unsafe candidates, candidateExecuted=false
mpc_optimize	MPC optimisation (server-side strategies safe_small_step / precise_search)
twin_gate_evaluate	gate + benefit evaluation; only a pass issues a RecommendationCertificate
twin_calibration_request	continuous-calibration registration (duplicates deduped inside the cooldown)

Note: the injected twin tools are gated by profile (a legacy channel gets none of them). The table above is the full twin surface of a hybrid_twin channel; the declarative physics model (PhysicsSpec) protocol and validator are specified in docs/aml-agent-auto-twin-builder-plan.md §4.3/§6.

Recommendation-only by default: insufficient data → safe_small_step; model cleared gates → precise_search; no real DCW write happens until a certificate is certified. Plan and acceptance record: docs/aml-hybrid-twin-mpc-integration-plan.md, docs/aml-hybrid-twin-implementation-acceptance.md.

Closed-loop wiring (v0.7.47):

1. twin_snapshot_create accepts auto_daq: true and samples every node through the Agent's real DAQ bindings; the scene contract's observations / states must carry nodeId — the built-in default scene does not, so inject it via scene_json. Otherwise the watermark stays 0 (watermark=0) and the tool now returns an actionable diagnostic instead of a bare fresh=false.
2. twin_trial_run only accepts the full snapshot artifact produced by twin_snapshot_create (missing snapshotId / dataQuality / snapshotHash is rejected outright) and always reports candidateExecuted=false.
3. When twin_gate_evaluate receives model_id, it writes the 12-check verdict back into that model's twinEligibility (gatePassed / recommendationEligible / uqPassed / oodPassed / physicsPassed), and mpc_optimize then upgrades its server-side strategy from safe_small_step to precise_search — the step where a trained AML model actually enters the twin loop, still without writing DCW directly.
4. Hard-constraint evaluation is fail-closed: every constraints[].id must map to a trajectory observation or guard (weight / flash_rate / sink_rate / melt_temperature / cavity_pressure / pressure / temperature). Unmapped ids and empty trajectories are judged not passed, with a cannot be mapped to an observation/guard detail. v0.7.47 and earlier skipped such ids and reported "whole trajectory passed", which handed callers a false constraint verdict when scene_json carried a wrong id (caught by a real Agent team: a predicted 30.3 g weight below the window floor was still reported as passing).

Scenes are no longer hard-wired into the tools: physics cores / scene packs / objectives / solvers can be registered by a plugin through ctx.twin into the Twin Provider Registry, and the twin tools resolve them via SceneContract.physicsProfileId (falling back to the bundled twin-injection-default). A hot reload stages a new generation and drains the old one while in-flight trials and MPC hold leases, so running work is never invalidated. Field tables, lifecycle and a minimal example: plugin development guide → hybrid-twin provider plugins.

Any-scene modeling loop (v0.7.50)

The loop is no longer tied to a bundled scene — any line with bound nodes can walk "model → gate → deploy":

1. twin_scene_discover infers semantics with evidence → twin_scene_compile drafts the SceneContract → you confirm → twin_scene_freeze freezes it (immutable versions);
2. twin_physics_spec_draft proposes a skeleton PhysicsSpec → you refine the equations against real operating data → twin_physics_spec_validate / twin_physics_spec_compile (whitelisted AST, no arbitrary code);
3. aml_job_submit { job_kind: 'hybrid_residual' } needs no code: the built-in reference trainer runs the A→B→D protocol — physics parameter calibration (stage A) → a 3-member bounded residual ensemble (stage B) → conformal q90 coverage (stage D); the calibrated θ is written back into the embedded spec and shared by evaluation, prediction and trials;
4. twin_trial_run / mpc_optimize with a model_id drive the rollout from the trained hybrid model (ModelBackedHybridProvider), with UQ coverage and member disagreement coming from training artifacts; a dataset whose line/product/recipe mismatches the scene is rejected (TWIN_MODEL_SCENE_MISMATCH, recipe isolation);
5. signals registered via twin_calibration_request are consumed by the 60 s continuous calibration worker, which submits retraining candidates automatically; promotion still goes through HITL + Twin Gate (AML_TWIN_WORKER_DISABLED=1 turns the worker off).

Two-mode optimization channel templates (v0.7.50)

Two built-in templates split "training" and "optimizing" into mutually exclusive channels — training while optimizing is impossible by construction:

Template	Tool face	Can do
chtpl-aml-training-default	modeling face (22 tools): datasets / jobs / scene compile / physics specs / training plans	train and evaluate; no plant writes, no MPC
chtpl-aml-optimization-default	acting face (6 tools): snapshot / trial / gate / MPC / twin_bayes_optimize / optimization_explore	search and fine-tune; no training-family tools

At instantiate time the optimization channel picks its optimizationMode: exploration (governed micro-writes, each setpoint paired with its response read-back in the optimization_explorations ledger; no model required) or aml (boundModelId required — a gate-cleared model must be bound before the acting face unlocks).

The built-in team team-aml-shadow ("AML shadow modeling team") packs 1 lead (chief data scientist) + 3 workers (data engineer / training engineer / evaluation engineer) with omp as the default harness; add members when creating an instance from the team. The team playbook lives in .AgentWorkShop/prompts/aml-playbook.md.

Python runtime

The first training run detects uv and Python automatically and provisions pinned dependencies into ./aml/.venv (aml/python/requirements.txt) — this is the only stage that touches the network, and aml.python.indexUrl can point it at a mirror. The "Runtime environment" panel on the /aml page provides the full walkthrough:

1. uv detection — order: the aml.python.uvBin setting > ./aml/tools (installed by the platform) > the ./aml/runtime/uv.json record > system PATH; the source is shown with the status;
2. one-click uv install — runs the official installer with UV_INSTALL_DIR=./aml/tools and UV_NO_MODIFY_PATH=1: no system PATH changes, no administrator rights; on failure it falls back to python -m pip install --target ./aml/tools uv;
3. create/rebuild the training environment — with uv it runs uv venv + uv pip install, without uv it falls back to python -m venv; readiness is keyed on a marker file, so a dependency or mirror change marks the environment for rebuild;
4. re-detect — install uv/Python by hand and it applies without a restart;
5. metadata ↔ entity reconciliation — see the REST tree above; orphan entities can be pruned in one click.

Without Python a job reports AML_PYTHON_MISSING with guidance; starting with AML_STUB=1 runs the whole chain through a stub runner (a stub model is unpredictable, so promotion is blocked by the deep check).

Gates G1–G5

Gate	Criterion	Default threshold	Setting key
G1	one-step NRMSE	≤ 0.10	aml.gates.nrmse
G2	multi-step rollout NRMSE	≤ 0.25	aml.gates.rolloutNrmse
G3	generalization gap |test−val|/val	≤ 20%	aml.gates.valTestGap
G4	training rows / batches	≥ 500 rows, ≥ 3 batches	aml.gates.minRows / aml.gates.minRuns
G5	artifact completeness	ONNX + io_spec present	—

Authoritative metrics are recomputed by the platform evaluator — never taken from the agent's own report — and every threshold is live tunable.

Governance and safety boundaries

• Training code runs only inside the job workspace (./aml/jobs/<id>/workspace/); the platform executes it as a separate subprocess with wall-clock / stall / concurrency limits and kills the whole process tree on cleanup;
• Dataset node sets ⊆ the agent's DAQ bindings (they can only shrink, never grow); on the REST side line grants apply;
• Every side effect goes through recordOps (audit_log + live ops.log frames), and agent actions are attributed to "Channel/Member";
• SQL is fully parameter-bound, workspace paths are validated against escaping the root, and deletion protection preserves the reproducibility trail of model artifacts.

Related pages

• Multi-harness agent teams — harness selection and capability surface of the built-in AML team
• Configuration — the descriptor system the 16 aml.* keys live in, and their effect
• Line-level permissions — the authorization model for dataset reads and model promotion