Resource broker
Why
One machine, several competing workloads.
Tactic
Measure memory first, then allow, queue, or deny work before it can destabilize the host.
Cerbi Lab
Build notes from the
engineering bench.
Measured experiments in local AI, agent evaluation, reliability, and the systems behind Cerbi. Short on hype, long on outcomes.
What this is
A working lab, not an AI demo reel.
The goal is to make local agents measurably better at useful work while keeping the machine recoverable, the experiments disposable, and sensitive data local.
Public-safe engineering notes only
Exploratory by design
These are experiments and measured build notes, not product commitments or roadmap promises.
Fail closed
Uncertain privacy, bad validation, missing evidence, or resource pressure means the experiment stops rather than improvises.
Evidence earns promotion
Models, prompts, and agent policies advance only when builds, tests, benchmarks, and held-out evaluations say they should.
Learning loop
Improve from outcomes, not self-opinion.
A model does not get to declare itself better. Real tasks generate evidence. Verified evidence becomes curated learning data. Challengers then have to beat the current champion on frozen evaluations before anything changes.
Frontier teacher + local student
Hard cases can be escalated to a stronger teacher when policy allows, but teacher output still has to survive local verification before it is eligible for training.
01
Observe
Capture real work and evidence
02
Verify
Builds, tests, security, outcomes
03
Curate
Promote only useful examples
04
Adapt
Prompt, policy, LoRA or training
05
Challenge
Run frozen evaluations
06
Promote
Only when evidence wins
Learning data
Medallion architecture, adapted for agent experience.
Raw trajectories are useful, but they are not automatically good training data. We separate collection, verification, curation, evaluation-only data, and uncertain material so provenance survives the entire learning cycle.
Bronze
Raw experience
Prompts, tool calls, patches, failures, tests, timings, and provenance. Append-only and local when sensitive.
Silver
Verified + normalized
Schema-valid evidence, deduplicated tasks, normalized outcomes, sensitivity tags, and classified failures.
Gold
Training eligible
High-value examples with verification evidence. Gold means eligible for training, never automatic inclusion.
Holdout stays sealed
Frozen regression and adversarial tasks are evaluation-only. Dataset generation should fail if holdout material leaks into training.
Quarantine uncertain data
Possible secrets, malformed trajectories, unverified teacher output, and uncertain provenance stay out of the learning pipeline until resolved.
Tactics
Different problems need different controls.
We are deliberately testing improvements to tools, memory, prompts, model weights, evaluation, and orchestration separately so a score increase has a traceable cause.
Shadow challengers
Why
A new agent can look better while quietly regressing.
Tactic
Run challengers beside the champion in isolated worktrees and compare outcomes before promotion.
Failure memory
Why
A failed run is wasted if the system can repeat it tomorrow.
Tactic
Turn verified failures into reproducible regression candidates and permanent evaluation cases.
Local-first privacy
Why
Some code and data should never leave the machine.
Tactic
Separate offline, cloud-allowed, and adversarial workloads with explicit policy boundaries.
Daily log
What changed, and what actually happened.
Entries are intentionally compact. Private infrastructure details, credentials, personal information, and confidential source material are excluded.
August 23, 2026
Cold-start recovery and persistent compile-cache reuse are now validated
The local AI lab advanced from containment and rollback testing into reproducible cold-start engineering. A version-scoped persistent compile cache was prepared, populated from a fresh immutable inference runtime, sealed with provenance, and then reused in a separate warm-start experiment. Warm reuse cut compilation and initialization time substantially while preserving the same model profile and recovery boundary. Follow-on source and timeline analysis then isolated the remaining low-level NVIDIA allocation event to the post-KV, pre-autotune startup window, ruling out cold compilation, FlashInfer autotuning, and CUDA graph capture as its immediate trigger.
Control-plane tests
52 passed
Compile cache
Cold + warm validated
Warm compile
~9s vs ~36s cold
Training runs
0 — by design
Built / changed
- Validated a disposable offline experiment with no network, no GPU or host-device access, a non-root workload, read-only root filesystem, bounded temporary storage, dropped capabilities, and fixture-only read access.
- Measured unified-memory allocation across the resident inference services and separated observed values from estimates so projected memory savings cannot become scheduling authority by accident.
- Extended the resource broker so authorization requires validated evidence rather than zero-value placeholders or unverified estimates.
- Exercised a real failed model-startup rollback: the unsafe candidate was removed and the preserved production runtime returned to health while dependent services stayed available.
- Split profile validation into independent steady-state, cold-start, rollback, performance, and authorization states instead of treating a healthy endpoint as proof of every operating condition.
- Detected and repaired an evidence-sealing defect by stopping and joining writers before hashing, checking for writable file descriptors, performing stable two-pass hashing, writing a terminal seal, and rejecting normal post-seal mutation.
- Added generation-scoped persistent compiler-cache storage with runtime fingerprints, stable storage identities, artifact manifests, cryptographic inventory, and separate cold-validated and warm-validated states.
- Populated the cache from one fresh immutable cold start, then reused the exact validated generation in a separately controlled warm start without changing the model profile.
- Reclassified low-level NVIDIA memory events contextually: isolated events that recover without system pressure are distinct from repeated allocation failures accompanied by real pressure or failed readiness.
- Traced the installed inference runtime's post-KV startup path and compared sealed runs to locate the recurring NVIDIA allocation event before both FlashInfer autotuning and CUDA graph capture.
Outcomes
- The persistent compile cache reached both cold-validated and warm-validated states with complete artifact inventories and repeatable hash verification.
- Warm reuse reduced compilation from about 36 seconds to about 9 seconds, engine initialization from about 60 seconds to about 26 seconds, and startup-to-health by roughly 32 seconds.
- Model loading, KV capacity, and overall inference allocation stayed effectively equivalent between the cold and warm comparisons, keeping the cache as the principal changed variable.
- The same isolated NVIDIA allocation event still appeared after KV sizing and before later autotuning and graph-completion work, so cold compilation is no longer treated as its sole explanation.
- Across multiple phase-resolved immutable starts, the allocation event stayed in the same narrow KV-to-autotune window even when compile time changed substantially.
- Installed-source analysis confirmed CUDA graph capture and FlashInfer autotuning had not begun when the event occurred, narrowing the next investigation to KV/backend/metadata allocation and immediate warmup setup.
- No Xid, kernel OOM, GPU reset, sustained swapping, or persistent pressure accompanied the validated cold or warm cache runs.
- Production rollback remained deterministic: each authorized startup test restored the preserved production runtime and dependent services returned to their expected health state.
- Fifty-two control-plane tests pass, with NORMAL-A inference performance and steady-state qualification still intentionally unapproved and training still disabled.
What we learned
- A service being healthy in steady state does not prove that it can be reproduced safely from a cold start.
- Compile-cache validity, profile validity, and production authorization are separate states; proving one must not silently promote the others.
- Persistent AOT and compiled-graph reuse can materially improve startup time without changing model loading or KV capacity.
- Driver error strings need phase and pressure context. An identical event can be recoverable in one startup and part of a repeated failure pattern in another.
- Startup phase correlation matters more than proximity alone: a later optimization stage should not be blamed for an event that consistently occurs before that stage begins.
- Experiment outcome and evidence validity are separate states: invalid evidence cannot authorize policy, even when the underlying runtime behavior is still useful diagnostically.
- Evidence is not immutable until every writer is stopped, hashes are stable across rereads, and later mutation is rejected.
Next
- Trace the allocation and metadata work immediately after KV sizing to identify the narrowest supported single-variable isolation test.
- Keep CUDA graph and FlashInfer-autotune controls unchanged unless later evidence places the event inside those phases.
- Only after startup behavior is understood should NORMAL-A move into inference, context-size, concurrency, latency, throughput, and longer stability qualification.
- Keep training unauthorized until measured reserves, recovery behavior, performance, and evidence-integrity gates are validated for the training operating mode.
August 22, 2026
Learning control plane is now in place
The local AI lab moved from architecture sketches to a tested, shadow-only learning control plane: provenance, Medallion promotion, evaluation, resource planning, experiment manifests, model governance, and public-safe outcome tracking now exist before any training workload is allowed to run.
Control-plane tests
26 passed
Training runs
0 — by design
Production disruption
None
Runtime mutation
None
Built / changed
- Completed the first shadow-only learning control plane using standard-library Python and SQLite-backed metadata.
- Implemented Bronze, Silver, Gold, Holdout, and Quarantine data states with deterministic provenance and content hashing.
- Added hard training protections so Holdout, Quarantine, unverified, duplicate, or explicitly non-trainable artifacts cannot enter generated datasets.
- Added immutable experiment manifests, a metadata-only job queue, and a fail-closed resource broker that can inspect the DGX without mutating runtime state.
- Added model-neutral L1–L7 evaluation scaffolding, failure-to-regression candidates, and conservative champion/challenger promotion rules.
- Added dry-run routing for local, specialist, frontier-cloud, and human-review paths with sensitivity-aware restrictions.
- Added initial synthetic Cerbi scenarios covering PII exposure, cardinality pressure, and OpenTelemetry backpressure.
- Expanded the local control CLI with registry, experiment, dataset, broker, memory, health, and dashboard inspection commands.
Outcomes
- Twenty-six unit and integrity tests passed, including holdout-contamination, experiment-overwrite, registry-integrity, and provenance protections.
- Static validation passed across Python, shell, JSON, YAML, Compose overlays, SQLite integrity, secret scanning, and read-only resource inspection.
- No production container, service, model allocation, network, listener, package set, or production data changed during this phase.
- The lab can now record evidence and reject unsafe training inputs before a model-training runtime is introduced.
- The next safe runtime experiment is containment-only: a short-lived, no-GPU, network-isolated job that proves evidence capture and cleanup without model loading.
What we learned
- The durable asset is the evaluation and provenance system, not any single base model.
- Gold should mean eligible for training, never automatically selected for training.
- Prompt and tool-policy improvements can be evaluated independently from model-weight changes.
- A resource broker should fail closed until memory reserves are measured rather than guessed.
- Teacher output is still untrusted data until independently verified; it belongs in Quarantine first.
Next
- Run the isolated no-GPU containment experiment and verify immutable evidence capture.
- Benchmark unified-memory headroom before enabling authoritative concurrency or profile budgets.
- Add the frozen benchmark runner and schema-aware creation commands.
- Begin collecting verified trajectories and regression cases before enabling any training runtime.