What’s shared
L2 to L4. Provider adapters, the tool registry, tracing, policy and the durable runtime are the parts every application needs and nobody should rebuild.
How I engineer AI systems that a business can actually run on: a lightweight shared integration core, a durable agent harness with human approval as a first-class state, and independent applications on top. Built to be observable, recoverable, testable and safe to operate.
01 Design thesis
Most failures in production AI come from giving a model too much freedom and too little structure. Every system here follows five rules.
The model selects from validated, schema-defined options: which tool, which fields, which widget. Deterministic code compiles and executes those choices. In the analytics module the model never writes SQL, never writes UI code and never types the numbers on screen.
Model output, retrieved documents, emails and tool responses are data, never instructions. Authority comes from the policy layer and from humans, not from text in the context window.
A run is a persisted state machine with checkpoints, not a chat transcript held in memory. It can pause for days waiting on approval, survive a deploy and resume without repeating side effects.
Every run emits a trace with token usage, latency and cost. Behaviour is pinned with scripted-model tests and regression datasets, so model or prompt changes are evaluated, not eyeballed.
A small common core handles providers, tools and observability. Each application keeps its own frontend, data model and deployment. No monolith, no framework built ahead of real use cases.
02 Architecture
Requests flow down through the stack. Nothing below the permission boundary is reachable without passing the policy layer, whatever the model asks for.
L2 to L4. Provider adapters, the tool registry, tracing, policy and the durable runtime are the parts every application needs and nobody should rebuild.
Frontends, data models, databases and deployments. Each application at L5 is independently useful and independently deployable, usually on its own subdomain.
New capabilities like retrieval, document parsing or a business API arrive as registered tools with schemas and a side-effect class. The core doesn’t change to accommodate them.
03 Anatomy of a run
Follow a single inbound quote request through the harness. Step through it, or let it play. Illustrative payloads, real design.
The inbound email is normalised into an event and persisted before any model is called. The message ID becomes the idempotency key, so a duplicate webhook delivery maps to the same run instead of a second one.
POST /runs
{
"workflow": "enquiry.triage",
"trigger": { "type": "email.received", "id": "msg_8c1e…" },
"idempotencyKey": "msg_8c1e…"
}
→ 201 { "run": "7f3a", "status": "queued" }
System instructions, the tool manifest for this workflow only, and relevant records. Inbound and retrieved content is wrapped and labelled as untrusted data. Token budgets are set per run, and the model is told to wrap up well before the ceiling.
context = [
system("enquiry.triage@v4"),
tools(["crm.find_contact", "crm.create_lead", "email.draft_reply"]),
untrusted("email", msg.body), // data, never instructions
retrieved("price_list", chunks, { cite: true }),
]
budget = { maxTokens: 80_000, wrapUpAt: 0.7 }
The provider adapter sends one normalised request and returns one normalised tool call, whether the model behind it is OpenAI or Anthropic. Schemas are written once in Zod and converted per provider.
tool_call crm.create_lead {
"contact": { "name": "Sam Okafor", "company": "Riverside Café" },
"intent": "quote",
"items": [{ "sku": "MUG-BR-12", "qty": 40 }],
"dueDate": "06-03"
}
Every tool input is parsed against its schema server-side. A failure doesn’t crash the run: it returns to the model as a structured tool result so it can correct itself, with a hard cap on repair attempts.
CreateLead.safeParse(args)
✗ dueDate: expected ISO date (YYYY-MM-DD), got "06-03"
tool_result { ok: false, error: "dueDate must be YYYY-MM-DD" }
→ model retries: "dueDate": "2027-06-03"
✓ schema valid (attempt 2 of 3)
Each tool is registered with a side-effect class. Reads run freely within scope; writes to business systems and anything outbound require approval under this workflow’s policy. The run checkpoints and pauses.
registry.get("crm.create_lead")
effect: "write:crm"
scope: "leads:create" // least-privilege credential
policy: requireApproval
⏸ checkpoint 7f3a·05 saved · status → paused · approval #219 opened
The approval lands in the ops console with the proposed action, the evidence behind it and the trace so far. Approval can take seconds or days. The run holds no process or connection while it waits; it’s just state.
approval #219 crm.create_lead · run 7f3a
proposed by: model (claude-opus-5-5)
evidence: email msg_8c1e…, price_list §3
decision: approved by owner (edited: note added to lead)
▶ resume from checkpoint 7f3a·05 // steps 1–4 are not re-run
The write carries a key derived from the run and step, so a retry after a timeout can’t create a duplicate lead. Transient failures back off and retry; permanent ones surface in the console with the full context.
crm.create_lead(args, { idempotencyKey: "7f3a·07" })
attempt 1 504 upstream timeout retry in 2s
attempt 2 200 ok · lead_5521 (deduplicated upstream)
email.draft_reply(…) ok // draft only: sending is a separate, gated tool
Automatic checks run on the output, the trace is stored with tokens, latency and spend, and a sample of runs flows into regression datasets used to evaluate future prompt and model changes.
eval schema ✓ pii-redaction ✓ tone ✓ grounded ✓
trace 7 spans · 2 model turns · 3 tool calls · 1 approval
usage 9,412 in / 611 out tokens · $0.004 · 1.9s active
status completed · sampled → datasets/enquiry-triage
04 Core components
The reusable pieces, and the decisions inside them. Advanced packages are extracted only when a real project needs them.
One interface for model calls across providers, without flattening them into a lowest common denominator.
Tools are typed, discoverable units with explicit risk, not functions the model happens to be able to call.
Work that outlives a single request, a deploy or a human’s lunch break.
Grounding answers in a business’s own documents, and only the ones the asker may see.
Every decision the system makes can be inspected after the fact.
Measuring whether the system does its job, before and after every change.
05 Security model
Never allow a model-generated command, query or request to run merely because the model asked. Here’s how each class of risk is handled.
All code is written on the assumption it may be published: .env.example files with placeholders, server-side auth checks, ownership validation on every resource, and no privileged infrastructure reachable from public demos.
06 Proof: the first module, built
The analytics module, built end to end as a standalone application. Ask a market-research question in plain English; an agent queries real data and composes a dashboard while you watch. It’s where the core’s patterns were proven first.
The model picks registered widgets and maps columns from query results by ID. A broken widget is dropped; the rest of the view survives.
Figures quoted in AI-written insights are matched against the underlying results. Anything that can’t be matched is flagged on screen.
Per-visitor rate limits, a persisted daily token budget, per-run ceilings, concurrency caps and a kill switch that needs no restart.
Query compiler, hydration and the agent loop are covered by tests with a scripted fake model. No API calls, no cost, deterministic.
07 Module matrix
Each application exists to demonstrate a distinct engineering problem, not just a feature list.
08 Decisions & trade-offs
Short decision records. Each one has a cost, and that cost is accepted on purpose.
Chose: a small integration layer shared by independent apps.
Over: one application hosting every demo, or a framework built up front.
Cost: some duplication across frontends. Worth it to keep each app shippable alone.
Chose: the model fills validated specs that deterministic code compiles.
Over: letting the model write SQL, UI components or scripts.
Cost: less open-ended flexibility. Every new capability needs a schema.
Chose: a single tool-calling loop as the default.
Over: multi-agent orchestration by default.
Cost: fewer impressive diagrams. Delegation is added only where it measurably beats the simpler design.
Chose: a common adapter interface with an escape hatch to native features.
Over: an abstraction that hides what makes each provider good.
Cost: adapters carry provider-specific code paths and tests.
Chose: human approval modelled in the run state machine with checkpoints.
Over: synchronous confirmation inside a request.
Cost: durable storage and resume logic from day one.
Chose: recorded runs replayed at zero cost, with a guarded live mode.
Over: an open live endpoint paid for by every visitor’s curiosity.
Cost: visitors see curated questions first; live questions are rationed.
09 Roadmap
The full reference system will run on its own subdomain. It’s built in phases, each shipping something that works on its own.
Agent Workbench, the analytics module, deployed as a standalone app with replayed and guarded live runs.
LiveWorkflow automation, knowledge assistant and a full-stack business app. The shared core is extracted here.
NextSecure tool execution, persistent state, tracing, approvals, evaluation, recovery.
FlagshipResearch and reporting, document processing, marketing studio, extended analytics.
Planned10 Talk shop
Hiring, collaborating, or evaluating whether I can build this for your business? I’m happy to walk through the architecture, the code and the trade-offs in detail.
bloondevelopment@gmail.com