Knowing an AI agent still works after you change the prompt
A staged conversational agent that guides a home buyer from initial discovery through market research and property matching to a generated offer document, dispatching to more than fifteen backing services.
Problem
The early stages of buying a home consume a large amount of expert time: discovery, market research, property matching, permit and school checks, comparables, and eventually drafting an offer.
Building an agent that covers that journey means orchestrating many external data sources under one conversation, holding state across sessions, and producing a legal document at the end.
The harder problem is not building it. It is knowing whether it still works tomorrow.
Constraints
- The journey is staged and cannot be skipped — offer generation is gated behind earlier steps being genuinely complete.
- Property data comes from external providers with their own coverage gaps and failure modes.
- Conversations span sessions, so state and memory have to persist and stay coherent.
- The output document has legal significance, so its generation is not a place for creative variation.
- Every prompt edit, model version bump or service change can alter behaviour anywhere in the journey.
My role
Primary engineer. I built the orchestrator, the service dispatch layer, the memory design and the evaluation harness.
Architecture
A unified orchestrator issues one model call per turn that returns structured output, with an optional second pass after any backing services have executed. Responses stream to the client over server-sent events.
More than fifteen services sit behind the dispatcher, covering property search and lookup, public records, comparables, permits, schools, weather, document analysis and web search.
Conversation history and memory summaries are cached in Redis; longer-term memory is vector-indexed in Pinecone. PostgreSQL holds the durable journey state.
Offer documents are produced from a template and rendered through a document toolchain bundled into the service image, so the model fills fields rather than generating a legal document freehand.
Key decisions
Evaluation built as infrastructure, not as a script
The system ships with a regression harness combining model-graded metrics with hard rules — assertions that must hold regardless of phrasing, like never generating an offer before the gating steps are complete. Alongside it, buyer personas are replayed through the full journey as automated tests. This is the part of the project I would point at first. Most AI systems have no way to detect that a prompt change degraded something, and without it every change is a guess.
Prompt-driven dispatch rather than native function calling
The model emits structured service tags that the application parses and routes, instead of using a provider's function-calling API. The benefit is portability and full control of the dispatch contract — swapping or mixing providers does not mean rewriting orchestration. The cost is real: parsing is more fragile than a typed API and needs its own validation and retry handling. With provider APIs having converged since, this is a decision I would revisit for a new system rather than defend as universally correct.
Structured generation for the document, free generation for the conversation
The conversation benefits from a model's flexibility. The offer document does not. Field-filling against a fixed template keeps the legally significant artifact deterministic while leaving the conversational surface natural.
Gating the journey in code, not in the prompt
Instructing a model not to skip ahead works most of the time, which is not good enough when the last step produces a purchase offer. Phase state is enforced by the dispatcher — services that belong to a later stage are simply unavailable until the earlier one completes.
Result
The agent runs the full journey from discovery to generated offer, with the evaluation harness catching behavioural regressions before they reach users.
The practical outcome of the harness is that prompt and model changes became routine rather than risky.
What I learned
- Accuracy is not one number for a system like this. What you need is a set of specific, checkable claims about behaviour, and a way to re-check all of them cheaply.
- Anything with legal or financial consequence should be produced by a template the model fills, not by generation the model controls.