RAG Implementation: Starting at the base metal
Good AI solutioning starts with understanding a client's pain points, appetite for change, and real-world constraints such as budget, regulatory compliance, data readiness, and competitive positioning.
I've been helping a medical practice think through how AI can reduce day-to-day operational friction. While prototyping an internal chatbot, I went to the base metal: mapping what information each prompt actually needs and which data source the system should use to answer it.
Beyond prompt engineering, the system has to handle actual operational constraints around data access, audit, provenance, and guardrails.
The biggest mistake in enterprise AI is treating all data the same.
In reality, the data dictates the path:
🔹 Structured data requires exact retrieval from a source of truth.
🔹 Unstructured knowledge requires semantic retrieval based on meaning and context.
An LLM should never be allowed to infer concrete, fixed information such as an appointment date or a medication list. Semantic RAG is not the solution when the answer lives as structured data.
The real architecture isn't the chatbot interface. It is the routing layer underneath: deciding when to use a structured query, when to use semantic retrieval, when to synthesize, and when to escalate instead of faking confidence.
#AIArchitecture #EnterpriseAI #HealthTech #SolutionsArchitect #SystemDesign
Model Routing Architecture
The most capable model isn't always the best model. Healthcare just happens to make that painfully obvious.
In a clinical setting, one request might be: "Pull the payer's prior authorization criteria for this medication." The next might be: "Read this 15-page faxed discharge summary, extract the failed step therapies, align them to payer criteria, and draft a prior authorization."
Those requests have fundamentally different computational needs. Treat them the same, and the tradeoffs show up immediately. Route everything to a frontier model and you incur unnecessary latency, token consumption, inference cost, and lower throughput. Route everything to a smaller model and you risk degraded accuracy, missed clinical context, and failures on the requests that actually require deeper reasoning.
The answer isn't a better model. It's better architecture.
A deterministic routing layer evaluates complexity, intent, and modality before a prompt is processed, then sends the request to the appropriate engine. Straightforward retrieval goes to a smaller model paired with deterministic systems like APIs or vector search, where you don't need an agent improvising against your schema. Complex multimodal reasoning goes to a frontier model, but only when the task actually warrants it.
The hard part isn't the routing logic. It's understanding the workload well enough to classify requests so you can optimize for quality, latency, throughput, and cost simultaneously. Going to the base metal means starting with the workload to identify its complexity, modalities, failure modes, and performance requirements and letting those characteristics drive the architecture.
These are the kinds of tradeoffs I enjoy thinking about. The interesting engineering challenge isn't finding the model that tops the latest SWE-bench leaderboard. It's building an architecture that consistently delivers the best outcome within real-world constraints.
Continuous Evaluation of AI Models (Part 1 of 3): Why Successful Pilots Don't Predict Production Performance
Over the next three posts, I want to go to the base metal of a question every organization deploying AI will eventually face: how do you know the models supporting your core workflows are still doing what you need them to do?
Every AI deployment follows the same arc: define evaluation criteria, test edge cases, validate performance, and run a pilot. The pilot succeeds, stakeholders sign off, and the model goes into production. And then, over time, production starts asking questions the pilot was never designed to answer. That's because many organizations make a critical category error.
Pre-deployment and production evaluation answer entirely different questions. A pilot proves that a model performs well on anticipated scenarios, much like testing a car on a closed track. It tells you very little about how that same model behaves in the "downtown traffic" of real users, evolving workflows, changing data, and unforeseen edge cases.
I saw this firsthand on a recent healthcare AI pilot. In healthcare, physician trust depends on having confidence that AI models continue to perform reliably after deployment. We built a rigorous pre-deployment evaluation framework using real patient data, and the model performed exceptionally well. But once a model enters production, the challenge shifts. Workflows and requirements evolve, user behavior changes, upstream systems change, and real-time ground truth isn't always available. Those are conditions a pilot simply cannot simulate.
Passing the pilot isn't the finish line for evaluating AI models. It's where continuous evaluation begins.
In Part 2, I'll explore what continuous evaluation should look like once an AI system is in production.
#ArtificialIntelligence #EnterpriseAI #AIArchitecture #AIModels #ModelEvaluation #ContinuousEvaluation #AIGovernance #HealthcareAI #AgenticAI #GenAI
Continuous Evaluation of AI Models (Part 2 of 3): What "Working" Actually Looks Like in Production
In Part 1, I discussed one of the biggest mistakes organizations make: assuming that because an AI model performed well in pilot, it will continue performing well once it's deployed. If you missed it, you can read it here: https://lnkd.in/eFRfWedT
Production is where the real test begins.
Continuous evaluation is not just about monitoring a model. It is about detecting when something has changed and deciding what to do about it. A dashboard is reporting. Evaluation begins when the information leads to action.
In practice, that happens in three ways:
- Detect problems before users notice them
- Monitor for data drift to identify shifts in incoming data
- Treat drift as an early warning sign before performance degrades
- Review what the model is actually producing
- Use human review for uncertain or high-impact decisions
- Use an LLM to evaluate another LLM at scale against a defined rubric
- Validate changes before deployment
- Run new models in shadow mode before replacing the current one
- Compare against ground truth when available. Otherwise, use signals like user feedback, override rates, and staff corrections
Each approach answers a different question. Drift tells you something may be changing. Review tells you what is actually going wrong. Validation gives you confidence that a new model is ready before it reaches users.
The goal isn't to monitor everything. It's to make sure what you detect leads to action. That's what separates reporting from continuous evaluation.
Part 3: Who should own continuous evaluation? Why it often falls between organizational silos, and why solving that ownership problem is becoming a competitive advantage.
AI Solutioning Perspectives