RAG Evaluation Harness for Production Systems

Most enterprise RAG failures never happen during model generation. They happen silently in the retrieval layer: missing access controls that expose confidential documents, stale embeddings that return outdated policies, or fluent hallucinations that invent answers when retrieval returns zero relevant spans. Evaluating these systems requires more than running generic academic benchmarks like MMLU or generic semantic similarity scores.

At Vasiva, we approach enterprise RAG as an operational distributed system. Before an enterprise search or knowledge assistant reaches production, it must pass through an automated evaluation harness wired into continuous integration. This article outlines the exact architecture, metric definitions, and human-gate placements we use to prevent retrieval drift and permission breaches.

Core Metrics for Production Evaluation

Rather than tracking vanity metrics like generic satisfaction ratings, enterprise evaluation requires five concrete dimensions with clear failure thresholds:

Harness Architecture and Execution Pipeline

A production evaluation harness must be lightweight enough to run nightly or on pull requests modifying chunking strategies, embedding models, or vector databases. The architecture separates deterministic checks from probabilistic generation:

1. The Golden Dataset

A common mistake is creating thousands of synthetic LLM-generated test questions. Instead, start with 50 to 100 high-leverage queries derived from real user support tickets, compliance guidelines, and historic edge-case failures. Each test entry in the golden corpus must specify:

• Query string representing actual user phrasing and terminology.
• Expected Document IDs (or an empty array for negative test cases that require refusal).
• Caller Security Context: tenant identifier, user role, and document ACL clearance level.
• Required Output Shape: explicit citation requirement, direct answer, or security refusal.

2. Same-Path Production Execution

Evaluation harnesses that test an isolated Python script while production runs on a separate Go or Node retrieval microservice will inevitably mask deployment discrepancies. The harness must call the production retrieval interface directly, using the same chunking parameters, embedding endpoints, and hybrid search weights.

Scoring Layers: Deterministic First, Advisory Second

We divide evaluation scoring into three sequential layers to minimize cost and prevent non-deterministic test flakiness:

1. Layer 1: Deterministic Invariants. Tests checking whether unauthorized doc IDs were retrieved, whether expected doc IDs appear in top-k results, and whether latency exceeds thresholds. A single failure in this layer immediately fails the CI run.
2. Layer 2: Model-Assisted Advisory Heuristics. Running an auxiliary LLM prompt to score the answer clarity or nuance. This layer is strictly advisory — it emits telemetry warnings but does not possess merge-blocking authority.
3. Layer 3: Human Domain Review. A weekly sampled review of low-confidence answers and all queries touching sensitive legal or financial domains, conducted by engineers who own the system.

Human Gate Placement Across the Release Lifecycle

Automated test suites can evaluate document recall, but human engineering review is irreplaceable when systemic policies or corpus boundaries change. We map human sign-offs to specific release stages:

Implementation Blueprint for Engineering Teams

When implementing this harness in your own infrastructure, adhere to these operational principles:

• Store the golden test questions in the same version-controlled repository as the RAG service codebase.
• Run retrieve-only tests on every pull request that touches embeddings, vector databases, or chunking code.
• Schedule full retrieve-plus-generate tests on a nightly cadence to control token consumption costs.
• Treat permission-leak test failures with zero tolerance: any cross-tenant data leak is a P0 release blocker.
• Publish internal telemetry dashboards tracking retrieval latency percentiles and hit rate trends over time.

How This Integrates with Vasiva's Delivery

Across our sequential product engineering work (WorkHub, AimHabits, TalkBetter, and Dima) and enterprise client engagements, we treat RAG not as an experimental toy or a simple wrapper around an OpenAI endpoint, but as mission-critical enterprise software. We build deterministic verification pipelines into every system from day one.

Engineering Ground Truth: What We Refuse to Claim

In accordance with Vasiva's engineering standards, we state our limitations clearly:

• No fabricated client metrics or invented Fortune 500 case studies.
• No absolute accuracy percentages without a published evaluation dataset and reproducible methodology.
• No unsubstantiated claims of beating commercial search engines without fair, public benchmark scripts.