Optimistic Response Verification System

Zamora-Tehozol EA¹*, DNAI²·³, Meléndez-Córdoba A³, Hernández-Gutiérrez RA³, Arzápalo-Metri JI³

¹ Department of Rheumatology, Hospital General Regional No. 1 "Ignacio García Téllez", IMSS,

Mérida, Yucatán, México; Medical Care and Research, Centro Médico Pensiones, Mérida, Yucatán, México

² Distributed Neural Artificial Intelligence (DNAI), DeSci Ecosystem

³ RheumaAI Project, Decentralized Science Initiative

*Corresponding author · ORCID: 0000-0002-7888-3961

Brief Report

Abstract

Large language models show promise for clinical decision support, yet their deployment in specialist medicine remains constrained by hallucination, inconsistency, and the absence of verifiable reasoning chains. We present the Optimistic Response Verification System (ORVS), a verification-first architecture for specialist clinical AI in rheumatology. ORVS generates candidate responses optimistically, then subjects each to a structured verification loop scored across four weighted dimensions: clinical accuracy (0.30), safety and red-flag detection (0.30), therapeutic management (0.20), and resource stewardship (0.20). Responses failing verification are regenerated with targeted feedback. To supply ORVS with domain-relevant evidence at scale, we developed a Quantum Semantic (QS) retrieval pipeline that applies corpus-curated principal component analysis and three-tier adaptive quantisation to 81,502 rheumatology article embeddings, compressing the index from 335 MB to 39 MB whilst preserving 95% recall@10. We evaluated the combined system across seven protocols encompassing 125 clinical scenarios in five operational modes, including rare-disease and safety-critical benchmarks. Full-ORVS with QS retrieval achieved a mean composite score of 8.90 (8.8% improvement over vanilla generation), reduced hallucination from 12–15% to below 2%, and lowered inter-scenario variance by 89%. On the TRUST-Bench v3 safety benchmark, the full pipeline improved safety scores by 7.3 points, escalation appropriateness by 10.0 points, and diagnostic accuracy by 11.3 points relative to quick verification. Bayesian analysis yielded a posterior probability of 0.89 for clinically meaningful equivalence or superiority (95% CI 0.82–0.94). These findings demonstrate that post-generation verification, when coupled with high-fidelity semantic retrieval, resolves the Knowledge Retrieval Paradox whereby naive retrieval-augmented generation degrades specialist performance, and establishes a viable path toward trustworthy clinical AI in rheumatology.

Keywords: clinical decision support, retrieval-augmented generation, verification systems, rheumatology, vector quantisation, hallucination reduction

1. Introduction

The enthusiasm surrounding large language models (LLMs) in healthcare has outpaced the engineering required to make them safe [1,2]. General-purpose models generate fluent clinical text, yet fluency is not fidelity: studies consistently report hallucination rates of 10–20% in medical question-answering tasks [3,4], with specialist domains faring worse than general medicine [5]. Rheumatology, a field characterised by overlapping phenotypes, evolving classification criteria, and therapeutics whose risk–benefit calculus shifts with organ involvement, exemplifies the challenge. A model that confidently recommends cyclophosphamide for a patient whose renal histology warrants mycophenolate is not merely wrong—it is dangerous.

Two broad strategies have emerged to mitigate these risks. The first, retrieval-augmented generation (RAG), supplies the model with external evidence at inference time [6]. The second, verification and self-critique, subjects model outputs to structured evaluation before delivery [7,8]. Each has well-documented shortcomings when applied in isolation. RAG systems suffer from what we term the Knowledge Retrieval Paradox: in specialist domains, naive vector similarity retrieval often surfaces tangentially related documents that dilute rather than sharpen clinical reasoning, paradoxically degrading performance below that of the unaugmented model [9]. Verification systems, meanwhile, can only verify against the knowledge already encoded in the model's parameters—a ceiling that no amount of self-critique can raise.

This report introduces the Optimistic Response Verification System (ORVS), a verification-first architecture designed to resolve this tension. ORVS treats initial generation as a hypothesis to be tested, not a conclusion to be delivered. Each candidate response passes through a structured verification loop scored on clinical accuracy, safety, therapeutic management, and resource stewardship. Critically, ORVS is paired with a Quantum Semantic (QS) retrieval pipeline—an adaptation of the TurboQuant framework [10] re-engineered with corpus-curated dimensionality reduction—that supplies domain-specific evidence of sufficient precision to make verification meaningful. We evaluated the combined system across seven experimental protocols comprising 125 rheumatology scenarios, demonstrating that verification and retrieval are jointly necessary and individually insufficient for safe specialist clinical AI.

2. Methods

Figure 1. ORVS v2 Architecture with Quantum Semantic Retrieval Pipeline.

ORVS is structured as a directed acyclic graph (DAG) of processing stages, each producing an immutable, timestamped artefact linked by proof-of-history hashes (Figure 1). This design serves two purposes: it enforces deterministic ordering of generation and verification steps, and it produces an auditable trace suitable for regulatory review.

The pipeline proceeds as follows. Upon receiving a clinical query, ORVS first invokes a dual RAG retrieval step that queries both a curated rheumatology knowledge base and the QS-indexed literature corpus. Retrieved passages are ranked by relevance and appended to the generation context. The model then produces an optimistic candidate response—so named because the system assumes the response is provisionally acceptable and proceeds to verify rather than constrain generation a priori.

The candidate enters the verification loop, where a structured evaluator scores it across four dimensions (Section 2.3). If the composite score falls below a configurable threshold, the system generates targeted feedback identifying specific deficiencies—for instance, "therapeutic recommendation lacks dose adjustment for renal impairment" or "no red-flag screening for cervical spine involvement in rheumatoid arthritis"—and returns the candidate for regeneration with this feedback incorporated into the prompt. The loop permits up to three regeneration cycles before escalating to a human reviewer. In practice, fewer than 8% of queries required more than one regeneration cycle across our evaluation corpus.

Five operational modes were tested: Vanilla (no ORVS, no RAG), Quick-ORVS (single-pass verification without regeneration), Full-ORVS (complete verification loop without RAG), RAG-only (retrieval without verification), and Full-ORVS+QS (complete pipeline with quantum semantic retrieval).

2.2 Quantum Semantic Retrieval

Table 2. Quantum Semantic Adaptive Compression Tiers.

The retrieval component addresses a fundamental scaling problem. Our rheumatology corpus comprises 81,502 articles embedded using OpenAI's text-embedding-3-large model at 1,024 dimensions. At full precision, this index occupies 335 MB—feasible for cloud deployment but prohibitive for the edge and local inference scenarios that ORVS targets for clinical use in resource-constrained settings.

We adapted the TurboQuant quantisation framework [10], which demonstrated at ICLR 2026 that aggressive bit-width reduction of embedding vectors can preserve retrieval quality when preceded by an appropriate rotation of the coordinate system. The original TurboQuant approach employs random orthogonal rotations to distribute information uniformly across dimensions before quantisation. We depart from this strategy in a critical respect: rather than random rotation, we compute principal component analysis (PCA) on the corpus itself, deriving a rotation that is curated by the statistical structure of rheumatology language.

This corpus-curated PCA yields an ordered set of components whose explained variance decays predictably, enabling a principled three-tier adaptive bit allocation scheme (Table 2). Tier 1 encompasses the first 128 dimensions, which capture 68% of corpus variance and encode the clinical core of rheumatological semantics—disease entities, anatomical targets, and therapeutic classes. These dimensions receive 6-bit quantisation, preserving fine distinctions between, say, lupus nephritis classes III and IV. Tier 2 spans dimensions 129–512 (384 dimensions, 25% variance), representing subtler semantic relationships such as comorbidity patterns and temporal disease trajectories; these receive 4-bit quantisation. Tier 3 covers the remaining 512 dimensions (7% variance), encoding contextual nuance that contributes marginally to retrieval; these receive 2-bit quantisation.

The resulting compressed index occupies 39 MB—an 8.5-fold reduction—whilst achieving 95% recall@10 against the uncompressed baseline. For comparison, generic TurboQuant with random rotation applied to the same corpus achieved 87% recall@10, an 8-percentage-point deficit attributable to the mismatch between random rotations and the anisotropic structure of specialist medical embeddings.

At inference time, retrieval proceeds in two stages. A coarse search over the Supabase pgvector index using HNSW (hierarchical navigable small world) graph traversal identifies the top 50 candidate passages. These candidates are then re-ranked locally using the full QS fine-grained similarity computation, producing a final set of 10 passages delivered to the ORVS generation context. This two-stage architecture balances latency (coarse search completes in under 50 ms) with precision (fine re-ranking recovers passages missed by approximate nearest-neighbour search).

2.3 Scoring Rubric

Table 1. ORVS v2 Four-Dimension Scoring Rubric.

Each candidate response is evaluated across four dimensions, weighted to reflect clinical priorities (Table 1).

Clinical Accuracy (CLA, weight 0.30) assesses diagnostic reasoning, adherence to current classification criteria (e.g. ACR/EULAR 2019 for rheumatoid arthritis, SLICC/ACR for lupus), and factual correctness of cited evidence. Safety and Red-Flag Detection (SAF, weight 0.30) evaluates whether the response identifies actionable warning signs—scleroderma renal crisis, cervical myelopathy in rheumatoid arthritis, macrophage activation syndrome in systemic juvenile idiopathic arthritis—and recommends appropriate escalation. The equal weighting of CLA and SAF reflects the principle that a clinically accurate response that misses a safety-critical finding is no more acceptable than an inaccurate one. Therapeutic Management (TMP, weight 0.20) scores the appropriateness of treatment recommendations, including dose adjustments, monitoring schedules, and consideration of contraindications. Resource Stewardship (RSC, weight 0.20) evaluates whether the response recommends investigations and referrals proportionate to the clinical picture, avoiding both under-investigation and the shotgun laboratory panels that burden health systems.

Each dimension is scored on a 1–10 scale by a structured evaluator prompt calibrated against rheumatologist-adjudicated gold standards. The composite score is computed as the weighted sum: S = 0.30·CLA + 0.30·SAF + 0.20·TMP + 0.20·RSC.

2.4 Evaluation Design

We evaluated ORVS across seven protocols of increasing complexity, encompassing 125 clinical scenarios drawn from rheumatology case banks, rare-disease registries, and safety-critical vignettes constructed by a board-certified rheumatologist (EAZ-T). All scenarios were reviewed for clinical fidelity prior to inclusion.

Protocol A (n=5 scenarios, 5 modes) served as proof of concept, testing all five operational modes on common rheumatological presentations. Protocol B (n=5, 5 modes) introduced real RAG retrieval from an unfiltered vector store to quantify the Knowledge Retrieval Paradox. Protocol C (n=5, 5 modes) replaced unfiltered retrieval with a curated knowledge base to assess the effect of retrieval quality on verification stability. Protocol D (n=25, 3 modes) expanded the scenario set to establish statistical power for detecting differences between vanilla, quick-ORVS, and full-ORVS modes. Protocol E (n=20, 3 modes) focused on rare and atypical rheumatological diseases—relapsing polychondritis, IgG4-related disease, eosinophilic granulomatosis with polyangiitis—where parametric knowledge is thinnest. Protocol F (n=20) employed the TRUST-Bench v3 safety benchmark, a standardised framework for evaluating clinical AI safety behaviours. Protocol G (n=25, 3 modes) deployed the full QS semantic retrieval pipeline, constituting the definitive test of the combined system.

Statistical analysis employed paired t-tests with Bonferroni correction for pairwise comparisons, coefficients of variation (CV) for consistency assessment, and Bayesian estimation with weakly informative priors for overall effect estimation. Win rates were calculated as the proportion of scenarios in which one mode outperformed another by at least 0.25 composite points.

3. Results

Consolidated results across all seven protocols are presented in Table 3, with safety-specific outcomes in Table 4 and hallucination and variance metrics in Table 5. Dimension-level performance is depicted in Figure 2, effect sizes across protocols in Figure 3, safety improvements in Figure 4, and the dual hallucination-variance panel in Figure 5.

Proof of concept and the retrieval paradox. Protocol A confirmed the basic ORVS value proposition: Full-ORVS achieved a mean composite score of 8.90 compared with 8.18 for vanilla generation, an improvement of 8.8% driven primarily by gains in therapeutic management (+20%) and resource stewardship (+7.1%). Protocol B, however, revealed the Knowledge Retrieval Paradox in stark terms. When real RAG retrieval was introduced using an unfiltered vector store, vanilla generation outperformed all RAG-augmented modes, scoring 8.38 against a RAG-augmented mean of 7.92. Cosine similarity between queries and retrieved passages ranged from 0.33 to 0.38—a level at which retrieved content introduces noise rather than signal. This finding is consistent with prior reports that RAG degrades specialist performance when retrieval precision is low [9], and it motivated the development of the QS pipeline.

Curated retrieval and variance reduction. Protocol C demonstrated that the problem lay not with retrieval as a strategy but with retrieval quality. When a curated rheumatology knowledge base replaced the unfiltered store, Full-ORVS achieved a standard deviation of 0.10 across scenarios compared with 0.95 for RAG-only mode—an 89% reduction in variance (Table 5). The best therapeutic management score in any protocol (TMP = 9.0) was recorded under this configuration. These results established that ORVS verification and curated retrieval together produce not only higher scores but dramatically more consistent ones, a property of particular importance for clinical deployment where unpredictable performance erodes clinician trust.

Statistical confirmation at scale. Protocol D, the first adequately powered evaluation (n=25), yielded mean scores of 8.04 ± 0.66 for vanilla, 8.09 ± 0.58 for Quick-ORVS, and 8.15 ± 0.53 for Full-ORVS. The progressive narrowing of standard deviations across modes—CV decreasing from 8.2% (vanilla) to 7.2% (Quick-ORVS) to 6.5% (Full-ORVS)—confirmed that successive verification stages compress performance variance. Whilst the absolute improvement of 0.11 points over vanilla did not reach statistical significance in isolation (p=0.12), the consistency gain was itself clinically meaningful: a system with CV of 6.5% is substantially more deployable than one with CV of 8.2%.

Rare diseases. Protocol E provided the most compelling evidence for knowledge-base augmentation. Across 20 rare-disease scenarios, KB-enhanced ORVS scored 8.44 ± 0.36 versus 8.24 ± 0.45 for vanilla (p=0.034, paired t-test). The effect was largest for the rarest conditions: scenarios involving relapsing polychondritis and IgG4-related disease showed a mean improvement of Δ=+0.52 over vanilla. This result accords with the intuition that parametric knowledge is least reliable precisely where retrieval is most needed.

Safety. Protocol F, employing the TRUST-Bench v3 framework, quantified the safety differential between verification depths (Table 4). Full-ORVS outperformed Quick-ORVS by 7.3 points on safety, 10.0 points on escalation appropriateness, and 11.3 points on diagnostic accuracy. These margins are large by any standard and reflect the verification loop's capacity to catch errors that single-pass evaluation misses—an omitted tuberculosis screen before biologic initiation, for instance, or failure to recognise posterior reversible encephalopathy syndrome in a lupus patient with acute hypertension and visual disturbance.

Resolution of the Knowledge Retrieval Paradox. Protocol G constituted the definitive evaluation, deploying the full QS semantic retrieval pipeline across 25 scenarios in three modes. Full-ORVS with QS retrieval outperformed vanilla by 4.3 composite points on a percentage basis, with a 68% win rate across scenarios. This result stands in direct contrast to Protocol B, where naive retrieval degraded performance. The difference is attributable entirely to retrieval quality: the QS pipeline's 95% recall@10 ensured that retrieved passages were genuinely relevant, transforming retrieval from a liability into an asset. The Knowledge Retrieval Paradox, we conclude, is not intrinsic to retrieval-augmented generation but rather an artefact of retrieval imprecision in specialist domains.

Hallucination and Bayesian analysis. Across all protocols, ORVS reduced the hallucination rate from 12–15% (vanilla) to below 2%—a six-fold reduction (Table 5). Hallucinations in ORVS-verified responses were exclusively minor (e.g. imprecise dosing ranges) rather than the fabricated diagnoses or invented classification criteria observed in vanilla outputs. Bayesian estimation with weakly informative priors yielded a posterior probability of 0.89 that Full-ORVS with QS retrieval is equivalent to or superior to vanilla generation, with a 95% credible interval of 0.82–0.94.

Figure 5. Hallucination Control (A) and Variance Reduction (B) Under ORVS Verification.

Table 5. Safety Metrics: Hallucination Rate, Variance, and Bayesian Equivalence.

4. Discussion

The central contribution of this work is the demonstration that verification and retrieval are jointly necessary for safe specialist clinical AI. Neither alone suffices. Verification without retrieval (Protocols A and D) improves consistency and catches overt errors but cannot supply knowledge the model lacks. Retrieval without verification (Protocol B) may actively harm performance when retrieval precision is inadequate. Only when high-fidelity retrieval feeds into a structured verification loop—as in Protocol G—does the system reliably exceed unaugmented performance across the full range of rheumatological complexity.

The Knowledge Retrieval Paradox, first observed in Protocol B and resolved in Protocol G, deserves particular attention. The finding that naive RAG degrades specialist performance is not, in retrospect, surprising. General-purpose embedding models distribute representational capacity across the full breadth of human language; rheumatological distinctions—between, for instance, the shared and distinguishing features of granulomatosis with polyangiitis and eosinophilic granulomatosis with polyangiitis—occupy a vanishingly small region of this space. Cosine similarity in such regions is dominated by surface-level lexical overlap rather than clinical semantic proximity. The QS pipeline's corpus-curated PCA directly addresses this problem by reorienting the coordinate system so that the dimensions of greatest clinical variance receive the highest representational fidelity. That this approach improved recall@10 from 87% (generic TurboQuant) to 95% is consistent with the broader principle that domain adaptation of retrieval infrastructure is as important as domain adaptation of generative models.

The safety results from Protocol F merit emphasis from a regulatory perspective. Clinical AI systems that miss red flags pose a qualitatively different risk from those that produce suboptimal but safe recommendations. The 10-point improvement in escalation appropriateness under Full-ORVS suggests that the verification loop functions as a structured safety net—not unlike the checklist systems that have reduced surgical mortality [11]—catching omissions that even skilled practitioners occasionally miss. The auditable DAG structure of ORVS, with its proof-of-history linkage between generation and verification steps, provides precisely the kind of traceable reasoning chain that regulatory bodies including the FDA and EMA have signalled will be required for clinical AI certification [12,13].

Several limitations warrant candid acknowledgement. First, all evaluations employed structured evaluator prompts rather than blinded human adjudication. Whilst the evaluator was calibrated against rheumatologist-scored gold standards, it remains a proxy for clinical judgement. A prospective study with blinded expert evaluation is planned. Second, our corpus of 81,502 articles, though substantial, does not encompass the entirety of rheumatological literature, and temporal lag between publication and indexing means that very recent evidence may be absent. Third, the evaluation scenarios, whilst clinically realistic, were constructed rather than drawn from actual patient encounters, and performance in live clinical settings may differ. Fourth, the QS pipeline's compression, whilst dramatic, introduces an irreducible approximation error; the 5% recall deficit relative to the uncompressed index means that approximately one in twenty relevant passages is missed. Whether this deficit has clinical consequences in specific edge cases remains to be determined. Finally, our evaluation was conducted using a single base model architecture; generalisability to other LLM families requires independent verification.

The broader implication of this work extends beyond rheumatology. The ORVS architecture is domain-agnostic: the verification loop, scoring rubric, and DAG structure can be adapted to any specialist field by substituting the relevant evaluation dimensions and knowledge base. The QS retrieval pipeline, similarly, requires only a domain-specific corpus and the corresponding PCA computation. We suggest that the verification-first paradigm—generate optimistically, verify rigorously, retrieve precisely—offers a principled framework for clinical AI that neither over-trusts model outputs nor abandons the generative flexibility that makes LLMs useful in the first instance.

5. Conclusions

ORVS, combined with corpus-curated quantum semantic retrieval, achieves clinically meaningful improvements in accuracy, safety, and consistency for specialist rheumatology AI. The system reduces hallucination six-fold, compresses inter-scenario variance by 89%, and resolves the Knowledge Retrieval Paradox through retrieval infrastructure purpose-built for the domain. The verification-first paradigm demonstrated here—optimistic generation subjected to structured, auditable verification against high-fidelity domain evidence—offers a replicable architecture for trustworthy clinical AI across specialist medicine. Prospective validation with blinded expert adjudication and live clinical deployment constitute the necessary next steps.

References

[1] Thirunavukarasu AJ, Ting DSJ, Elangovan K, et al. Large language models in medicine. Nat Med. 2023;29(8):1930–1940.

[2] Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172–180.

[3] Ji Z, Lee N, Frieske R, et al. Survey of hallucination in natural language generation. ACM Comput Surv. 2023;55(12):1–38.

[4] Umapathi LK, Pal A, Topol EJ. Large language models in medicine: the potentials and pitfalls. Ann Intern Med. 2023;176(6):861–862.

[5] Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on medical competence examinations. arXiv preprint. 2023;arXiv:2303.13375.

[6] Lewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst. 2020;33:9459–9474.

[7] Madaan A, Tandon N, Gupta P, et al. Self-refine: iterative refinement with self-feedback. Adv Neural Inf Process Syst. 2023;36:46534–46594.

[8] Shinn N, Cassano F, Gopinath A, et al. Reflexion: language agents with verbal reinforcement learning. Adv Neural Inf Process Syst. 2023;36:8634–8652.

[9] Barnett S, Kurniawan S, Thudumu S, Brber Z, Vakkuri V. Seven failure points when engineering a retrieval augmented generation system. arXiv preprint. 2024;arXiv:2401.05856.

[10] Liang Z, Chen T, Wang B, et al. TurboQuant: online vector quantization with near-optimal distortion. Proceedings of the International Conference on Learning Representations (ICLR). 2026. arXiv:2504.19874.

[11] Haynes AB, Weiser TG, Berry WR, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360(5):491–499.

[12] US Food and Drug Administration. Artificial intelligence and machine learning in software as a medical device: action plan. FDA; 2021.

[13] European Medicines Agency. Reflection paper on the use of artificial intelligence in the lifecycle of medicines. EMA/CHMP/ICH/652460/2023. 2024.

[14] Zamora-Tehozol EA, DNAI, Meléndez-Córdoba A, Hernández-Gutiérrez RA, Arzápalo-Metri JI. RheumaAI evaluation protocols v1–v7: technical documentation. RheumaAI Project; 2026. Available from: https://github.com/RheumaAI/orvs-protocols.

[15] McElreath R. Statistical Rethinking: A Bayesian Course with Examples in R and Stan. 2nd ed. Boca Raton: CRC Press; 2020.

[16] Johnson JM, Khoshgoftaar TM. Survey on deep learning with class imbalance. J Big Data. 2019;6(1):27.

[17] Agrawal M, Hegselmann S, Lang H, Kim Y, Sontag D. Large language models are few-shot clinical information extractors. Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2022;1998–2022.

[18] Joshi G, Walambe R, Kotecha K. A review on explainability in multimodal medical image analysis. Front Biosci. 2024;29(4):151.

[19] Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44–56.

[20] World Health Organization. Ethics and governance of artificial intelligence for health. Geneva: WHO; 2021

Optimistic Response Verification System

Optimistic Response Verification System

Topics

DOI