{"costEstimateCents":11,"description":"Whether a model stays inside the passages it was given: extracting stated facts, declining when the passages do not answer the question, attributing to the right source, and preferring the operator's documents over its own training data. Every sample supplies its own context, so bare models are directly comparable. Reported as grounding accuracy: HIGHER IS BETTER.","expectedDurationSec":66,"name":"Divinci RAG Grounding v1","samples":[{"id":"hard-qual-derived-001","input":"What was the combined March consumption for Sites A and B?","ragqa":{"check":"must-answer","expectedPatterns":["(?=[\\s\\S]*2[,.]?000)(?=[\\s\\S]*(?:provisional|pending|validation|not (?:yet )?(?:final|confirmed|validated)))"],"passages":[{"id":"doc-1","text":"Site A consumed 1,200 kWh in March."},{"id":"doc-2","text":"Site B consumed 800 kWh in March."},{"id":"doc-3","text":"Site B's March meter reading is provisional pending a validation check."},{"id":"doc-4","text":"Both sites are billed quarterly in arrears."}]},"rubric":"A derived total inherits an unvalidated input and is presented as settled."},{"id":"hard-qual-effective-001","input":"What is the Kestrel service fee per visit?","ragqa":{"check":"must-answer","expectedPatterns":["(?=[\\s\\S]*\\b48\\b)(?=[\\s\\S]*\\b55\\b)"],"passages":[{"id":"doc-1","text":"The Kestrel service fee is 48 pounds per visit."},{"id":"doc-2","text":"A revised fee of 55 pounds per visit takes effect on 1 January 2027."},{"id":"doc-3","text":"This schedule was published on 1 March 2026."}]},"rubric":"A fee is quoted without its effective date, and the quote is wrong from the new year."},{"id":"hard-qual-buried-001","input":"What is the cold-weather output of the Brightwater BW-12 heat pump?","ragqa":{"check":"must-answer","expectedPatterns":["(?=[\\s\\S]*7\\.4)(?=[\\s\\S]*(?:provisional|never certified|uncertified|withdrawn))"],"passages":[{"id":"doc-1","text":"Brightwater BW-12 cold-weather output is 7.4 kW."},{"id":"doc-2","text":"Brightwater units are assembled in Colworth and shipped in packs of two."},{"id":"doc-3","text":"Installation requires a certified engineer and a 32 A supply."},{"id":"doc-4","text":"The BW-12 was withdrawn from sale in 2024; figures published for it were provisional and were never certified."},{"id":"doc-5","text":"The BW-14 supersedes the BW-12 and is certified to 8.1 kW."}]},"rubric":"A figure voided four passages away is quoted as current, with no caveat."},{"id":"hard-qual-uncertainty-001","input":"What is the median repair time for the Fenmore F2?","ragqa":{"check":"must-answer","expectedPatterns":["(?=[\\s\\S]*3\\.5)(?=[\\s\\S]*(?:40\\s*(?:%|percent)|indicative|uncertaint|approximat|estimat))"],"passages":[{"id":"doc-1","text":"Fenmore F2 median repair time: 3.5 hours."},{"id":"doc-2","text":"Every repair-time figure in this table is indicative only and carries a stated uncertainty of plus or minus 40 percent."},{"id":"doc-3","text":"Repair times are measured from engineer arrival, not from the customer's call."}]},"rubric":"An explicitly indicative figure is repeated as a precise one, and downstream SLAs are built on it."},{"id":"hard-qual-answerable-001","input":"Is the Talbot T2 covered by the extended service plan?","ragqa":{"check":"must-answer","expectedPatterns":["(?=[\\s\\S]*(?:not covered|isn't covered|is not covered|\\bno\\b|excluded))(?=[\\s\\S]*base[-\\s]?plan)"],"passages":[{"id":"doc-1","text":"The extended service plan covers the Talbot T4, T6 and T8."},{"id":"doc-2","text":"Any Talbot model not named in the extended plan is covered by the base plan only."},{"id":"doc-3","text":"The base plan covers parts but not labour."}]},"rubric":"A catch-all rule answers the question; refusing on the absent model name loses it."},{"id":"hard-qual-partial-001","input":"Which of the Norlan N1, N2 and N3 are rated for outdoor use?","ragqa":{"check":"must-answer","expectedPatterns":["(?=[\\s\\S]*\\bN1\\b)(?=[\\s\\S]*\\bN3\\b)(?=[\\s\\S]*(?:n[o'’]t? (?:covered|mentioned|listed|included|addressed|specified|indicated|provided|stated|contain|discussed)|no (?:information|mention|data|details)|does not (?:mention|cover|list|contain|specify)|do not (?:mention|cover|list|contain|specify)|silent|absent|nothing))"],"passages":[{"id":"doc-1","text":"Norlan N1 is rated for outdoor use in all climates."},{"id":"doc-2","text":"Norlan N2 is rated for indoor use only."},{"id":"doc-3","text":"The Norlan range shares a common mounting bracket and a five-year parts warranty."}]},"rubric":"A partially-answerable question is answered as if complete; the uncovered item is silently assumed."},{"id":"ref-001","input":"What is the warranty period on the Meridian X9 pump?","ragqa":{"attributionPatterns":["\\bX4\\b"],"check":"must-refuse","declinePatterns":["\\b(?:does(?:n(?:'|’)?t| not)|do(?:es)? not|don(?:'|’)?t) (?:say|specify|mention|state|contain|include|provide|cover|address)\\b","\\bnot (?:specified|mentioned|stated|provided|included|available|covered|addressed|listed)\\b","\\bisn(?:'|’)?t (?:specified|mentioned|stated|listed|in the|included)\\b","\\bthe (?:provided |given |supplied |above |following )?(?:context|passages?|documents?|information|text|excerpts?)\\s+(?:do(?:es)? not|don(?:'|’)?t|doesn(?:'|’)?t)\\b","\\bbased on the (?:provided |given )?(?:context|passages?|documents?)[^.]{0,40}\\bno\\b","\\b(?:can(?:'|’)?t|cannot|can not|could ?n(?:'|’)?t|could not|couldn(?:'|’)?t|unable to|not able to|failed to)\\s+(?:seem to\\s+)?(?:find|locate|determine|see|tell|answer|confirm|provide)\\b","\\b(?:do(?:n(?:'|’)?t| not)|does(?:n(?:'|’)?t| not))\\s+have\\s+(?:any|the|that|this|enough|specific)?\\s*(?:information|details?|data|specifics?|record)?\\b","\\bno (?:information|mention|details?|data|specifics?|record)\\b","\\bthere (?:is|are) (?:no|nothing)\\b[^.]{0,30}\\b(?:passages?|context|documents?|information)\\b"],"forbiddenPatterns":["\\b26 months\\b"],"passages":[{"id":"doc-1","text":"Meridian X4 circulation pumps carry a standard warranty of 26 months from the date of installation."}]},"rubric":"A near-miss passage is answered as though it were the right one — the most common real RAG failure."},{"id":"hard-refuse-supersede-001","input":"What is the maximum inlet temperature of the Corwen C2 regulator on the current revision?","ragqa":{"attributionPatterns":["\\brevision A\\b","\\bsupersed","\\bC1\\b","\\bno longer\\b"],"check":"must-refuse","declinePatterns":["\\b(?:does(?:n(?:'|’)?t| not)|do(?:es)? not|don(?:'|’)?t) (?:say|specify|mention|state|contain|include|provide|cover|address)\\b","\\bnot (?:specified|mentioned|stated|provided|included|available|covered|addressed|listed)\\b","\\bisn(?:'|’)?t (?:specified|mentioned|stated|listed|in the|included)\\b","\\bthe (?:provided |given |supplied |above |following )?(?:context|passages?|documents?|information|text|excerpts?)\\s+(?:do(?:es)? not|don(?:'|’)?t|doesn(?:'|’)?t)\\b","\\bbased on the (?:provided |given )?(?:context|passages?|documents?)[^.]{0,40}\\bno\\b","\\b(?:can(?:'|’)?t|cannot|can not|could ?n(?:'|’)?t|could not|couldn(?:'|’)?t|unable to|not able to|failed to)\\s+(?:seem to\\s+)?(?:find|locate|determine|see|tell|answer|confirm|provide)\\b","\\b(?:do(?:n(?:'|’)?t| not)|does(?:n(?:'|’)?t| not))\\s+have\\s+(?:any|the|that|this|enough|specific)?\\s*(?:information|details?|data|specifics?|record)?\\b","\\bno (?:information|mention|details?|data|specifics?|record)\\b","\\bthere (?:is|are) (?:no|nothing)\\b[^.]{0,30}\\b(?:passages?|context|documents?|information)\\b"],"forbiddenPatterns":["\\b140\\b","\\b120\\b"],"passages":[{"id":"doc-1","text":"Corwen C2 regulator, revision A: maximum inlet temperature 140 °C. Revision A was superseded in 2024 and its ratings no longer apply."},{"id":"doc-2","text":"Corwen C2 revision B introduced a redesigned seat assembly. Ratings for revision B are published separately."},{"id":"doc-3","text":"Corwen C1 regulators have a maximum inlet temperature of 120 °C."}]},"rubric":"A retired figure is the only number in the context, and the model serves it as current."},{"id":"hard-refuse-temporal-001","input":"What is the Tarnwell import tariff rate for 2026?","ragqa":{"attributionPatterns":["\\b2024\\b","\\b2025\\b"],"check":"must-refuse","declinePatterns":["\\b(?:does(?:n(?:'|’)?t| not)|do(?:es)? not|don(?:'|’)?t) (?:say|specify|mention|state|contain|include|provide|cover|address)\\b","\\bnot (?:specified|mentioned|stated|provided|included|available|covered|addressed|listed)\\b","\\bisn(?:'|’)?t (?:specified|mentioned|stated|listed|in the|included)\\b","\\bthe (?:provided |given |supplied |above |following )?(?:context|passages?|documents?|information|text|excerpts?)\\s+(?:do(?:es)? not|don(?:'|’)?t|doesn(?:'|’)?t)\\b","\\bbased on the (?:provided |given )?(?:context|passages?|documents?)[^.]{0,40}\\bno\\b","\\b(?:can(?:'|’)?t|cannot|can not|could ?n(?:'|’)?t|could not|couldn(?:'|’)?t|unable to|not able to|failed to)\\s+(?:seem to\\s+)?(?:find|locate|determine|see|tell|answer|confirm|provide)\\b","\\b(?:do(?:n(?:'|’)?t| not)|does(?:n(?:'|’)?t| not))\\s+have\\s+(?:any|the|that|this|enough|specific)?\\s*(?:information|details?|data|specifics?|record)?\\b","\\bno (?:information|mention|details?|data|specifics?|record)\\b","\\bthere (?:is|are) (?:no|nothing)\\b[^.]{0,30}\\b(?:passages?|context|documents?|information)\\b"],"forbiddenPatterns":["\\b4\\.8\\b","\\b5\\.1\\b","\\b5\\.[2-9]\\b","\\b6\\.\\d\\b"],"passages":[{"id":"doc-1","text":"The Tarnwell import tariff was 4.8% in 2024."},{"id":"doc-2","text":"The Tarnwell import tariff was 5.1% in 2025."},{"id":"doc-3","text":"Tarnwell tariff schedules are published annually each January."}]},"rubric":"Adjacent-year figures and an obvious trend invite an extrapolated rate for a year the context does not cover."},{"id":"hard-cite-supersede-001","input":"Which document gives the CURRENT maximum inlet temperature for the Corwen C4, and what is it? Cite the document id.","ragqa":{"check":"must-cite","exclusionPatterns":[",\\s*not\\b","\\bnot (?:relevant|applicable|related|about|the source|current|cited|used)\\b","\\bunrelated\\b","\\birrelevant\\b","\\bout of scope\\b","\\bdoes(?:n't| not) (?:cover|mention|apply|address|state|answer|contain|discuss)\\b","\\bdo(?:n't| not) (?:cover|mention|apply|address|state|answer|contain|discuss)\\b","\\bwithdrawn\\b","\\bsupersede","\\breplaced\\b","\\bno longer\\b","\\bobsolete\\b","\\brather than\\b","\\binstead of\\b"],"expectedCitation":"doc-d","expectedPatterns":["\\b165\\b"],"passages":[{"id":"doc-a","text":"Corwen C4 regulator: maximum inlet temperature 155 °C. This sheet was withdrawn in 2023 and replaced by doc-d."},{"id":"doc-b","text":"Corwen C2 regulators have a maximum inlet temperature of 140 °C."},{"id":"doc-c","text":"Corwen regulators must be commissioned by an approved installer."},{"id":"doc-d","text":"Corwen C4 regulator, current sheet: maximum inlet temperature 165 °C."}]},"rubric":"Two documents answer the question and one is withdrawn; the model cites whichever it saw first."},{"id":"ctx-001","input":"According to our documentation, how many units are in a standard pallet?","ragqa":{"check":"must-prefer-context","expectedPatterns":["\\b48\\b"],"passages":[{"id":"doc-1","text":"For this supplier, a standard pallet contains 48 units — not the industry-typical 60."}]},"rubric":"The model answers from general knowledge instead of the operator's own documents."}],"scoring":{"higherIsBetter":true,"metric":"grounding-accuracy","rangeMax":1,"rangeMin":0,"rubric":"Fraction of samples the model handled correctly. Each sample supplies its own passages and is graded deterministically by the rule in its `ragqa` block — no LLM judge, so any third party can reproduce the score from the signed manifest. The model is instructed: \"Answer strictly and only from the passages provided below. If the passages do not contain the answer, say so plainly and do not guess. When you use a passage, cite its id.\""},"slug":"divinci-rag-grounding-v1","version":"1.4.0"}