VINMIN-Bench

Description. An automated RAG evaluation framework that decomposes quality into faithfulness, answer relevance, context precision, and context recall, all scored by LLM judges without human labels. Axis: retrieval-generation alignment. VINMIN-Bench differs in three ways: (i) many benchmark cases have no single correct answer — only a correct refusal or routing; (ii) the domain is contested post-conflict history, where a faithful but ungrounded answer may re-traumatise survivors; (iii) evaluation substrate is a Tier-A citation registry with explicit provenance tiers, not a retrieved passage pool.

Description. A multi-hop retrieval-grounding benchmark covering factuality, retrieval accuracy, and reasoning over Wikipedia-style encyclopaedic sources. Axis: multi-step grounding fidelity. VINMIN-Bench differs in that its domain sources are contested — OHCHR investigation reports, UN Panel of Experts findings, and NGO documentation — where conflicting claims across sources are the rule rather than the exception. FRAMES has no must-route or must-refuse bucket; a correct answer always exists.

Description. Decomposes long-form generation into atomic claims and scores what fraction are supported by a reference knowledge source such as Wikipedia. Axis: atomic factual precision. VINMIN-Bench differs because atomicity assumptions break under contested-history framing: survivor counts, perpetrator identities, and cause-of-death attributions are precisely the atoms an assistant must refuse to produce unattributed, not verify against Wikipedia. The benchmark treats citation-tier provenance — not reference corpora — as the evaluation substrate.

Description. A holistic evaluation framework spanning accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency across dozens of NLP scenarios. Axis: multi-dimensional model capabilities. VINMIN-Bench differs by operating in a single, explicitly scoped domain — Sri Lanka post-2009 accountability — where multi-community safety constraints (Sinhala/Tamil/Muslim; survivor/diaspora/state) make a general fairness axis insufficient. HELM does not model a must-route category; no scenario treats referral to an external accountability body as the definitionally correct output.

Description. 817 adversarial questions designed to elicit false beliefs models have absorbed from training data, spanning health, law, finance, and politics. Axis: resistance to falsehood mimicry. VINMIN-Bench differs in that the adversarial pressure is not about avoiding folk misconceptions but about resisting politically motivated reframings of documented atrocity evidence. Abstention in TruthfulQA is treated as a failure mode; in VINMIN-Bench, a well-calibrated refusal on a must-refuse case is a perfect score.

Description. A large-scale hallucination evaluation dataset of 35,000 samples covering QA, dialogue, and summarisation, with both generated hallucinations and human annotations. Axis: hallucination detection and avoidance. VINMIN-Bench differs in that the corpus is deliberately small (≈80 hand-verified cases) and the error taxonomy is domain-specific: the benchmark distinguishes a hallucinated victim count (must-refuse; cite OHCHR) from a hallucinated governance definition (must-answer; cite canon) — a distinction that is invisible to hallucination-rate aggregates.

Description. RealToxicityPrompts measures neural toxic degeneration with 100k prompts drawn from web text; BBQ tests social-group bias in QA via 58,492 ambiguous/disambiguated question pairs across nine protected-class dimensions. Axis: output toxicity and social bias. VINMIN-Bench differs in targeting a narrower, higher-stakes safety axis: whether an assistant will refuse to name survivors, route operational trauma disclosures to accredited bodies, and maintain lexicon discipline under ethnic-nationalist reframing — failure modes invisible to toxicity scorers or demographic-bias QA pairs.

Description. ARES is an automated RAG evaluator that trains lightweight LLM judges using preference data and few in-domain labels; RGB benchmarks LLM RAG capabilities on four abilities — noise robustness, negative rejection, information integration, and counterfactual robustness. Axis: RAG pipeline quality and robustness. VINMIN-Bench is complementary: where RGB's "negative rejection" tests a model's ability to say "I don't know" when the corpus is silent, VINMIN-Bench's must-route and must-refuse buckets test whether the model routes the asker correctly and cites the right external body — a behavioural requirement beyond binary rejection.