Benchmarks¶
Generated by scripts/gen_catalog_doc.py from the component and benchmark
registries. Do not edit by hand: run the script.
31 public benchmarks are registered. List them with lasagna benchmark list,
and print the license and citation of each with lasagna benchmark licenses.
Lasagna redistributes no benchmark data. Each dataset is fetched from its own
source on first use, under its own terms, and those terms are yours to honor,
including for anything you publish. The smoke benchmark is separate: 24
documents and 12 queries held inside the package, for checking that a component is
wired in. A score on it is never a result.
| Column | Meaning |
|---|---|
| Scores | What the benchmark measures: retrieval, generation, or both |
| Queries, Documents | Size as the adapter records it; blank when the adapter does not say |
| Live | yes if the benchmark ships a retrievable corpus, so a run can ingest it and search it. Without one, only generation against gold context can be scored |
| License | As the registry records it. Varies by dataset means check the upstream source |
| Benchmark | Task | Scores | Queries | Documents | License | Live | Source |
|---|---|---|---|---|---|---|---|
bamboogle |
Multi-hop QA | Generation | 125 | MIT | - | arXiv:2304.09848 | |
beir/arguana |
Single-hop QA | Retrieval | 1.4K | 8.7K | Varies by dataset | yes | |
beir/climate-fever |
Single-hop QA | Retrieval | 1.5K | 5.4M | Varies by dataset | yes | |
beir/dbpedia-entity |
Single-hop QA | Retrieval | 400 | 4.6M | Varies by dataset | yes | |
beir/fever |
Single-hop QA | Retrieval | 6.7K | 5.4M | Varies by dataset | yes | |
beir/fiqa |
Single-hop QA | Retrieval | 6.6K | 58K | Varies by dataset | yes | |
beir/hotpotqa |
Single-hop QA | Retrieval | 7.4K | 5.2M | Varies by dataset | yes | |
beir/msmarco |
Single-hop QA | Retrieval | 6.9K | 8.8M | Varies by dataset | yes | |
beir/nfcorpus |
Single-hop QA | Retrieval | 3.2K | 3.6K | Varies by dataset | yes | arXiv:1610.02806 |
beir/nq |
Single-hop QA | Retrieval | 3.5K | 2.7M | Varies by dataset | yes | |
beir/quora |
Single-hop QA | Retrieval | 10K | 523K | Varies by dataset | yes | |
beir/scidocs |
Single-hop QA | Retrieval | 1K | 25K | Varies by dataset | yes | |
beir/scifact |
Single-hop QA | Retrieval | 1.1K | 5.2K | Varies by dataset | yes | arXiv:2004.14974 |
beir/trec-covid |
Single-hop QA | Retrieval | 50 | 171K | Varies by dataset | yes | |
fever |
Fact verification | Generation | 185K | CC BY-SA 3.0 | - | arXiv:1803.05355 | |
hotpotqa |
Multi-hop QA | Retrieval and generation | 113K | CC BY-SA 4.0 | yes | arXiv:1809.09600 | |
kilt/nq |
Single-hop QA | Generation | MIT | - | arXiv:2009.02252 | ||
kilt/triviaqa |
Single-hop QA | Generation | MIT | - | arXiv:2009.02252 | ||
ms_marco |
Passage ranking | Generation | 809K | 8.8M | MIT | - | arXiv:1611.09268 |
multihop_rag |
Multi-hop QA | Retrieval and generation | 2.6K | MIT | - | arXiv:2401.15391 | |
musique |
Multi-hop QA | Generation | 25K | CC BY 4.0 | yes | arXiv:2108.00573 | |
natural_questions |
Single-hop QA | Generation | 7.8K | Apache 2.0 | - | arXiv:1901.08634 | |
popqa |
Single-hop QA | Generation | 14K | CC BY-SA 4.0 | - | arXiv:2212.10511 | |
ragbench/hotpotqa |
Single-hop QA | Retrieval and generation | Apache 2.0 | yes | arXiv:2407.11005 | ||
ragbench/msmarco |
Single-hop QA | Retrieval and generation | Apache 2.0 | yes | arXiv:2407.11005 | ||
ragbench/pubmedqa |
Single-hop QA | Retrieval and generation | Apache 2.0 | yes | arXiv:2407.11005 | ||
ragbench/squad |
Single-hop QA | Retrieval and generation | Apache 2.0 | yes | arXiv:2407.11005 | ||
ragbench/techqa |
Single-hop QA | Retrieval and generation | Apache 2.0 | yes | arXiv:2407.11005 | ||
squad_v2 |
Single-hop QA | Generation | 12K | CC BY-SA 4.0 | yes | arXiv:1806.03822 | |
triviaqa |
Single-hop QA | Generation | 95K | Apache 2.0 | - | arXiv:1705.03551 | |
wiki_multihop |
Multi-hop QA | Generation | 193K | Apache 2.0 | - | arXiv:2011.01060 |
Descriptions¶
| Benchmark | Description |
|---|---|
bamboogle |
Bamboogle: 125 multi-hop questions that Google cannot answer directly |
beir/arguana |
ArguAna: argument retrieval (1.4K queries, 8.7K docs) |
beir/climate-fever |
Climate-FEVER: climate claim verification (1.5K queries) |
beir/dbpedia-entity |
DBPedia-Entity: entity retrieval (400 queries, 4.6M docs) |
beir/fever |
FEVER (BEIR): fact verification retrieval (6.7K queries) |
beir/fiqa |
FiQA: financial opinion QA (6.6K queries, 57.6K docs) |
beir/hotpotqa |
HotpotQA (BEIR): multi-hop reasoning (7.4K queries, 5.2M docs) |
beir/msmarco |
MS MARCO (BEIR split): passage retrieval (6.9K dev queries) |
beir/nfcorpus |
NF Corpus: biomedical IR benchmark (3.2K queries, 3.6K docs) |
beir/nq |
Natural Questions (BEIR): Google search questions (3.5K queries) |
beir/quora |
Quora Duplicate Questions: duplicate detection (10K queries) |
beir/scidocs |
SciDocs: scientific document recommendations (1K queries) |
beir/scifact |
SciFact: scientific claim verification (1.1K claims, 5.2K docs) |
beir/trec-covid |
TREC-COVID: COVID-19 related retrieval (50 queries, 171K docs) |
fever |
FEVER: Fact Extraction and VERification (185K claims) |
hotpotqa |
HotpotQA: multi-hop question answering with supporting facts |
kilt/nq |
KILT Natural Questions: knowledge-intensive QA |
kilt/triviaqa |
KILT TriviaQA: trivia questions as knowledge-intensive task |
ms_marco |
MS MARCO: passage ranking (8.8M passages) |
multihop_rag |
MultiHop-RAG: multi-hop RAG benchmark from news articles |
musique |
MuSiQue: multi-hop questions requiring 2-4 composition steps |
natural_questions |
Natural Questions: Google search questions with Wikipedia answers |
popqa |
PopQA: entity-centric QA testing knowledge across popularity tiers (14K pairs) |
ragbench/hotpotqa |
RAGBench (hotpotqa): TRACe-style RAG evaluation |
ragbench/msmarco |
RAGBench (msmarco): TRACe-style RAG evaluation |
ragbench/pubmedqa |
RAGBench (pubmedqa): TRACe-style RAG evaluation |
ragbench/squad |
RAGBench (squad): TRACe-style RAG evaluation |
ragbench/techqa |
RAGBench (techqa): TRACe-style RAG evaluation |
squad_v2 |
SQuAD v2: reading comprehension with unanswerable questions |
triviaqa |
TriviaQA: trivia question answering (95K QA pairs) |
wiki_multihop |
2WikiMultiHopQA: multi-hop reasoning over two Wikipedia articles |