Dataset Generator
Build high-quality relevance datasets for search evaluation
Dataset Generator (DAGE) is an open-source command-line tool that automates one of the most time-consuming tasks in Information Retrieval: building relevance datasets.
By combining search engines with Large Language Models (LLMs), Dataset Generator retrieves documents from an existing index, generates realistic queries, assigns relevance judgments, and exports evaluation datasers ready to be used with search quality tools such RRE, Quepid and MTEB.
THE CHALLENGE
Building relevance datasets is difficult
Manual relevance annotation is expensive, time-consuming, and hard to scale.
As annotation sessions become longer, reviewers lose focus and consistency, making relevance judgments less reliable.
Human judgments are subjective and often inconsistent across annotators.
Despite having queries from production logs, assigning relevance ratings remain a major bottleneck.
- Dataset Generator automates this process using LLMs to generate queries and relevance judgments, making the workflow faster, reproducible, and fully configurable.
KEY FEATURES
Dataset Generator is everything you need to generate evaluation datasets
LLM-powered queries
Generate natural language queries from your indexed documents, with optional support for user-defined queries.
Automated relevance judgments
Assign binary or graded relevance ratings using LLMs, dramatically reducing manual effort.
Multiple search engines
Works with Apache Solr, Elasticsearch, OpenSearch and Vespa.
Flexible configuration
Configure search engines, document filters, fields, query templates, relevance scale, LLM provider, output format, and more via YAML.
Multiple output formats
Export the dataset in the native Dataset Generator format (evaluation_dataset), or in formats compatible with RRE, Quepid, and MTEB.
Explainability
Optionally store LLM explanations for every relevance judgment to improve transparency and reproducibility.
HOW IT WORKS
From documents to evaluation datasets in four steps
1
Retrieve documents
Connect to your search engine and retrieve documents according to your filters and configuration.
2
Generate queries
The LLM generates keyword or natural language queries from the document contents. You can also add your own queries.
3
Assign relevance ratings
Each query-document pair is evaluated by the LLM using binary or graded relevance scales.
4
Export the dataset
The dataset is exported in the evaluation_dataset format or in formats compatible with RRE, Quepid or MTEB, ready for offline search evaluation.
SUPPORTED TECHNOLOGIES
Search Engines
- Apache Solr
- Elasticsearch
- OpenSearch
- Vespa
Output Formats
- evaluation_dataset (Native Dataset Generator Format)
- Rated Ranking Evaluator (RRE)
- Quepid
- Massive Text Embedding Benchmark (MTEB)
FAQ
From documents to evaluation datasets
Everything you need to know about generating high-quality relevance datasets with LLMs.
Why is building relevance datasets difficult?
Building relevance datasets manually is expensive, time-consuming, and difficult to scale. While extracting queries from production logs is often straightforward, assigning meaningful relevance ratings to query-document pairs relies on human judgment, making the process repetitive, subjective, and prone to inconsistency.
What search engines are supported?
Dataset Generator currently supports:
- Apache Solr
- Elasticsearch
- OpenSearch
- Vespa
The search engine, collection or index, filters, and document fields are configured through a YAML configuration file.
What LLM providers are supported?
Dataset Generator currently includes built-in support for OpenAI and Google Gemini. Since it is built on LangChain, it can be extended to support any LLM provider available through LangChain by adding the corresponding integration.
The provider, model, token limits, API credentials, and other model-specific parameters are defined in the LLM configuration file.
What output formats are available?
Dataset Generator can export datasets in multiple formats, including:
- Dataset Generator native format (evaluation_dataset)
- MTEB
- Rated Ranking Evaluator (RRE)
- Quepid
Each format produces the files required by the corresponding evaluation workflow.
What relevance scales can be used?
Dataset Generator supports two relevance scales:
Binary
- 0 = not relevant
- 1 = relevant
Graded
- 0 = not relevant
- 1 = maybe ok
- 2 = that’s my result
The relevance scale is defined in the configuration file.
Can I use my own queries?
Yes. Dataset Generator can use a text file containing predefined queries. Those queries can either replace generated queries or be combined with them, depending on the configuration.
Does Dataset Generator explain its relevance ratings?
Yes. Optionally, Dataset Generator can save the explanation produced by the LLM for each relevance judgment. The output contains query, document ID, rating, and explanation, supporting transparency and reproducibility.
How does Dataset Generator integrate with other search evaluation tools?
Dataset Generator produces datasets compatible with common search evaluation workflows. Depending on the selected output format, it generates resources that can be used with MTEB, Rated Ranking Evaluator (RRE) or Quepid.
Is Dataset Generator configurable?
Yes. The entire generation process is controlled through a YAML configuration file. You can configure the search engine, document filters, fields used for query generation, query templates, LLM provider, relevance scale, output format, autosave behavior, and explainability options.
Search Quality Ecosystem
Dataset Generator integrates naturally with other Sease open source projects to help you build, evaluate and improve search quality systems end to end.
OPEN SOURCE PROJECT BUILT BY SEARCH EXPERTS
Vector Search Doctor
Evaluate embedding models and vector search performance on your own data.