First of all, you need to define your ground truth:
<query, document> pairs tagged with a relevance label that states how relevant a document is for the given query.
The rating files, which are provided in JSON format, are the core input of RRE.
Each rating file is a structured set of <query, document> pairs (i.e. relevant documents for a given query).
In the ratings file, we can define all aspects of the information need supported by RRE: corpus, topics, query groups, and queries.
The current implementation uses a configurable judgement range:
e.g.
- 1 => marginally relevant
- 2 => relevant
- 3 => very relevant
within the “relevant_documents” node, you can provide the judgements in one of the following (alternative) ways: