The first step in any search quality evaluation process is to define the ground truth:
a set of triplets <query, document, rating> that specifies how relevant is a document for a certain query.
The ratings can be produced in an explicit or implicit approach:
-
- Explicit – a team of human judges(domain experts hopefully) assigns the rating for the <query, document> pair
- Implicit – user interactions(clicks, downloads, add to carts, add to favourites…) are collected to estimate the relevance rating for the <query, document> pair
The RRE-Enterprise ecosystem supports both.
Let’s see how in this tutorial:
To start configuring your ratings, you need to click on the settings button and select the ‘Ratings’ tab.
This opens you to three functionalities:
-
- Upload of a JSON file containing the ratings in the RRE Open Source format
- Generate ratings from implicit interactions
- View the ratings and edit them
Upload
Video: Ratings1
If you have used Rated Ranking Evaluation in the past, you are familiar with the JSON format supported for the rating file:
{
"tag": "<string>",
"index": "<string>",
"corpora_field": "<string>",
"id_field": "<string>",
"topics": [
{
"description": "<string>",
"query_groups": [
{
"name": "<string>",
"queries": [
{
"template": "<string>",
"placeholders": {
"$key": "<value>",
}
}
],
"relevant_documents": [
{
"document_id": {
"gain": "<number>"
}
}
]
}
]
}
],
"query_groups": [
{
"name": "<string>",
"queries": [
{
"template": "<string>",
"placeholders": {
"$key": "<value>",
}
}
],
"relevant_documents": [
{
"document_id": {
"gain": "<number>"
}
}
]
}
],
"queries": [
{
"template": "<string>",
"placeholders": {
"$key": "<value>",
}
}
]
} -
tag: the unique identifier associated to this rating set (you can later retrieve a set of ratings by tag)index: index name in Elasticsearch/collection name in Solrcorpora_file: the file containing the corpus of information the ratings are built on (the Solr or Elasticsearch documents to index). This is necessary if you want to spin up an embedded evaluation target search engine.
If the target of your evaluation is an existing Elasticsearch/Solr(with its index already there) you don’t need thisid_field: the unique identifier field in the Elasticsearch mapping/Solr schematopics: optional list of topics and/or query groupsdescription: Title of a topic tha groups the inner groups of queriesquery_groups: list of queries that are grouped with the same name and topicname: query group identifierqueries: list of queries to execute for the grouptemplate: String name for the template this query uses- placeholders: Object of key-value pairs to substitute in the template
relevant_documents: list objects with mapping documents to gain or relevance values
query_groups:optional list of objects with related queries.queries: required* list of objects for template and placeholder substitutions for evaluations.- *If
topicsandquery_groupsare not defined
- *If
Video: Ratings2
Curious to know how to automate and simplify the explicit rating judging process?
Our Judgement Collector Browser Plugin got you covered!
A blog about this component is coming soon.
Generate (from implicit interactions)
We have seen so far how to manage explicit ratings, but what about implicit ratings?
RRE-Enterprise offers you the ability to collect user interactions and generate ratings out of configurable metrics.
A <query, document> interaction is represented in JSON and can be collected by a dedicated endpoint in RRE-Enterprise:
POST http://localhost:8080/1.0/rre-enterprise-api/input-api/interaction
{
"collection": "papers",
"query": "interleaving",
"blackBoxQueryRequest": "GET /query?q=interleaving HTTP/1.1\r\nHost: localhost:5063",
"documentId": "1",
"click": 1,
"timestamp": "2021-06-23T16:10:49Z",
"queryDocumentPosition": 0
}collection: index name in Elasticsearch/collection name in Solrquery: a human readable query representation. Most of the time it is going to be the free text queryblackBoxQueryRequest: the full http request associated to the query. Most of the time it is a request to search-API layerdocumentId: the unique identifier for the document (this must be the same used in Elasticsearch/Solr)impression: 1 – if this interaction is an impression (document showed to the user in response to the query)click: 1 – if this interaction is a click of the document in response to the queryaddToCart: 1 – if this interaction is an add to cart of the document in response to the querysale: 1 – if this interaction is a sale of the document in response to the queryrevenue: <int> – the amount of revenue associated to the sale of the document in response to the querytimestamp: the time when the interaction happenedqueryDocumentPosition: the position in the search result list for the document in response to the query
Video: Ratings4-5 (fino 3:17)
Once RRE-Enterprise has been populated with interactions, it is possible to estimate <query, document, rating> triplets, using the generate functionality:
- The Online Metric specifies what to use to estimate the relevance rating
e.g.
selecting Click Through Rate means that RRE-Enterprise will use the impressions and clicks for a certain <query, document> pair to estimate the relevance rating, comparing such value with the minimum and maximum across all queries in the collected interactions - The Filter Query allow to filter the interactions by any property (collection, date..)
- The Tag is the unique identifier associated to the rating set we are generating
Video: Ratings5 fino a 6:04 – Ratings6
View
Once you have uploaded or generated your ratings, you can view and edit them.
First of all, you can retrieve the rating set by tag:
Each row is a single <query, document, rating> triplet.
Clicking on the rating you can edit the rating value and save it automatically.
Clicking on the query shows the details of the triplet.
You can also additionally filter your ratings by collection, topic, and query group, to refine your navigation.
Video: Ratings3
Rated Ranking Evaluator Enterprise
Subscribe to our newsletter
Did you like this post about RRE-Enterprise and How to Manage Your Ratings? Don’t forget to subscribe to our Newsletter to stay always updated on the Information Retrieval world!
