Ground truth in a search evaluation consists of a <query, document, rating> dataset that specifies how relevant is a given document for a given query.
We already described how RREE manages ratings. This article would like to focus on explicit ratings and how RREE can help you generate them.
A rating process is defined as “explicit” when the user explicitly and directly provides a value about an item’s relevance.
The user can be offered a list of search results and be provided with choices to select their favourite items out of that list.
So far, so good: the process sounds clear. However, producing high-quality explicit ratings can be very hard for several reasons:
- it is a human process, therefore may be discretionary, and it could depend on the actor’s perspective.
- there should be a user-friendly tool that helps and facilitates the lists retrieval and judgment assignment: remember, in this context, a user is a domain expert; they are not a technician
RREE addresses the second point above with a tool called Judgments Collector Browser Plugin (JCBP).
We are talking about generating explicit ratings for search-based applications. Those applications have a website where end-users can search, browse, see, and buy things, right?
Why don’t we use the same website for generating explicit ratings? That should be the most familiar tool for who should develop them.
That is precisely the idea behind JCBP: as the name suggests, it is a plugin installed in the user’s browser which applies an overlay on top of a given website/web portal.
In a nutshell, if this is a website, we want to generate explicit ratings:
After activating JCBP, the overlay applies a judgment layer where each results can be rated:
In the example above, the user ran a search on the website like in the first screenshot. The difference is the judgment widget on top of each result, which allows expressing a rating.
In that specific case, the user gave 2 (Very relevant) to the first result and 1 (Relevant) to the second one.
The plugin, based on a screen-scraping approach, requires a configuration specific for each website we want to involve in the rating generation process. When it is activated, the plugin is able to detect existing (previously saved) configuration for a website and applies the corresponding overlays.
The popup says “only one” because theoretically, we could have multiple configurations for the same website: for example, multiple search results page using a different structure would probably require a different configuration.
Once the ratings have been collected they can be saved and sent to RREE by means of a specific button.
That’s all! The power of our browser plugin is the simplicity: customers can create explicit ratings directly using their web portal, without any other tool any application.
The learning curve of the overlay applied by JCBP is very shallow: all that a user is supposed to do is to:
- run a search
- express a rating for each result
- send the ratings to RREE
Want to see it in action? Here’s a demonstration video:
Rated Ranking Evaluator Enterprise
Subscribe to our newsletter
Did you like this post about RRE-Enterprise: Make Explicit Ratings Easy? Don’t forget to subscribe to our Newsletter to stay always updated in the Information Retrieval world!
