Ground truth in a search evaluation consists of a <query, document, rating> dataset that specifies how relevant is a given document for a given query.
We already described how RREE manages ratings. This article would like to focus on explicit ratings and how RREE can help you generate them.
A rating process is defined as “explicit” when the user explicitly and directly provides a value about an item’s relevance.
The user can be offered a list of search results and be provided with choices to select their favourite items out of that list.
So far, so good: the process sounds clear. However, producing high-quality explicit ratings can be very hard for several reasons:
- it is a human process, therefore may be discretionary, and it could depend on the actor’s perspective.
- there should be a user-friendly tool that helps and facilitates the lists retrieval and judgment assignment: remember, in this context, a user is a domain expert; they are not a technician
RREE addresses the second point above with a tool called Judgments Collector Browser Plugin (JCBP).
We are talking about generating explicit ratings for search-based applications. Those applications have a website where end-users can search, browse, see, and buy things, right?
Why don’t we use the same website for generating explicit ratings? That should be the most familiar tool for who should develop them.
That is precisely the idea behind JCBP: as the name suggests, it is a plugin installed in the user’s browser which applies an overlay on top of a given website/web portal.
In a nutshell, if this is a website, we want to generate explicit ratings: