A Software Engineer is always required to give his customers a concrete evidence about deliverables quality. A Search Engineer deals with a specialisation of such generic Software Quality, which is called Search Quality.
What is Search Quality? And why is it so important in a search infrastructure? After all, the “Software Quality” should be omni-comprensive, it should always include everything (and actually it is), but when we are dealing with search systems, the quality is a very abstract term, which is very hard to define in advance.
The functional correctness of a search infrastructure (assuming the correctness is the only factor which influences the system quality – and it isn’t) is naturally associated with human judgments, with opinions, and unfortunately we know opinions can be different among people.
The business stakeholders, which will get a value from a search system, can belong to different categories, can have different expectations, and they can have in mind a different idea about the expected system correctness.
In this scenario a Search Engineer is facing many challenges in terms of choices, and at the end, he has to provide concrete evidences about the functional coverage of those choices.
This is the context where we developed the Rated Ranking Evaluator (hereafter RRE).
Haystack 2019 day 1 - grab your popcorn, it's Relevance Avengers time - Opensource Connections
June 18, 2020[…] stop the Avengers references now before I infer one of our speakers was green and angry) on the Rated Ranking Evaluator relevance testing tool. He showed us the heirarchical model for test queries they have developed […]