We’ve seen in our previous blog post tutorial how to define the golden truth, a fundamental milestone for Search Quality Evaluation.
This blog focuses on the set up of the target of the evaluation: your search engine based on top of Elasticsearch or Solr.
RRE-Enterprise supports two modalities:
-
- Embedded: RRE-Enterprise spins up with Docker an Elasticsearch/Solr instance, using the configurations and test data provided. The instance will be available for the time necessary to run the evaluation and then it is switched off.
N.B. you need docker installed on your machine - External: RRE-Enterprise points to an existing Elasticsearch/Solr instance.
N.B. the instance must be network accessible from RRE-Enterprise server
- Embedded: RRE-Enterprise spins up with Docker an Elasticsearch/Solr instance, using the configurations and test data provided. The instance will be available for the time necessary to run the evaluation and then it is switched off.
EMBEDDED Configuration Upload
When configuring an Embedded approach you need to provide the search engine configurations.
These configurations depend on your search engine implementation (Elasticsearch or Solr).
e.g.
– solrconfig.xml, schema.xml, synonyms.txt ect for Solr
– mapping parameters and settings for Elasticsearch
RRE-Enterprise supports a direct upload for such config: so you just need to provide the zipped folder with all the configurations:
Video:TargetSearchEngine1
EMBEDDED Configuration from Git
It’s a good practice to source control your search engine configuration with Git.
In this way you can approach the work of tuning your configurations incrementally, keeping track of various versions.
RRE-Enterprise got you covered in this scenario: you can fetch the configuration directly from your Git repository, a specific branch, and a specific commit.
Once given a Unique Identifier to the repository, the URL, and set up a public key that RRE-Enterprise uses to programmatically access the repo, you are good to go and this will be usable at evaluation time to fetch a specific version of your configurations.
Video:TargetSearchEngine2
EMBEDDED Query Templates
When using the Embedded approach, you won’t have a black box search API in the middle (you are assuming your front end directly talks with the final search engine Elasticsearch or Solr).
RRE-Enterprise spins the search engine instance, so when evaluating it, it will run the queries from the rating set directly.
To simplify the JSON rating file, we can define a set of Query Templates, that contain the query structure (Elasticsearch DSL query or Solr query).
The templates are referenced in the ratings and will be used by RRE-Enterprise to build the queries to run against the target search engine at evaluation time.
N.B. Query Templates are associated with a specific search engine configuration version that you can specify when uploading them
{
"query": {
"multi_match": {
"query": "$query",
"fields": ["title","overview","cast.name","directors.name"]
}
}
}multi_match_with_titles_and_starring.json
Video:TargetSearchEngine3
External
Most of the time your front end doesn’t talk directly with Elasticsearch or Solr: there’s a search API layer in the middle.
This search-API builds any sort of Elasticsearch or Solr query, it may contain very complex business logic.
We call it Black Box Search API: because RRE-Enterprise doesn’t need to know what are the implementation details of such middle layer nor the programming language involved, it just needs to know the REST endpoint to use to run the evaluation.
N.B. RRE-Enterprise supports REST Black Box Search API only
When configuring the target of your evaluation to be an external search engine you need to make sure you configure all the information RRE-Enterprise needs to talk with such search engine instance.
With this approach, you can point directly to your custom Black Box Search API!
RRE-Enterprise is capable of evaluating directly your search-API (that internally talks with Elasticsearch or Solr)
Let’s see how you can configure this:
First, you define the endpoint of the Black Box Search API:
Then you need to configure the endpoints to access the search engine instance itself (Elasticsearch or Solr):
In this example, we are configuring an Elasticsearch instance.
You may notice an unusual configuration: ‘Log Extractor port’.
This is a simple script you need to install in the environment that is hosting the Elasticsearch instance.
It’s available for free, let us know if you need help with the installation!
Video:TargetSearchEngine4
And that’s it! Now you are able to set up your target search engine and proceed with the next episode:
RRE-Enterprise: How to Manage Your Data Collections
Rated Ranking Evaluator Enterprise
Subscribe to our newsletter
Did you like this post about Drop constant features: a real-world Learning to Rank scenario? Don’t forget to subscribe to our Newsletter to stay always updated from the Information Retrieval world!
