The Importance of Online Testing in Learning to Rank – Part 1
You have just trained a learning to rank model and you now want to know how it performs.
You can start by looking at the evaluation parameters returned by the train on the test set, but you are still not sure of which will be the impact in using it in a real website.
This is where online testing comes in. It remains the optimal way to check how your model performs on a real scenario and it can give you the necessary information to evaluate, improve and better understand the behavior of your model.
In this blog post I present you the online testing for learning to rank.
I talk about the advantage of this approach and why it should be used, finally I show you how it can be implemented and which are the mistakes to avoid during this realization.
In this first Part I talk about the state of the art, while in Part 2 I show you how to implement interleaving.
In the Industry
Unfortunately there are still many industries today that don’t use online testing in learning to rank evaluation. Maybe they don’t know it exists, or they don’t have the right tools to do it; potentially they don’t have the competence to understand the results.
In any case this is a shame because online testing is vital for the correct tuning of the model and for its improvement.
We can surely rely on offline evaluation, checking the performance of the model on the test set or through the analysis of different offline metrics, but we can’t really be sure what the impact will be on the website.
There are several problems that are hard to be detected with an offline evaluation.
With a wrong Test Set is possible to obtain model evaluation results that aren’t reflecting the real model improvement
First of all it could happen that we don’t correctly train and/or test the model. If we make mistakes in creating the training and/or test set we will have unreliable results that can be misunderstood.
With a wrong test set it is possible to obtain better model results that aren’t reflecting a real model improvement. In this case, we have an issue that can’t be caught using the offline evaluation.
When is a Test Set wrong?
When it doesn’t represent and generalise well the ranking problem.
Some examples:
-
- One sample per query group: ranking a single result for a query is not a ranking problem at all.
- One relevance label for all the samples of a query group: so whatever ranking the model does, offline evaluation metric will be the same.
- Starting from the data set: which interactions do you consider for the training and which for the test? The split in training and test set is a very important phase that need to be studied in depth. Ideally we would like to create two sets that follow the same probability distribution in terms of queries and relevance labels.
Another problem is finding a direct correlation between the offline evaluation metrics and the parameters we consider for the online model performance.
It isn’t that obvious and neither simple to understand how an improvement on the offline metrics will reflect on page views, clicks, sales and revenues.
Relying on generated relevance labels can be a problem.
These aren’t always reliable and doesn’t always reflect the real users needs
The most commonly used technique for the offline evaluation is the Cranfield one.
In this approach a group of experts create a set of relevance judgments starting from a selected set of documents. In particular, they assign a relevance label to each document of the selected set. The application of this paradigm can bring to two main issues:
-
- The effort required for the relevance set creation, this is a very expensive phase.
- Relying on generated relevance labels. These aren’t always reliable and doesn’t always reflect the real users needs.
If we want to overcome these issues we have to pass to the online testing.
Online Testing
As already said before, using online testing can lead to many advantages:
-
- The reliability of the results. In this approach we can directly observe the user behavior and from this, understand what are the documents of greatest interest to him.
- Give a direct interpretation of the results. With online evaluation we can directly see what are the consequence of the model in terms of number of page views, clicks, sales or revenues.
- We can also better understand the model behavior. We can see how the user interact with the model and figure out how to improve it.
Clearly these benefits do not come without an effort, but they are worth it.
At the moment, in the world of learning to rank and beyond, there are two types of online testing that are widely spread in industry: A/B testing and Interleaving .
A/B Testing
Let’s start taking a look at how A/B testing works.
Since we are talking about learning to rank, let’s say you have two trained learning to rank models: model A and model B. These models aren’t identical and you want to compare them in order to find the best one.
A/B testing allows you to make this comparison. In particular, it allows you to divide your query traffic in two groups. The first group of users interact with the website using model A, while the second one interact with the website using model B. In this way we can observe the user behavior in both the scenarios and look to our target metrics like clicks, sales, revenues, to understand which model performs better.
This is a great tool, it gives us many useful information and above all it relies on real user interactions that should reflect the real user’s needs.
Even if A/B testing is great, we have to be very careful in how we implement it.
What not to do
Let’s say we have a website, in which we use our learning to rank model for the query search pages. Suppose we have 2 pages:
-
- One is the homepage. Here we show few interesting new documents that can be directly bought. These document are static and are not ordered by the learning to rank model.
- One is the search page. Here we can make a query and search through the results the most interesting documents for us. These documents are actually ranked by our learning to rank model.
When A/B Testing we need to make sure we consider only the interactions coming from search result pages ranked by the models we are comparing
When testing we have to make sure that all the results we are considering, come from the search page and not from the homepage.
A wrong analysis could indeed lead to a wrong conclusion on models’ performances.
In particular I would like to highlight two possible situations:
-
- We conclude that model A is better that model B when it is the opposite.
- We correctly conclude that model A is better than model B, but we are unable to identify the percentage of improvement.
Let’s see the first scenario.
Suppose we are analyzing model A. We obtain: 10 sales from the homepage and 5 sales from the search page.
Then suppose to analyze model B. We obtain: 4 sales from the homepage and 10 sales from the search page.
If we look only at the total we will see: 15 sales from model A and 14 sales from model B, concluding that model A is the best one. This is true for the homepage (10 sales > 4 sales), but not for the search page (5 sales < 10 sales) where we are actually using the learning to rank model.
So what do we make of that?
In this way we wrongly assert that model A is the best one, while we can clearly seen that in the search page model B performs better.
We aren’t therefore evaluating the model on the page where it actually acts, but we draw conclusions lead by the homepage where the model is not involved.
The same misunderstanding can arise when the improvement happen only in the homepage and not in the search page where the models are equivalent. Below you find an example of the situation.
Suppose we are analyzing model A. We obtain: 10 sales from the homepage and 10 sales from the search page.
Then suppose to analyze model B. We obtain: 5 sales from the homepage and 10 sales from the search page.
Here, if we look only to the total sales, we can see that model A obtains 20 sales while model B obtains 15 sales. In this case we will conclude that model A is better than model B.
So what do we make of that?
If we simply look to the results for the search page, this isn’t true because both obtains the same number of sales (10).
The improvement is due to the homepage, that doesn’t use the model, but we could wrongly attribute it to the model performance.
Let’s now see the second scenario.
Suppose we are analyzing model A. We obtain: 12 sales from the homepage and 11 sales from the search page.
Then suppose to analyze model B. We obtain: 5 sales from the homepage and 10 sales from the search page.
Here, if we look to the total sales, we can see that model A obtains 23 sales while model B obtains 15 sales. In this case we will conclude that model A is better than model B.
So what do we make of that?
This is true because, even looking only to the search page, model A obtain more sales than model B (11 sales > 10 sales). But.. how much was the improvement? Looking to the total we see that model A obtains 8 more sales than model B, a great improvement! But in reality, looking only to the search page we can see that model A obtains just 1 sale more than model B, not such a big difference.
It is important to filter the interactions during testing.
We have to consider only interactions that are really related to the models under evaluation
These examples show that it is important to filter our interactions during testing. We have to consider only interactions that are really related to the model if we want to evaluate it. Considering also the others, like the ones of page one, brought to an addition of noise that can hide the real model performances.
Interleaving
A/B testing is widely used in industry, it’s very useful and it isn’t too complicate to implement.
For these reasons, this approach has been the focus of a large number of studies with the purpose of improving the current implementation.
These studies led to a new similar approach called: interleaving.
Also in this kind of approach two models are compared. The main difference with A/B testing is that, here, we directly compare the two models showing both their results to the same user at the same time.
Let’s explain more in details how it works:
Interleaving prevents the chance of exposing users to a bad model for the entire duration of the test experiment
There are several advantages in interleaving with respect to A/B testing:
-
- It reduces the problem with users’ variance due to their separation in groups (group A and group B).
- It is more sensitive in the comparison between models.
- It requires less traffic.
- It requires less time to achieve reliable results.
- It doesn’t necessarily expose a bad model to a subpopulation of users
Let’s see how interleaving works and how it achieves these improvements.
How it works
Suppose we have two learning to rank models called: model_A and model_B.
Given a query q each model responds with a ranked list of documents: lA and lB.
At this point, instead of returning the list lA to a group of users A and the list lB to a group of users B, a unique result list lI is created and returned to the user.
There isn’t anymore a separation of the query traffic, each user is exposed to this result list created from both the models with the same interleaving approach.
This unique list will therefore contain search results from both model_A and model_B.
There are several ways of implementing this kind of list, all trying to be as fair as possible.
What do we mean with fair?
This concept is highly related to the choice of the search results.
Ideally, we would like the list to contain an equal number of documents from both the models, so the preference of one model with respect to the other doesn’t depend on the number of shown items.
Secondly, we would like that the position of the shown items doesn’t influence the preference of the user for one of the two models. If we have all the documents from model_A on the top 10 positions and all the documents from model_B in the underlying positions, it is possible that documents from B aren’t chosen only because of their low ranks.
The first survey of how testing learning to rank models happens in the industry and how it looks in the state of the art ends here, but if you’re curious of learning more details about how this can be implemented let’s meet in Part 2!
Shameless plug for our training and services!
Did I mention we do Learning To Rank and Search Quality Evaluation training?
We also provide consulting on these topics, get in touch if you want to bring your search engine to the next level!
Subscribe to our newsletter
Did you like this post about Online Testing in Learning to Rank? Don’t forget to subscribe to our Newsletter to stay always updated from the Information Retrieval world!
