This blog post aims to illustrate how to generate the query Id and how to manage the creation of the Training Set
Common errors and warnings in manipulating feature stores and models in Solr. Pay attention also to JVM Heap and Zookeeper.
TUE 30TH MARCH 2021STARTING FROM 06:00 PM (GMT) We are pleased to announce the eighth London Information Retrieval Meetup, a free evening meetup aimed to Information Retrieval passionates and professionals who are curious to explore and discuss the latest trends in the field.This time we go fully remote, given the COVID-19 situation and the impossibility…
This blog post explains how to list, upload and delete feature and model stores in Apache Solr for learning to rank.
This blog post is about several analysis on a LTR model and its explanation using the open source library SHAP
In this blogpost I want to explore what possibilities elasticsearch gives us for storing fields and retrieve them at query time from the performance point of view. In fact, lucene, the underlying library upon which elasticsearch and solr are built, provides two ways for storing and retrieving fields: stored fields and docvalues. In addition, elasticsearch…
Introduction With Rated Ranking Evaluator Enterprise approaching soon, we take the occasion of explaining in details why Offline Search Quality Evaluation is so important nowadays and what you can do already with the Rated Ranking Evaluator open-source libraries. More news will come soon as we are approaching the V1 release date. Stay tuned! Search Quality…
This blog post aims to illustrate step by step a Learning to Rank project on a Daily Song Ranking problem using open source libraries.
Interleaving is an online evaluation approach for ranking functions, contributed to Apache Solr Learning to Rank by Sease.
After the success of the past editions, we are happy to announce the seventh London Information Retrieval Meetup, a free evening meetup aimed to Information Retrieval passionates and professionals who are curious to explore and discuss the latest trends in the field.This time we go fully remote, given the COVID-19 situation and the impossibility of…
Are you looking for a complete apache solr beginner course? Here you can find the best, online and with a small price!
In this blog post we will briefly recap things we can’t do when using the alias field type in Elasticsearch.
Join us at our sixth Information Retrieval meetup (fully remote), to explore and discuss the latest trends in the field.
In this post we describe what is an Intervals Table and how to build it using a Behaviour-Driven-Development (BDD) approach.
Introduction A common problem with machine learning models is their interpretability and explainability.We create a dataset and we train a model to achieve a task, then we would like to understand how the model obtains those results. This is often quite difficult to understand, especially with very complex models. In this blog post, I would…
After the very warm reception of the first year, the fifth London Information Retrieval Meetup is approaching (23/06/2020) and we are excited to add more details about our speakers and talks!The event is going to be fully remote (given the COVID-19 situation) and free! You are invited to register : Our second speaker is Martin…
If you have read Part 1 of this blog post, you should know by now how many fantastic things can be done with online testing! In particular, the advantages that interleaving brings with respect to A/B testing, but you are still waiting for the answer to a question: how to implement it? Let’s see together…
You have just trained a learning to rank model and you now want to know how it performs. You can start by looking at the evaluation parameters returned by the train on the test set, but you are still not sure of which will be the impact in using it in a real website. This…
This blog post aims to give a better understanding of Docvalues and stored fields in Apache Solr for the operations in which they can be used interchangeably.
I’ve always loved R&D and I’ve always been fascinated by seeing in action the implementation of my ideas. It was in July 2010 when I started my professional journey in the Open Source search landscape, I was a Junior software engineer at the time, and after few months of post graduate research at Roma3 University…
This blog post is about our latest contribution to the Apache Lucene/Solr project:introducing the ability of assigning different weights to synonyms.This contribution aims to help users that deal with complex synonyms dictionaries where it’s important to associate a numerical weight to each of them, for example to boost the ones that are more important in…
In this post we describe what is an Intervals Table and how to build it using a Behaviour-Driven-Development (BDD) approach.
In this post we describe an approach to solve the problem of an application that requires both Full and Atomic Updates, using one of the powerful concepts in Object Oriented Programming: Polymorphism.
After the very warm reception of the first year, the fourth London Information Retrieval Meetup is approaching (11/02/2020) and we are excited to add more details about our speakers and talks!The event is free and you are invited to register : https://www.eventbrite.com/e/london-information-retrieval-meetup-february-2020-tickets-89056738101 Our first speaker is Anna Ruggero, one of our R&D software engineers: Anna…
Let’s say you need to write a component, a request handler, or in general some piece of custom code that needs to be plugged into Solr. Or, you need to have a deeper understanding about some Lucene/Solr internals, following what actually happens within the code. I know: unit tests, integration tests, everything to make…
It was the spring of 2018, Andrea was strenuously working on a customer project, continuously tuning search configurations and checking the ground truth for certain queries manually. That was pretty much the standard at the time, the brilliant Quepid[1] from our friends at Open Source Connection helped in some use cases, but there was nothing…
This is the last post of the Entity Search with graph embeddings serie. In Part 2 and Part 3 we illustrated the core of the dissertation describing in detail the implementation of our solution pipeline. In this final part we will see some evaluation measures and results. We will draw some conclusions explaining which were…
The target of this blog post is to shed some light on the topic of request handlers and their relation with the QT request parameter in recent Apache Solr ( >= 7.0). Definitions A Request Handler is defined with a name and a class.It defines an API endpoint to handle and process http requests. The…
Third part of the journey into Entity Search trough embeddings. Focus of the post is the ranking phase.
Let’s continue our journey into this entity search thesis! In Part 1 we have described what entities and entity search are. We have explained how this search is implemented in the state-of-the-art. We have also introduced the new approach of this dissertation specifying also the dataset and the test collection used. Finally we have described…
This series of blog posts wants to describe my master degree dissertation done with the supervision of Prof. Gianmaria Silvello at the University of Padova. The main focus of this project is in the use of graph embeddings in order to create virtual documents for the Information Retrieval Entity Search task. This thesis description is…
After the very warm reception of the first and second edition, the third London Information Retrieval Meetup is approaching (21/10/2019) and we are excited to add more details about our speakers and talks!The event is free and you are invited to register : https://www.eventbrite.com/e/london-information-retrieval-meetup-october-tickets-74403100677 Our second speaker is Andrea Gazzarini, our founder and software engineer:…
When using nested documents and the Apache Solr Block Join functionality it is a common requirement to query for an entity (for example the parent entity) and then retrieve for each search result all(or some of) the related children. Let’s see the most important aspects of such functionality and how to apply complex queries when…
After the very warm reception of the first edition, the second London Information Retrieval Meetup is approaching (25/06/2019) and we are excited to add more details about our speakers and talks!The event is free and you are invited to register : https://www.eventbrite.com/e/london-information-retrieval-meetup-june-tickets-62261343354 Our first speaker is René Kriegler, freelance search consultant and search engineer : René Kriegler René…
This blog is a quick summary of my (subjective) experience at Haystack 2019 : the Search Relevance Conference, hosted in Charlottesville (Virginia, USA) from 24/04/2019 to 25/04/2019.References to the slides will be updated as soon as they become available. First of all my feedback on the Haystack Conference is extremely positive.From my perspective the conference…
The London Information Retrieval Meetup is approaching (19/02/2019) and we are excited to add more details about the speakers and talks!The event is free and you are invited to register :https://www.eventbrite.com/e/information-retrieval-meetup-tickets-54542417840 After Sambhav Kothari, software engineer at Bloomberg and Elia Porciani, R&D software engineer at Sease, our last speaker is Andrea Gazzarini, founder and software engineer at…
What happens with facets aggregations on fields when documents in the results have been filtered by Access Control Lists ?In such scenarios it is important to use the facet mincount parameter.That specifies the minimum count in the result set for a facet value to appear in the response: mincount=0, all the facet values present in…
Apache Solr distributed faceting feature has been introduced back in 2008 with the first versions of Solr (1.3 according to this jira[1]) . Until now, I always assumed it just worked, without diving too much into the details. Nowadays distributed search and faceting are extremely popular, you can find them pretty much everywhere (in the…
In this post we’ll cover two additional synonyms scenarios and we’ll try to summarise all previous tips in a coincise form. Following the approach of the previous posts [1] [2] [3], everything can be applied both to Apache Solr and Elasticsearch. Preconditions Synonyms and stopwords at query time: this is not just a “theoretical” constraint;…
The Context Brief recap of where we arrived in the preceding article: we had the following synonyms and stopwords settings: synonyms = {“out of warranty”,”oow”} stopwords = {“of”} Both of those filters were configured exclusively at query-time; the synonym filter first and then the stopwords filter. Using the built-in StopFilter we had a synonym detection…
The Context The scenario description is quite simple: we want to use synonyms and stopwords. Following the path of our previous article, we will introduce an additional component in the analysis chain: a StopFilter, which, as the name suggests, removes a set of words from an incoming token stream. We will use the following data…
This flash blog post will address a very specific and common problem : how to manage entities/concepts composed by multiple terms in a vanilla Apache Solr/Elasticsearch instance ( no plugins or extensions to install). The (deployment) context An Elasticsearch or Apache Solr infrastructure where you cannot install third-party components (e.g. plugins, filters, query parsers). This can…
A Software Engineer is always required to give his customers a concrete evidence about deliverables quality. A Search Engineer deals with a specialisation of such generic Software Quality, which is called Search Quality. What is Search Quality? And why is it so important in a search infrastructure? After all, the “Software Quality” should be omni-comprensive,…
The Apache Lucene/Solr suggesters are important to Sease : we explored the topic in the past[1] and we strongly believe the autocomplete feature to be vital for a lot of search applications. This blog post explores in details the current status of the Lucene BlendedInfixSuggester, some bugs of the most recent version ( with the…
At the time we speak ( Solr 7.3.1 ) SolrCloud is a reliable and stable distributed architecture for Apache Solr. But it is not perfect and failures happen. Apache Zookeeper[1] is the system responsible of managing the communications across the SolrCloud cluster. It contains the shared collections configurations and it has the view of the…
Scenario You’re working as a search engineer for XYZ Ltd, a company which sells electric components. XYZ provided you the application logs of the last six months, and some business requirements. Two kinds of customers, two kinds of requirements, two kinds of search The log analysis shows that XYZ has mainly two kinds of customers:…
At the time we speak ( Solr 7.3.0 ) SolrCloud is a reliable and stable distributed architecture for Apache Solr. But it is not perfect and failures happen. This lightening blog post will present some practical tips to follow when a specific shard of a collection is down with no leader and the situation is…
Quantity detection? What is a quantity? And why do we need to detect it? A quantity, as described by Martin Fowler in his “Analysis Patterns” [1] is defined as a pair which combines an amount and unit (such as 30 litres, 0.25 cl, or 140 cm). In search-based applications, there are many cases where you may…
This blog is a quick summary of my (subjective) experience at ECIR 2018 : the 40th European Conference on Information Retrieval, hosted in Grenoble (France) from 26/03/2018 to 29/03/2018. Deep Learning and Explicability Eight long papers accepted were about Deep Learning. The topics “Neural Network” and “Word Embedding” were the most occurring in the accepted…
What are “Invisible Queries”? This is an extract of an article [1] on Lucidworks.com, by Grant Ingersoll, talking about invisible queries: “It is often necessary in many applications to execute more than one query for any given user query. For instance, in applications that require very high precision (only good results, forgoing marginal results), the…
Distributed search is the foundation for Apache Solr Scalability : It’s possible to distributed search across different Apache Solr nodes of the same collection ( both in a legacy[1] or SolrCloud[2] architecture), but it is also possible to distribute search across different collections in a SolrCloud cluster. Aggregating results from different collections may be useful…
Last Stage Of The Journey This blog post is about the Apache Solr Learning To Rank ( LTR ) integration. We modelled our dataset, we collected the data and refined it in Part 1 . Trained the model in Part 2 . Analysed and evaluate the model and training set in Part 3 . We…
Apache Solr Learning to Rank – Things Get Serious This blog post is about the Apache Solr Learning to Rank Tools : a set of tools to ease the utilisation of the Apache Solr Learning To Rank integration. The model has been trained in Part 2, we are ready to deploy it to Solr, but…
Model Training For Apache Solr Learning To Rank If you want to train a model for Apache Solr Learning To Rank , you are in the right place. This blog post is about the model training phase for the Apache Solr Learning To Rank integration. We modelled our dataset, we collected the data and refined it…
Learning To Rank In Apache Solr Introduction This blog post is about the journey necessary to bring Learning To Rank In Apache Solr search engines. Learning to Rank[1] is the application of Machine Learning in the construction of ranking models for Information Retrieval systems. Introducing supervised learning from user behaviour and signals can improve the relevancy…
Introduction This blog post is about the Lucene Inverted Index and how Apache Solr internally works. When playing with Solr systems, understanding and properly configuring the underline Lucene Index is fundamental to deeply control your search. With a better knowledge of how the index looks like and how each component is used, you…
This blog post is about the Apache Solr Autocomplete feature. It is clear that the current documentation available on the wiki is not enough to fully understand the Solr Suggester : this blog post will describe all the available implementations with examples and tricks and tips. Introduction If there’s one thing that months of Solr-user…
Introduction This blog post is about the Solr classification module and the way Lucene classification has been integrated at indexing time. In the previous blog [1] we have explored the world of Lucene Classification and the extension to use it for Document Classification . It comes natural to integrate Solr with the Classification module and…
Introduction This blog post describes the approach used in the Lucene Classification module to adapt text classification to document ( multi field ) classification. Machine Learning and Search have been always strictly associated. Machine Learning can help to improve the Search Experience in a lot of ways, extracting more information from the corpus of documents,…