Berlin Buzzwords
live talk
Germany’s most exciting conference on storing, processing, streaming and searching large amounts of digital data, with a focus on open-source software projects.
Location: Kulturbrauerei, Berlin
Date: 12th-14th June 2022
Neural Search Comes to Apache Solr:
Approximate Nearest Neighbor,
BERT and More (Buzzwords)!
The first integrations of machine learning techniques with search allowed to improve the ranking of your search results (Learning To Rank) – but one limitation has always been that documents had to contain the keywords that the user typed in the search box in order to be retrieved. For example, the query “tiger” won’t retrieve documents containing only the terms “panthera tigris”. This is called the vocabulary mismatch problem and over the years it has been mitigated through query and document expansion approaches.
Neural search is an Artificial Intelligence technique that allows a search engine to reach those documents that are semantically similar to the user’s query without necessarily containing those terms; it avoids the need for long lists of synonyms by automatically learning the similarity of terms and sentences in your collection through the utilisation of deep neural networks and numerical vector representation.
This talk explores the first Apache Solr official contribution about this topic, available from Apache Solr 9.0.
During the talk we will give an overview of neural search: we will describe vector representations for queries and documents, and how Approximate K-Nearest Neighbor (KNN) vector search works.
We will show how neural search can be used along with deep learning techniques (e.g, BERT) or directly on vector data, and how we implemented this feature in Apache Solr.
Word2Vec model to generate synonyms
on the fly in Apache Lucene
If you want to expand your query/documents with synonyms in Apache Lucene, you need to have a predefined file containing the list of terms that share the same semantic. It’s not always easy to find a list of basic synonyms for a language and, even if you find it, this doesn’t necessarily match with your contextual domain.
The term “daemon” in the domain of operating system articles is not a synonym of “devil” but it’s closer to the term “process”.
Word2Vec is a two-layer neural network that takes as input a text and outputs a vector representation for each word in the dictionary. Two words with similar meanings are identified with two vectors close to each other.
This talk explores our contribution to Apache Lucene that integrates this technique with the text analysis pipeline.
We will show how you can automatically generate synonyms on the fly from an Apache Lucene index and how you can use this new feature along with Apache Solr
Daniele Antuzi
SOFTWARE ENGINEER @ Sease
Ilaria Petreti
R&D SOFTWARE ENGINEER @ Sease