Understanding Embeddings in the Italian Language – Part 1

Have you ever found yourself reading tons of documents related to a topic, ending up overwhelmed by the too-big load of information? Yeah, me too! Now, the next step is the following: can we develop a system capable of providing a more readable answer (for example, in a natural language fashion) for us? This could have several significant benefits to the company using this system.

First, the ease of interaction is greatly improved. When users ask a question in plain language, they expect a direct response that addresses their query, rather than having to search through multiple documents or links to find relevant information. Second, there is a notable reduction in information overload for the user. Simply presenting users with documents or links to resources can often overwhelm them, especially when the query is complex. In this case, natural-language answers make it more inclusive and user-friendly.

The main idea is to use a Large Language Model (LLM) as a kind of search engine, bypassing the need to examine various websites ranked by traditional algorithms or, even better, enhancing them. The goal is for the model to generate a natural language response directly. To achieve this, the most straightforward approach might be to rely on the assumption that the LLM has already learned the necessary information during its training. By simply providing a query describing the topic of interest, one could hope the model would infer the answer. Although this may seem like a viable alternative, there exists a phenomenon known as “hallucination”, where the model produces conceivable responses, but the content is incorrect or entirely fabricated. This occurs because LLMs generate output based on patterns learned during training.
This also means that the information learned by an LLM is fixed to the time of its training data. For instance, if we were to ask it questions about current events, it would either be unable to provide an answer or would likely generate fabricated information. As a result, they may often provide confident yet inaccurate responses, particularly when the query falls outside their knowledge base.

A potential solution to this problem is Retrieval-Augmented Generation (RAG). It combines the natural language generation capabilities of LLMs with external retrieval systems, offering a solution to some of the key limitations inherent in LLMs, particularly the issues of hallucination, incomplete or outdated information, and limited domain-specific knowledge.

In a RAG system, instead of relying solely on the pre-trained LLM parameters, which include knowledge only from its training data, the system retrieves the most relevant documents from a predefined external database. These documents are then passed as input to the LLM, ensuring that the model has access to up-to-date or domain-specific information to generate a response. This allows RAG systems to provide an additional layer of transparency by allowing users to trace the sources of the generated responses. When the LLM generates an answer based on retrieved documents, those documents can be presented alongside the response, enabling users to verify the information and build trust in the system.

A huge problem in RAG is to be able to retrieve the right documents to send as context to the LLM. The retrieval step represents a fundamental component of any RAG pipeline, as the quality of the retrieved documents directly determines the effectiveness and reliability of the generated output.

The company, Siav S.p.A., assigned me this work with the idea of starting to replace paid OpenAI’s embeddings with the goal of reducing costs and retrieving better quality results from vector search, since they had some complaints while using OpenAI’s model specifically for the Italian language.

Summing up, this study was conducted to evaluate the feasibility of replacing paid embeddings with privately fine-tuned or pre-trained models, in terms of cost, accuracy, and latency, on an Italian-only dataset. The next step will be to exploit these results for doing RAG, but this is out of scope for this work.