Sease at SDMX Experts workshop 2024

This talk illustrates how Large Language Models can improve user search and navigation of SDMX statistical data.
It draws its considerations from POCs designed and developed for SDMX sponsor organisations.
One important use case is to use AI for better accessibility and discoverability of data: while User eXperience techniques, lexical search improvements, and data harmonization can take organizations to a good level of accessibility, a structural (or “cognitive” gap) remains between the data user needs and the data producer constraints.
That is where AI – and most importantly, Natural Language Processing and Large Language Models – make the difference.
A natural language, conversational engine can facilitate access, navigation and filtering of statistical data, acting as an expert statistician at user’s full disposal.
The objective of the presentation is to propose a technical approach and a way forward to achieve this goal.
The key concept is to enable users to express their search queries in natural language, which the LLM then enriches, interprets, and translates into structured queries following the SDMX standard.
This approach leverages the LLM’s ability to understand the nuances of natural language and the structure of documents.
The LLM acts as an intermediary agent, offering a transparent experience to users automatically and potentially uncovering relevant documents that conventional search methods might overlook.
The presentation will include the results of this experimental work, lessons learned, best practices, and the scope of future work that should improve the approach and make it production-ready.