This guide sets out the rationale behind using artificial intelligence (AI) technologies in the Recherche Data Gouv catalogue and the methods used to implement these. The growing volume of scientific data and the varying quality of documents of this kind now represent a significant challenge for researchers who wish to identify relevant information rapidly. This means they require sophisticated methods for searching for and retrieving information. In this context, traditional search methods based on keyword matching are proving increasingly limited.

Introduction

The Recherche Data Gouv ecosystem offers researchers sovereign solutions for sharing, structuring, preserving and opening up their data. The national catalogue plays a central role in this system by harvesting and interlinking metadata from various institutional and thematic repositories, thereby creating an interdisciplinary one-stop shop. The aim of this catalogue is to enable the discovery of datasets across all disciplines, regardless of thematic areas.

AI in data exploration

Traditional keyword-based searching is energy-efficient in terms of computing time. However, it often fails to capture the conceptual relationships between scientific terms expressed using heterogeneous terminology. This limitation is particularly pronounced when working with multilingual datasets (typically in French or English for Recherche Data Gouv) where the same scientific concept may be described using different words depending on the language used.

Semantic searches offer a powerful alternative as they provide a comprehension of the meaning behind queries which enables the discovery of relevant information even if exact matches are not found. The use of text embedding models is the cornerstone of this approach. These models transform text into dense numerical vectors encoding semantic meaning which enable the mathematical comparison of textual content to facilitate the identification of semantically similar documents or passages.

Keyword Search vs Semantic Search

The fundamental difference between semantic searches and traditional keyword searches lies in their approach to understanding user queries and document content.

Keyword search works by matching the exact words in a query with those found in documents. The limits of this method derive from its inability to recognise synonyms, related concepts or the underlying meaning of a text.

Conversely, semantic search uses vector representations that are generated by embedding models to focus on the actual meaning of the query. Comparing the distance between a query’s embedding and those of the documents means the system can identify relevant results even if the exact keywords are not there.

As an example, in the context of scientific datasets, a user could search for “gene expression analysis” and a keyword search would only return documents that contain that exact phrase. A semantic search would also retrieve documents that deal with “RNA sequencing data” because this function recognises these terms’ strong conceptual relationship in the field of molecular biology.

The fundamental principles of text embeddings for semantic search 

To understand semantic search, embedding needs to be thought of as a ‘translator of meaning’. This process captures the meaning of a word, sentence, paragraph or entire document and converts it into a vector of N real numbers, typically representing around 3,000 dimensions.

To draw a simple analogy, let’s take the example of coordinates on a map. If two points have similar coordinates on this map, they are geographically close to one another. Similarly, in the catalogue’s vector space, two documents whose vectors are close to one another are semantically very closely related.

The main advantage of embedding models is their efficiency. Each vector is only calculated once when it is added to the catalogue and the search method just involves measuring the distance between the query and items in the catalogue. This is, of course, very computationally efficient but effective semantic search using embeddings is not the most accurate method. So, to optimise results, it is sometimes necessary to supplement semantic searches with more sophisticated re-ranking methods which is where the cross-encoder comes in.

Re-ranking with the cross-encoder

The very high accuracy in assessing relevance of the cross-encoder is its major advantage. Conversely, the disadvantage lies in its high resource consumption, as, unlike simple embeddings, it has to calculate a score by cross-comparing the user’s query and the entirety of the dataset’s metadata in pairs.

The Recherche Data Gouv catalogue contains over 100,000 datasets, so it would be technically impractical to perform this calculation on the whole database in real time. This is why an optimisation strategy has been implemented which restricts the cross-encoder calculation exclusively to the 1,000 datasets identified in advance as being the most semantically similar (via embeddings). Furthermore, this computational cost means the feature is not applied systematically and is instead offered as an option to users who wish to obtain more relevant results, albeit at an increased computational cost.

Hybrid searches with facets

The catalogue provides a hybrid search system that combines the power of the semantic approach with the efficiency of traditional facets to provide a comprehensive search function. Semantic search excels at understanding the overall intent behind a query but using structured filters is extremely practical for day-to-day users and also avoids users getting bogged down in an overwhelming volume of results.

This hybrid approach means users can freely combine their semantic text search with these key catalogue facets:

Publication date

Affiliations

Authors

Scientific field (aligned with the HCERES thesaurus)

The plan is for this list is to be gradually expanded to include other filtering criteria considered essential to the scientific community, like funding bodies, licence type, and so forth.

From a technical standpoint, this synergy is based on the usage of a Typesense database which natively manages and optimises this hybrid behaviour. Typesense enables instantaneous interweaving of semantic vector comparison and strict filtering by attributes, thus providing response times of a fraction of a second. For users, this translates into a wholly new search experience that combines the relevance of semantic AI with the strict precision of traditional document filters.

A summary of architectural choices

To sum up, the approach adopted in the Recherche Data Gouv catalogue is:

Firstly, all harvested documents are analysed by an embedding model which positions all of them within a semantic vector space. These are lightweight calculations that are only performed once, with the resulting vectors are stored in a database.

When a user submits a query, it is converted into an embedding. The system measures the distance between this query vector and those of the documents before returning the results ranked from closest to furthest. The system arbitrarily returns the 1,000 closest datasets. In this way, there is only one embedding calculation and one nearest-neighbour search.

If the initial results turn out not to be entirely satisfactory, the cross-encoder option enables a more in-depth bilateral comparison to be carried out on the top 1,000 to refine the relevance of the ranking. Of course, this option involves a computational cost and, therefore, an impact on energy consumption which means it should only be prioritised if the initial results are considered unsatisfactory.

It is important to note that large language models (LLMs) are not used directly for the search function in production. This decision was driven by environmental sustainability requirements. Also, this requirement is effectively met by the integration of a Model Context Protocol (MCP) server which can be activated on demand by directly drawing on the users’ own LLM resources.

Metadata alignment and curation 

Beyond the search algorithms, the usability of the catalogue depends fundamentally on the quality of its metadata but analysis of the harvested data has pinpointed a significant challenge, namely the extreme variety of information that users enter manually. This metadata is rarely aligned with controlled vocabularies and suffers from a high degree of heterogeneity involving issues like typos, local acronyms, textual abbreviations or variable syntax.

In practical terms, this inconsistency can mean that it is sometimes almost more difficult for a user to navigate these metadata fields than the datasets themselves. This is why a retrospective alignment process has been implemented across two major projects to simplify the user experience and ensure a functional catalogue. Each field that has been enriched or generated by AI is explicitly flagged in the catalogue’s human-computer interface (HCI) In the interests of transparency and to foster user confidence.

 

The alignment of affiliations with the ROR repository

Author affiliations represent essential metadata for the identification of where a dataset originates from but this data is often presented as a block of plain text, with no standardised format. This is why a hybrid method based on large language models (LLMs) is used to aligned affiliations with the ROR.

Alignment methodology:

Raw affiliations are extracted from the harvested metadata. In parallel, the ROR’s 120,000+ entries are stored locally in an SQLite database that is interconnected with the catalogue to avoid repeated external API calls, thus ensuring greater reliability and rapidity.

For each raw affiliation, an exact match is searched for in the ROR SQLite database. If a match is found, the official name from the ROR is inserted into the ‘Affiliation’ field.

If there is no direct match, an LLM is called upon in the background to break down the complex structures of intertwined affiliations. The isolated entities are then cleaned up and structured into a JSON file so the SQLite search process can then be reapplied to them. In the event of persistent failures of this kind, the history is logged to improve the process.

The alignment is submitted to a second LLM model that acts as a referee. If validation is rejected, the metadata added at this stage are immediately removed to preserve the quality of the catalogue.

Disciplinary categorisation via the HCERES thesaurus

To enable efficient and standardised thematic exploration, the text of the main metadata (title, abstract and keywords) has been automatically aligned with the HCERES thesaurus. This enrichment means users can reliably search for and filter datasets using a unified disciplinary domain criterion, regardless of the original terminology.

Aims and future prospects

These initial retrospective harmonisation projects have laid the foundations for a fully interconnected catalogue. The aim for the future stages of the project is to roll out a fully automated alignment system that is extended to include other reference thesauri along with complex ontologies to categorise datasets in even greater detail and enhance their semantic interoperability.

The models used and local execution

To work effectively on scientific corpora made up of titles and abstracts written in French and English, the models selected need to capture cross-linguistic nuances and the highly specialised terminology of scientific literature.

In compliance with a philosophy of eco-design, sovereignty and independence, the catalogue is exclusively based on open models, or 'open weights', that are precisely scaled to respond effectively to requirements. Furthermore, we draw on the ILAAS initiative for the local and shared execution of LLMs, embeddings and cross-encoders.

This infrastructure means the following models can be run optimally and operationally:

BGE-M3 which is used for generating semantic embeddings and for cross-encoding tasks.

Mistral-Small and Mistral-Medium which are used for back-office analysis, disaggregation and validation (LLM judge) tasks.

Energy consumption and environmental impact 

In compliance with best practices for the eco-design of digital services, the energy footprint of the AI processing in the catalogue is strictly monitored as follows:

Embedding calculations on harvested metadata and alignments are carried out once in the back office and stored in a MongoDB database.

On the front-end, embedding generation and cross-encoder analysis of the top 1,000 results are carried out in a fraction of a second per user query. These calculations are performed on very modest hardware and only consume a few watts.

The use of LLMs is prohibited in real-time browsing, with these solely used upstream, during the information preparation and curation quality analysis phases which run in the background.