Knowledge Delta based improvement and continuous evaluation of retrieval engines
Welcome to the web site dedicated to the Kodicare project, funded by French ANR and Austrian FWF.
Overview
Evaluating search systems requires setting up an environment: select a paradigm, metrics, a dataset, etc. The choice of an environment is rarely motivated objectively, and the impact of its variations (choosing a dataset against another, altering one) is rarely measured. Such objectivity comes from a quantifiable understanding of the differences between datasets, documents, or test queries. In Kodicare, we generically call such difference “knowledge delta”. Evaluation of several environments, knowing their knowledge deltas, leads to measuring and qualifying “results deltas”. Online systems require continuous evaluation with a stable and meaningful environment; which guarantees the reproducibility and explainability of systems results. The environment and result deltas will be able to support such continuous evaluation, and to provide explanations. The theoretical results will be confronted to real cases defined by a French company that deploys a web search engine (Qwant). Scientific and technical challenges: To our knowledge, no such framework dedicated to real continuous evaluation of information retrieval systems exist, due to the numerous parameters that must be handled. The deltas proposed by Kodicare are then a sensible way to tackle this problem. Continuous evaluation is only possible with real cases, which are often difficult to define without the help of web search companies. The large implication of Qwant will help the project define usable proposals.
Expected results
- The innovative theoretical solution explored by this project is to define in a common framework “knowledge delta” and “result deltas”, and quantify them such that: results are comparable over the time (like a regression test) and the search engine adapts to changes in user behaviour and information need;
- A transversal focus of the project will explore the transparency that such a continuous evaluation system must support;
- New evaluation paradigm with a potential impact on many connected communities.