MUMIA Mtg on Verifyable Results in Multi-lingual/Multi-faceted Search: Challenges in Sharing Data, Tools and Results

You are here

07 February 2014 1163 reads

TUESDAY 25TH MARCH - 09:00 – 17:00 


One of the goals of MUMIA is to establish solid, verifiable and comparable evaluations of research in multilingual/multifaceted search. These rely on the availability of benchmark data, comparable and repeatable processes. Yet, there are a range of challenges that inhibit progress in this domain, amongst which are, for example:

  • legal aspects of sharing benchmark data and evaluations
  • size aspects: how to host, distribute, provide access to TBs of data
  • metadata issues: how to annotate and describe benchmark data, its collection principles, underlying bias, etc.
  •  repeatability:

o   how to identify/cite in a machine-processable way the exact subset (out of a huge, growing database of benchmark data) that was used in a specific study

o   how to share not only data and results, but also the tools used in the (pre-)processing of data and results for comparability and repeatability of results

  •  costing issues: what are the financial implications, cost and business models for hosting/provisioning of such benchmark data collections

The meeting is inspired by articles such as:

The meeting will comprise of a small number of short focused presentations highlighting challenges in different benchmarking/evaluation settings, followed by a world-cafe style sessions to brainstorm and elaborate means to address selected sub-problems, followed by break-out and reporting-back sessions. 


Target Audience: (approx. 25 participants) Half of these are experts in Information Retrieval benchmarking invited by MUMIA, the other half of spaces is open to participants at the RDA plenary working in the area of Information Retrieval who want to attend this workshop. Non-MUMIA members should express their interest in participating by sending an email to  rauber[at]  including a brief statement of their areas of interest and involvement in the area.

This pre-registration is required as places are limited and we would like to distribute some  material up-front as well as consider topic suggestions for specific world cafe sessions by participants.