QPP++ 2023: Query Performance Prediction and Its Evaluation in New Tasks

co-located with 45th European Conference on Information Retrieval (ECIR)
April 6th, 2023, Hosted in Dublin, Ireland
About

About

Query Performance Prediction (QPP) is currently primarily used for ad-hoc retrieval tasks. The Information Retrieval (IR) field is reaching new heights thanks to recent advances in large language models and neural networks, as well as emerging new ways of searching, such as conversational search. Such advancements are quickly spreading to adjacent research areas, including QPP, necessitating a reconsideration of how we perform and evaluate QPP.

This workshop aims at stimulating discussion on three main aspects concerning the future of QPP:

  • What are the emerging QPP challenges posed by new methods and technologies, including but not limited to dense retrieval, contextualized embeddings, and conversational search?
  • How might these new techniques be used to improve the quality of QPP?
  • Can we claim that the current techniques for evaluating QPP are effective in all arising scenarios? Can we envision new evaluation protocols capable of granting generalizability in new domains?

We plan to foster the discussion via two focus groups led by the workshop's organizers.

The first focus group will identify what possibilities the QPP offers regarding new research models and IR tasks, primary considerations, issues linked to different aspects of the QPP, and the potentialities provided by new tools.

The second focus group will gather the community's concerns and solutions with respect to the QPP evaluation, especially for what concerns emerging domains.

Important Dates

Submission deadline: February 5th February 12th, 2023

Notification of acceptance: March 5th, 2023 March 13th, 2023

Camera ready: March 15th, 2023 March 20th, 2023

Conference days: April 3rd-5th, 2023

Workshop day: April 6th, 2023

Call for Papers

Themes and Topics

The workshop will focus on the following themes:

  • Query performance prediction applied to new tasks:
    Can existing QPP techniques be exploited, or which new QPP theories and models need to be devised for new tasks, such as passage-retrieval, Q&A, and conversational search?
  • Query performance prediction exploiting new techniques:
    How can new technologies, such as contextualized embeddings, large language models, and neural networks be exploited to improve QPP?
  • Evaluation of query performance prediction:
    How should QPP techniques be evaluated, including best practices, datasets, and resources, and, in particular, should QPP be evaluated the same for different IR tasks?

Papers

It is possible to submit three main categories of manuscripts to the workshop:

  • Full papers: up to 6 pages.
  • Short papers: up to 3 pages.
  • Discussion papers: up to 3 pages.

All manuscripts are expected to address the workshop's themes as mentioned above.

Full and short papers should contain innovative ideas and their experimental evaluation. We are also interested in works containing (methodologically sound) preliminary results and incremental endeavours.

Discussion papers should include work with or without preliminary results, position papers, and papers describing failures. Such papers should foster the discussion and thus are not required to contain full-fledged results. In this sense, the experimental evaluation of the submitted discussion paper is appreciated but not required. We are also interested in receiving contributions regarding (methodologically sound) failed experiments; since the workshop will focus on new research directions, we consider it necessary also to discuss the reasons and causes of failures.

Each manuscript will be peer-reviewed by at least two program committee members, following the single-blind paradigm.

Accepted papers will be published online as a volume of the CEUR-WS proceeding series.

Proceedings

To read the proceedings, visit the page: ceur-ws.org/Vol-3366

Organizers

Guglielmo Faggioli, University of Padova, Italy, faggiolidei.unipd.it

Nicola Ferro, University of Padova, Italy, ferrounipd.it

Josiane Mothe, Université de Toulouse, IRIT, France, josiane.motheirit.fr

Fiana Raiber, Yahoo Research, Israel, fianayahooinc.com

Program Committee

  • Negar Arabzadeh, University of Waterloo, Canada
  • Fabio Giachelle, University of Padova, Italy
  • Claudia Hauff, Spotify, Netherlands
  • Ornella Irrera, University of Padova, Italy
  • Stefano Marchesin, University of Padova, Italy
  • Jian-Yun Nie, University de Montreal, Canada
  • Haggai Roitman, eBay Research, Israel
  • Laure Soulier, Sorbonne Université-ISIR, France
  • Ellen Voorhees, NIST, US

Program

Where

Radisson Blu Royal Hotel - Field Suite

Schedule

9.00-9.10
Opening and Welcome
9.10-9.20
On correlation to evaluate QPP
Josiane Mothe
9.20-9.30
Performance Prediction for Conversational Search Using Perplexities of Query Rewrites
Chuan Meng, Mohammad Aliannejadi and Maarten de Rijke
9.30-9.40
A Large-Scale Dataset for Known-Item Question Performance Prediction
Maik Fröbe, Eric Oliver Schmidt and Matthias Hagen
9.40-9.50
On the Feasibility and Robustness of Pointwise Evaluation of Query Performance Prediction
Suchana Datta, Debasis Ganguly, Derek Greene and Mandar Mitra
9.50-10.00
A Supervised Neural Model for QPP that Combines Word Embedding Interactions and LETOR Feature Evidences
Suchana Datta, Debasis Ganguly, Josiane Mothe and Md Zia Ullah
10.00-10.10
Towards Result Delta prediction based on Knowledge Deltas for Continuous IR Evaluation
Gabriela González-Sáez, Alaa El-Ebshihy, Tobias Fink, Petra Galuščáková, David Iommi, Florina Piroi, Lorraine Goeuriot and Philippe Mulhem
10.10-10.20
Entropy-based Query Performance Prediction for Neural Information Retrieval Systems
Oleg Zendel, Binsheng Liu, J. Shane Culpepper and Falk Scholer
10.20-10.30
Focus Group(s) Organization
10.30-11.00
Coffee Break
11.00-12.10
Focus Group on how to apply and evaluate QPP on novel tasks and how to exploit novel technologies to improve QPP
12.10-12.30
Wrap-up and Closing