日本データベース学会

dbjapanメーリングリストアーカイブ（2022年）

[dbjapan] 参加募集: NTCIR-17 FairWeb-1 Task

  • To: "dbjapan [at] dbsj.org" <dbjapan [at] dbsj.org>
  • Subject: [dbjapan] 参加募集: NTCIR-17 FairWeb-1 Task
  • From: SAKAI Tetsuya <tetsuya [at] waseda.jp>
  • Date: Wed, 28 Sep 2022 05:29:21 +0000
  • Accept-language: ja-JP, en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=waseda.jp; dmarc=pass action=none header.from=waseda.jp; dkim=pass header.d=waseda.jp; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ucb379LiDnKHplf5ZD8w6IdJCyrKUf1ott5cTluI5mE=; b=K7yDLbulDhY1/2GZIH2QtukY4ES5DAJUSaBo2jBy80JKUxUkFgl3hAzbg2rIVw9GX20gd+b5G7kdyKhCj3MpolPngfQHDmlGNLSESV8jiJqM/IDK+x4COJyEez67cYOOuN4I17N2r0TI47uPzxQE98nStyLiyq98f30xLxtSvr2+IhOCu42RpCVKOR7OlA7kjTOzAGRneAGg6NLHejQX27qdAlGertAKX3CqjqWGSNnXS+HH+NFWIw9sWnD3ynnn1q5ITUJayB6TUc69hybuMYwwbxUTe3kCOWEu7VIAA1UbcsGi/rHWFH6yHeOmqZJR0eiE8qcfhxwqO/TOUHZN/g==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TBtEgN8BTwv9mVH13Cg5YjtO1oeiu6MOWmnl8HMavUE2vshhzuD18T28AoyDdsw4t/DeYownkv/8X0y8H6t+w4NiYzIxroaDIC2Fl1ora5Fp8jJ8FW52GcW0+TwKjSQuZwjR7KsqHoRqZuRhlbldpKrH80GgxgXk4krcOeaOtRo+lHRnAUgnBT/08sYk7TmL/Zp1h+W+5tYwiJYlV2q227Eddn+1AiBspVAmUygn5tUyRy2lpgwX6oy9gyDyatjxjnZ+2zcFT8wir12h8XWvT3ConX8TblfG+tdOpx/+Bwb1aWv2zxhAbFRTnz/iZkmjLkD4Cb0Sdl0d2lTnRuCZ8g==
  • Msip_labels:
  • Thread-topic: 参加募集: NTCIR-17 FairWeb-1 Task
日本データベース学会のみなさま

早稲田大学の酒井と申します。NTCIR-17の新タスク（グループフェアネスを考慮したウェブ検索タスク）
の参加募集第一弾をお送りいたします。ご検討いただければ幸いです。

* * *


FIRST CALL FOR TASK PARTICIPATION: NTCIR-17 FairWeb-1 Task

http://sakailab.com/fairweb1/

# Overview

The FairWeb-1 task is a new English web search task that considers both relevance and group fairness.
Each of our search topic seeks information about specific entity types: R (researher), M (movies), T (Twitter accounts), and Y (YouTube contents); accordingly, we have four topic types:
R-topics (e.g., information retrieval researchers)
M-topics (e.g., Daniel Craig 007 movies)
T-topics (e.g., Twitter accounts that provide info on COVID)
Y-topics (e.g., Coldplay covers on YouTube)

For evaluating group fairness, we consider the following attribute sets containing either ordinal or nominal groups:
R-topics: h-index (ordinal) AND gender (nominal)
M-topics: #reviews on IMDb (ordinal) AND geographic region (nominal)
T-topics: #followers (ordinal)
Y-topics: #subscribers of the YouTube account (ordinal)

For R- and M-topics, we will evaluate intersectional groupfairness.
For example, for each R-topic, we want the search engine result page (SERP)
to contain information about researchers with varying levels of h-index (not just high h-index people), AND with different genders (not just "men").


# Task input/ouput

INPUT:
- a search topic (R, M, T, or Y)
- attribute set(s) and target distribution(s)

OUTPUT:
- a TREC-style run (a SERP for each topic)


# Evaluation method

We will use the Group Fairness and Relevance (GFR) framework: please visit our webpage for details.


# Timeline

October 7, 2022   Release of 1st CFP with sample topics and evaluation protocol
December 16, 2022       Pilot relevance assessments for the sample topics and a few pilot runs released; topic set size determined
Dec 19-Feb 28, 2023     Topic development
March 1, 2023     Topics released; task registrations due
May 12, 2023      Run submissions due
May 15-July 31, 2023    Entity annotations; runs evaluated
August 1, 2023    Evaluation results and draft overview released
September 1, 2023       Draft participant papers due
November 1, 2023  Camera ready papers due
December 2023     NTCIR-17@NII, Tokyo, Japan


# Inquiries:

fairweb1org [at] list.waseda.jp


# Organisers:

Sijie Tao, Nuo Chen, Tetsuya Sakai (Waseda University, Japan)
Zhumin Chu (Tsinghua University, P.R.C.)
Nicola Ferro (University of Padua, Italy)
Maria Maistro (University of Copenhagen, Denmark)
Ian Soboroff (NIST, USA)
Hiromi Arai (RIKEN AIP, Japan)