Challenges

The goal of challenges is for the research community to define important problems on which multiple solutions should appear. Most challenges already include some system as a baseline and the goal is to produce a better working (e.g. more precise) system. Challenges should be self-contained – it should take a few hours for a newcomer to implement a naive solution to a challenge.

Criteria

For a problem to become a challenge, the following is needed:

• The challenge must define a task and a metric to optimize.
• The metric to optimize is challenging: the state-of-the-art does not fully solve the problem for all practical purposes and improvements are needed.
• A training dataset and an evaluation dataset is publicly available. The data should be accessible: using proprietary tools, writing emails or signing-up for services just to obtain the challenge or the data is not acceptable.
• A work in a research paper tried to solve (a variant of) the problem defined in the challenge.

Ideally a challenge is defined by its description, training and evaluation data. In case reading the data requires some special parser (or a database), this parser should be freely available. If you want to add a challenge, create a pull request here.