To grasp AI capabilities throughout these cognitive skills, we suggest a three-stage analysis protocol that benchmarks system efficiency in relation to human capabilities:
- Consider AI techniques throughout a broad suite of cognitive duties protecting every means, utilizing held-out take a look at units to forestall information contamination
- Accumulate human baselines for a similar duties from a demographically consultant pattern of adults
- Map every AI system’s efficiency relative to the distribution of human efficiency in every means
Going from principle to observe
Defining these cognitive skills is an important first step, however we want greater than a framework to measure progress. To place this principle into observe, we’re launching a brand new Kaggle hackathon — “Measuring progress toward AGI: Cognitive abilities”. The hackathon encourages the neighborhood to design evaluations for 5 cognitive skills the place the analysis hole is the biggest: studying, metacognition, consideration, govt capabilities and social cognition.
Contributors can use Kaggle’s newly launched Community Benchmarks platform to construct and take a look at their evaluations towards a lineup of frontier fashions.
We’re providing a complete prize pool of $200,000: $10,000 awards for the highest two submissions in every of the 5 tracks, and $25,000 grand prizes for the 4 best possible total submissions. Submissions are open March 17 by April 16, and we’ll announce the outcomes June 1. Head over to the Kaggle website to start out constructing.
