Welcome to the CogPilot Data Challenge 2.0!
The CogPilot Data Challenge 2.0 is focused on developing AI-based approaches to predict individual cognitive state and operational performance from multimodal physiological data, for assessing pilot training needs.
To collect multimodal data during a piloting task, we created an immersive virtual reality (VR) training environment. This setup allows us to replicate a difficult aviation task while simultaneously collecting behavioral and physiological measurements. During the session, a subject performs 12 Instrumented Landing System (ILS) approaches and landings at four difficulty levels. The subject’s performance data and all physiological signals were aggregated and synchronized using the Lab Streaming Layer (LSL) to support multimodal analyses. For full details about the experiment, please refer to the background and method sections of the PhysioNet page.
For this CogPilot Data Challenge 2.0, we have set up two tasks to predict difficulty level and the subject’s performance.
- Challenge Task 1: Classify the difficulty level (1-4) of a flight simulation run using only physiological metrics. This task will be evaluated using F1 score and Area Under the ROC Curve (AUC) to assess classification accuracy between predicted and actual difficulty level.
- Challenge Task 2: Estimate the pilot’s Cumulative Flight Performance during a flight simulation run using only physiological metrics. This task will be evaluated using a combination of Root Mean Squared Error, Pearson Correlation, and Spearman Correlation to assess accuracy of estimated Cumulative Flight Performance values relative to actual values.
More information regarding the Challenge tasks and adjudication process can be found on the pilotperformance.mit.edu
To get started with the CogPilot Data Challenge 2.0, the development dataset, together with data dictionary and Python starter code, can be downloaded from PhysioNet: https://doi.org/10.13026/azwa-ge48
You are welcome to use any language or toolset you would like. However, the CogPilot Data Challenge 2.0 is geared towards development in Python and we have created a set of Jupyter notebooks to help you get started. To start working with the Jupyter notebooks, you’ll need to have Python installed (we recommend installing Anaconda). Once you have Python and Jupyter installed, simply navigate to the directory where you have unzipped our starter kit and execute ‘jupyter notebook’ in your terminal window to get started.
Lastly, if you have any questions at all, please join us on Slack to interact with the CogPilot Team and other participants. You will receive a separate email soon with an invitation to join the Slack channel (mit-cogpilot.slack.com).