Welcome Email

Welcome to the CogPilot Data Challenge 2.0! 

The CogPilot Data Challenge 2.0 is focused on developing AI-based approaches to predict individual cognitive state and operational performance from multimodal physiological data, for assessing pilot training needs. 

To collect multimodal data during a piloting task, we created an immersive virtual reality (VR) training environment. This setup allows us to replicate a difficult aviation task while simultaneously collecting behavioral and physiological measurements. During the session, a subject performs 12 Instrumented Landing System (ILS) approaches and landings at four difficulty levels. The subject’s performance data and all physiological signals were aggregated and synchronized using the Lab Streaming Layer (LSL) to support multimodal analyses. For full details about the experiment, please refer to the background and method sections of the PhysioNet page.  

For this CogPilot Data Challenge 2.0, we have set up two tasks to predict difficulty level and the subject’s performance.  

• Challenge Task 1: Classify the difficulty level (1-4) of a flight simulation run using only physiological metrics. This task will be evaluated using F1 score and Area Under the ROC Curve (AUC) to assess classification accuracy between predicted and actual difficulty level.
• Challenge Task 2: Estimate the pilot’s Cumulative Flight Performance during a flight simulation run using only physiological metrics. This task will be evaluated using a combination of Root Mean Squared Error, Pearson Correlation, and Spearman Correlation to assess accuracy of estimated Cumulative Flight Performance values relative to actual values. 

More information regarding the Challenge tasks and adjudication process can be found on the pilotperformance.mit.edu

To get started with the CogPilot Data Challenge 2.0, the development dataset, together with data dictionary and Python starter code, can be downloaded from PhysioNet: https://doi.org/10.13026/azwa-ge48 

You are welcome to use any language or toolset you would like. However, the  CogPilot Data Challenge 2.0 is geared towards development in Python and we have created a set of Jupyter notebooks to help you get started. To start working with the Jupyter notebooks, you’ll need to have Python installed (we recommend installing Anaconda). Once you have Python and Jupyter installed, simply navigate to the directory where you have unzipped our starter kit and execute ‘jupyter notebook’ in your terminal window to get started.

Lastly, if you have any questions at all, please join us on Slack to interact with the CogPilot Team and other  participants. You will receive a separate email soon with an invitation to join the Slack channel (mit-cogpilot.slack.com).

Sincerely,  

CogPilot Team 

 

Monthly Email: November

Hello!  

Thank you for registering for the CogPilot DataChallenge 2.0! During the event, we will be sending out a monthly email covering different topics related to the Challenge.  

CogPilot DataChallenge 2.0 Schedule 

• Launch – Oct 21^st, 2022 
• Sign Up Deadline - Registration will be open through January 

• Evaluation Dataset Posted - no later than January 31^st, 2023 
• Submission Due Date - February 15^th, 2023 
• Results Posted - March 7^th, 2023 
• Closing Ceremony - Late March 2023 

Registration is open until January! If you know of anyone interested in participating, then please send them the registration form. 

Slack Channels 

We have set up a Slack channel (mit-cogpilot.slack.com) for participants to interact with the CogPilot Team and other participants. We will be posting information and answers to questions we receive throughout the Challenge.  If you did not receive an email invitation to join the channel, please let us know by responding to this email. 

Q&A 

Each month we will include questions that we receive that we would like to share with the community:  

1. Will the flight hours be provided in the evaluation set? Yes, the flight hours will be included in the eval set that will be posted by the end of January.  

• Can teams have their own slack sub-channels? Yes! Please message Christine (chriseb) directly on slack to set those up.  

• What are the ocuevts files? The ocuevts, or "oculomotor events", files are derived based on the eye tracking data. The code to show their calculation is built into the "Step 2" jupyter notebook in the starter code (link). 

If you have any questions, please don’t hesitate to reach out!  

Sincerely,  

CogPilot Team 

----------------------------------------------------------------------------- 

Please check our website (pilotperformance.mit.edu) for updates! 

 

 

Monthly Email: December