Leaderboard

2022

Task 1: Difficulty Level Classification

Team AUC F1 Rank AUC Rank F1 Avg. of Ranks Overall Team Rank
RAFT 0.914 0.74 1 1 1 1
Patrick Moore 0.877 0.66 2 2 2 2
ARL-HCxS 0.855 0.60 3 3 3 3
RSAF+ 0.765 0.44 4 4 4 4
Noah Lee 0.705 0.44 5 4 4.5 5
Team59TES 0.699 0.41 6 6 6 6
Cog In The Machine 0.653 0.41 7 6 6.5 7
John O'Donnell 0.606 0.31 8 7 7.5 8
Dennis Brown 0.528 0.29 9 8 8.5 9
Hans Gevers 0.486 0.25 10 10 10 10

 

Task 2: Performance Error Prediction

Team RMSE Pearson Correlation
(PCorr)
Spearman Correlation
(SCorr)
Rank RMSE Rank PCorr Rank SCorr Avg. of Ranks Overall Team Rank
Team59TES 3377.4 0.53* 0.70* 1 1 1 1 1
RAFT 3459.7 0.48* 0.46* 2 2 2 2 2
ARL-HCxS 3930.8 0.49* 0.27* 3 2 3 2.67 3
Dennis Brown 4342.7 -0.15 -0.02 4 5 4 4.33 4
John O'Donnell 4339.02 -0.11 -0.06 5 6 4 5 6
Noah Lee 5513.1 0.18 0.03 6 4 4 4.67 5

2021

Task 1: Difficulty Level Classification

*In ties, priority is given to greater AUC rank

Team AUC F1 Rank AUC Rank F1 Avg. of Ranks Overall Team Rank
USAFA-AFIT 0.75 0.47 1 1 1 1
RAFT LLC 0.69 0.35 2 3 2.5 2
BESPIN 0.68 0.39 3 2 2.5 3
ARL 0.63 0.47 4 1 2.5 4
GMU 0.49 0.26 5 4 4.5 5
HAF/A1XD 0.48 0.25 6 5 5.5 6
PTN 0.44 0.14 7 6 6.5 7

 

Task 2: Performance Error Prediction

*Statistically significant correlation, where indicated

Team RMSE

Pearson Correlation
(PCorr)

Spearman Correlation
(SCorr)

Rank RMSE Rank PCorr Rank SCorr Avg. of Ranks Overall Team Rank
RAFT LLC 2634 0.53* 0.42* 1 1 1 1 1
GMU 3499 0.33* 0.18 4 2 3 3 2
PTN 11674 0.33* 0.19 6 2 2 3.33 3
HAF/A1XD 2868 0.06 0.13 2 4 4 3.33 3
USAFA-AFIT 2973 0.10 0.04 3 3 5 3.67 4
ARL 7560 -0.05 -0.03 5 5 6 5.33 5