01
The benchmark could recognize the hospital
I led the benchmark-audit design, evaluation protocol, calibration analysis, tooling, result synthesis, and writing. The problem we examined was a pooled evaluation that held out people but left every dataset represented in both training and testing.
That distinction matters because recording hardware, electrode arrangements, filtering, and patient groups differ across datasets. If dataset identity is associated with the diagnosis label, a model can exploit that association without learning the signal we hoped to measure.
A diagnostic null makes the confound visible: predict each dataset’s majority class without looking at EEG. On the reconstructed pooled protocol, that no-EEG baseline reaches 0.901 segment-level and 0.700 subject-level balanced accuracy, comparable to the trained headline results.
The public evaluation code contains the setup. For me, the no-EEG comparison is essential: before interpreting a high score as evidence about brain activity, I want to know how well the benchmark can be solved without looking at that activity at all.
02
Hold out the site, not only the subject
We then held out an entire dataset. Training on several sites and testing on an unseen one asks a different question from holding out people inside a familiar mixture. It is closer to the transfer problem that motivates using multiple datasets in the first place.
Under this protocol, the supervised model retains subject-level ROC-AUC of 0.763 ± 0.034 across three seeded runs. The signal is above chance, but far less deployment-ready than the pooled score implies.
03
Ranking survived better than the threshold
At a fixed 0.5 decision threshold, balanced accuracy falls to 0.585 ± 0.014 and predictions collapse toward one class. Yet the higher ROC-AUC shows that the score ordering still carries information on unseen sites.
A threshold chosen only on training sites and then transferred unchanged reaches 0.643 ± 0.034 balanced accuracy. That does not solve cross-site detection; it identifies calibration as one substantial part of the failure instead of declaring the representation empty.
I find this distinction useful because it changes what I would investigate next. Poor decisions at one threshold do not tell me, by themselves, whether the representation has lost all useful information or whether the decision rule failed to transfer.
04
Self-supervision did not rescue transfer
VICReg pretraining was tested through frozen probes, fine-tuning, and a label-budget sweep. Frozen cross-site probes reached 0.581 ± 0.002 AUC from OpenNeuro pretraining and 0.526 ± 0.003 from a disjoint TUH clinical corpus—below the supervised baseline.
The negative result is conditional: one architecture, bounded compute, and the tested data regimes. It says that scale and unlabeled data did not automatically remove this domain shift—not that self-supervision can never help EEG.
05
Why I want the audit alongside the score
This is a benchmark audit, not a clinical deployment study. The evaluation covers four public datasets and only three held-out both-class sites. It does not establish a diagnostic device or performance on future hospitals.
For teams evaluating medical models, I would want the site-prior baseline, whole-site holdouts, and the threshold-selection procedure visible alongside the headline number. I would also keep unsuccessful transfer experiments in the account. Otherwise, it is too easy to mistake a benchmark shortcut for progress toward a system that will work somewhere new.