Research digest · MICCAI 2026 · AMAI Workshop

A Benchmark Audit of Site Confounds

Calibration and Self-Supervision in Cross-Dataset Parkinson’s EEG Detection

Cross-dataset Parkinson’s EEG detection can look clinically impressive while measuring dataset identity. This work separates site confounding, cross-site discrimination, threshold calibration, and the limits of self-supervised pretraining.

Accepted full paper · poster

Edward’s role: First author
Status: Accepted · MICCAI 2026 Workshop AMAI
Authors: Edward Lue Chee Lip · Van Mai · Saanvi Neema · Alexander Jameson · Karolina Torbus · Jithin Suresh
Artifacts: Code and evaluation tooling public
Reading time: 5 min read
Contribution: First-author benchmark-audit design, evaluation protocol, calibration analysis, tooling, result synthesis, and writing.
Claim state: E2 · accepted; artifact-backed
Last checked: 13 Aug 2026
Provenance: Official OpenReview decision evidence and the current public code/evaluation artifacts. Private reviews and ratings are excluded.
Citation: Workshop proceedings citation pending.

One result · visually

Evidence before interpretation.

A no-EEG site baseline matches the pooled headline; honest site holdout exposes the calibration gap.

Pooled protocol · site prior only, no EEG0.901balanced accuracy

Unseen site · supervised ranking (LODO)0.763ROC-AUC

Unseen site · decision, fixed threshold0.585balanced accuracy

Unseen site · decision, train-site threshold0.643balanced accuracy

Hospital identity can look like disease signal. The split and the threshold are part of the claim.

The same object · seven altitudes

From question
to falsifier.

A paper becomes easier to inspect when its claim, boundary, intervention, measurement, failure mode, and next disconfirming result are kept separate.

01Question
Does a pooled cross-dataset score measure Parkinson’s signal, or does it reward recognizing which dataset produced the EEG?
02Claim
The standard pooled protocol is strongly influenced by site-associated label imbalance; honest cross-site discrimination exists, but it is modest and calibration-bound.
03System boundary
Four public EEG datasets, their site-specific acquisition pipelines and label distributions, a TransformEEG encoder, and the evaluation protocol connecting them.
04Intervention
Add a no-EEG site-prior null, hold out an entire dataset with LODO, separate ranking from threshold choice, and test SSL under the same cross-site protocol.
05Measurement
Pooled balanced accuracy, subject-level LODO ROC-AUC, fixed and training-site-selected thresholds, three seeded runs, and explicit site-level limitations.
06Failure mode
A model can score highly by learning dataset identity; a fixed 0.5 threshold can then hide above-chance ranking when deployment shifts to an unseen site.
07Next falsifier
Replication over more independent hospitals, acquisition systems, populations, and architectures—where the site-prior null no longer explains the headline score.

The benchmark could recognize the hospital

The common pooled protocol holds out subjects but leaves every dataset represented in both train and test folds. Because acquisition hardware, montage, filtering, and cohort composition differ by dataset, the model can learn where a recording came from—and site is correlated with the label.

A diagnostic null makes the confound visible: predict each dataset’s majority class without looking at EEG. On the reconstructed pooled protocol, that no-EEG baseline reaches 0.901 segment-level and 0.700 subject-level balanced accuracy, comparable to the trained headline results.

Hold out the site, not only the subject

Leave-one-dataset-out evaluation trains on several sites and tests on a site the model has never seen. That changes the scientific object from interpolation within a mixed benchmark to transfer across acquisition and population shift.

Under this protocol, the supervised model retains subject-level ROC-AUC of 0.763 ± 0.034 across three seeded runs. The signal is above chance, but far less deployment-ready than the pooled score implies.

Ranking survived better than the threshold

At a fixed 0.5 decision threshold, balanced accuracy falls to 0.585 ± 0.014 and predictions collapse toward one class. Yet the higher ROC-AUC shows that the score ordering still carries information on unseen sites.

A threshold chosen only on training sites and then transferred unchanged reaches 0.643 ± 0.034 balanced accuracy. That does not solve cross-site detection; it identifies calibration as one substantial part of the failure instead of declaring the representation empty.

Self-supervision did not rescue transfer

VICReg pretraining was tested through frozen probes, fine-tuning, and a label-budget sweep. Frozen cross-site probes reached 0.581 ± 0.002 AUC from OpenNeuro pretraining and 0.526 ± 0.003 from a disjoint TUH clinical corpus—below the supervised baseline.

The negative result is conditional: one architecture, bounded compute, and the tested data regimes. It says that scale and unlabeled data did not automatically remove this domain shift—not that self-supervision can never help EEG.

What the result permits

This is a benchmark audit, not a clinical deployment study. The evaluation covers four public datasets and only three held-out both-class sites. It does not establish a diagnostic device or performance on future hospitals.

The defensible recommendation is methodological: report a site-prior null, hold out whole sites, separate discrimination from calibration, and keep negative transfer results visible. A medical score should not become a clinical claim before its acquisition shortcuts have been tested.