EEG × ResNet-50: a time-resolved RSA pilot
A focused, end-to-end pilot asking when human EEG responses align with visual model representations. Using THINGS-EEG subject 9 and 500 images, it compares a pixel baseline, an untrained ResNet-50 and a pretrained ResNet-50 under one representational similarity analysis (RSA) framework.
Setup
- THINGS-EEG (Gifford et al., NeuroImage 2022), subject 9, preprocessed release
- Epochs from −0.2 s to 0.8 s, 17 occipital and parietal channels
- 500 training images (seed 42), 4 repetitions averaged per image
- EEG RDM at every timepoint (correlation distance), Spearman correlation against each model RDM
Peak alignment by ResNet-50 block
| Block | Untrained | Pretrained |
|---|---|---|
| block1 | 0.045 @ 120 ms | 0.090 @ 120 ms |
| block2 | 0.056 @ 120 ms | 0.106 @ 120 ms |
| block3 | 0.051 @ 110 ms | 0.090 @ 120 ms |
| block4 | 0.046 @ 110 ms | 0.058 @ 120 ms |
Findings
- Pixel similarity stays weak and flat, with no meaningful peak.
- An untrained ResNet-50 gives low, unstable alignment (about 0.04–0.05).
- A pretrained ResNet-50 peaks consistently around 120 ms after stimulus onset.
- Mid-level features (block2) align most strongly, and deeper blocks less.
It's a single-subject pilot, so the conclusions are directional rather than final.
Next steps
- Extend to all subjects
- Permutation tests with correction across time and layers
- Noise ceilings to interpret effect size
- Model families beyond ResNet-50
Code and reference
Code and full analysisA. T. Gifford, K. Dwivedi, G. Roig, R. M. Cichy, "A large and rich EEG dataset for modeling human visual object recognition", NeuroImage 264, 119754, 2022. doi:10.1016/j.neuroimage.2022.119754