Epidemiological evidence shows an association between hearing loss and dementia in elderly people. However, the mechanisms that connect hearing impairments and cognitive decline are still unknown. Here we propose that a suprathreshold auditory-nerve impairment is associated with cognitive decline and brain atrophy. Methods: audiological, neuropsychological, and brain structural 3-Tesla MRI data were obtained from elders with different levels of hearing loss recruited in the ANDES cohort. The amplitude of waves I (auditory nerve) and V (midbrain) from auditory brainstem responses were measured at 80 dB nHL. We also calculated the ratio between wave V and I as a proxy of suprathreshold brainstem function. Results: we included a total of 101 subjects (age: 73.5 ± 5.2 years (mean ± SD), mean education: 9.5 ± 4.2 years, and mean audiogram thresholds (0.5–4 kHz): 25.5 ± 12.0 dB HL). We obtained reliable suprathreshold waves V in all subjects (n = 101), while replicable waves I were obtained in 92 subjects (91.1%). Partial Spearman correlations (corrected by age, gender, education and hearing thresholds) showed that reduced suprathreshold wave I responses were associated with thinner temporal and parietal cortices, and with slower processing speed as evidenced by the Trail-Making Test-A and digit symbol performance. Non-significant correlations were obtained between wave I amplitudes and other cognitive domains. Conclusions: These results evidence that reduced suprathreshold auditory nerve responses in presbycusis are associated with slower processing speed and brain structural changes in temporal and parietal regions.