The aim of this study is to investigate the relationships between road traffic noise exposure, annoyance caused by different noise sources and validated health indicators in a cohort of 1375 adults from the region of Basel, Switzerland. Road traffic noise exposure for each study participant was determined using modelling, and annoyance from various noise sources was inquired by means of a four-point Likert scale. Regression parameters from multivariable regression models for the von Zerssen score of somatic symptoms (point symptom score increase per annoyance category) showed strongest associations with annoyance from industry noise (2.36, 95% CI: 1.54, 3.17), neighbour noise (1.62, 95% CI: 1.17, 2.06) and road traffic noise (1.53, 95% CI: 1.09, 1.96). Increase in modelled noise exposure by 10 dB(A) resulted in a von Zerssen symptom score increase of 0.47 (95% CI: -0.01, 0.95) units. Subsequent structural equation modelling revealed that the association between physical noise exposure and health-related quality of life (HRQOL) is strongly mediated by annoyance and sleep disturbance. This study elucidates the complex interplay of different factors for the association between physical noise exposure and HRQOL.