The relationship between structural variants of the apolipoprotein E gene, APOE epsilon2/epsilon3/epsilon4, and dementia is well established, whereas the relationship of plasma apoE levels with dementia is less clear. Plasma apoE levels are under tight genetic control but vary widely within the various genotypes indicating that the APOE epsilon2/epsilon3/epsilon4 locus explains only a small fraction of this variation. Here we studied the association of plasma apolipoprotein E (apoE) levels with cognitive function in the elderly population at large. Within the Leiden 85-plus Study, a prospective population-based study of subjects aged 85 years, we measured plasma apoE level and genotype at base line. During a 5-year follow-up period, cognitive function was annually assessed using the Mini Mental State Examination (MMSE) and a standardized neuropsychological test battery. Among epsilon3epsilon3 carriers (n = 324), high plasma apoE levels associated with impaired global cognitive function (-1.10 points change in MMSE score per one standard deviation increase of plasma apoE level, P = 0.001), as well as lower attention (P = 0.064), speed and memory function (all P < 0.05). Adjustment for cardiovascular risk factors and exclusion of all subjects who suffered a stroke did not materially change the associations. Similar estimates were obtained in epsilon3epsilon4 carriers (n = 100), but not in epsilon2epsilon3 carriers (n = 90). We conclude that in old age, in non-epsilon2-allele carriers, high plasma apoE levels are associated with cognitive impairments, independent of genotype, cardiovascular risk factors, and stroke.