The impact of vascular health on the relations between structural brain changes and cognition was assessed in a longitudinal study of 46 adults, 23 of whom remained healthy for 5 years and 23 of whom had hypertension at baseline or acquired vascular problems during follow-up. At both measurement occasions, the volume of white matter hyperintensities (WMH) and regional brain volumes correlated with age. In 5 years, WMH volume more than doubled in the vascular risk group but did not increase in healthy participants. The frontal lobes had the highest WMH load at baseline and follow-up; the parietal WMH showed the greatest rate of expansion. In the vascular risk group, systolic blood pressure at follow-up correlated with posterior WMH volume. The fastest cortical shrinkage was observed in the prefrontal cortex and the hippocampus. Fluid intelligence correlated with WMH burden and declined along with faster WMH progression. In the vascular risk group, WMH progression and shrinkage of the fusiform cortex correlated with decline in working memory. Thus, poor vascular health contributes to age-related declines in brain and cognition, and some of the age-related declines may be limited to persons with elevated vascular risk.