The main objective of this study was to examine the effects of regional brain changes on cognitive decline and the modifying influence of vascular risk (VR) factors. We present latent difference score analyses of associations among 5-year changes in 12 regional brain volumes and age-sensitive cognitive functions in 87 adults (32 with identifiable VR factors). We found reliable individual differences in volume change for 11 of the 12 brain regions but not in the cognitive measures that showed average longitudinal decline. Thus, associations between rates of change in fluid intelligence and brain volumes could not be assessed. We observed, however, that lower levels of fluid intelligence were associated with smaller prefrontal and hippocampal volumes. Lower fluid intelligence scores were also linked to greater longitudinal shrinkage of the entorhinal cortex (EC). After accounting for the effects of age, sex, and VR factors, the orbitofrontal cortex and the prefrontal white matter (PFw) volumes as well as the 5-year change in the EC volume predicted fluid intelligence level. VR was independently associated with smaller prefrontal volumes and lower fluid intelligence. Thus, prefrontal and medial-temporal systems may play different roles in age-related differences and changes in cognitive performance.