Serial quantitative magnetic resonance imaging (MRI) allows the detection of subtle volumetric changes in brain volume. We used serial volumetry and voxel-based difference image analysis to quantify and characterize longitudinal changes in the hippocampus, cerebellum, and neocortex in younger and middle-age individuals. Paired volumetric MRI brain scans 3.5 years apart were performed on 90 healthy subjects 14 to 77 years old. Quantitative assessment of registered images included hippocampal volumetry, cerebellar volumetry, and automatically determined regional brain volumes. Longitudinal volume changes in three age epochs (<35, 35-54, >54 years) were compared and neocortical changes beyond regions of interest were visualized using filtered difference images. Cross-sectional analysis revealed a significant association between age and reduction in all brain volumes except hippocampal volume. Changes in normalized hippocampal and white matter volume were significantly different among the three groups. Individual analysis revealed 5 subjects with significant longitudinal volume changes lying outside the normative range. Difference image analysis showed global involutional changes in the >54 age group. Our findings suggest that cross-sectional observations in intracranial volume, cerebellar volume, and gray matter volume are likely to reflect uniform rates of volume loss or secular changes. Accelerated brain atrophy was seen from the age of 35-54 and increased rates of hippocampal atrophy from the age of 54. Our findings emphasize the importance of controlling for age effects when studying pathological brain changes over a wide age range.