Neurobiological studies of stress and cognitive aging seldom consider white matter despite indications that complex brain processes depend on networks and white matter interconnections. Frontal and temporal lobe white matter volumes increase throughout midlife adulthood in humans, and this aspect of aging is thought to enhance distributed brain functions. Here, we examine spatial learning and memory, neuroendocrine responses to psychological stress, and regional volumes of gray and white matter determined by magnetic resonance imaging in 31 female squirrel monkeys between the ages of 5 and 17 years. This period of lifespan development corresponds to the years 18-60 in humans. Older adults responded to stress with greater increases in plasma levels of adrenocorticotropic hormone and modest reductions in glucocorticoid feedback sensitivity relative to young adults. Learning and memory did not differ with age during the initial cognitive test sessions, but older adults more often failed to inhibit the initial learned response after subsequent spatial reversals. Impaired cognitive response inhibition correlated with the expansion of white matter volume statistically controlling for age, stress hormones, gray matter, and CSF volumes. These results indicate that instead of enhancing cognitive control during midlife adulthood, white matter volume expansion contributes to aspects of cognitive decline. Cellular and molecular research combined with brain imaging is needed to determine the basis of white matter growth in adults, elucidate its functions during lifespan development, and provide potential new targets for therapies aimed at maintaining in humans cognitive vitality with aging.