Changes in memory function in elderly individuals are often attributed to dysfunction of the prefrontal cortex (PFC). One mechanism for this dysfunction may be disruption of white matter tracts that connect the PFC with its anatomical targets. Here, we tested the hypothesis that white matter degeneration is associated with reduced prefrontal activation. We used white matter hyperintensities (WMH), a magnetic resonance imaging (MRI) finding associated with cerebrovascular disease in elderly individuals, as a marker for white matter degeneration. Specifically, we used structural MRI to quantify the extent of WMH in a group of cognitively normal elderly individuals and tested whether these measures were predictive of the magnitude of prefrontal activity (fMRI) observed during performance of an episodic retrieval task and a verbal working memory task. We also examined the effects of WMH located in the dorsolateral frontal regions with the hypothesis that dorsal PFC WMH would be strongly associated with not only PFC function, but also with areas that are anatomically and functionally linked to the PFC in a task-dependent manner. Results showed that increases in both global and regional dorsal PFC WMH volume were associated with decreases in PFC activity. In addition, dorsal PFC WMH volume was associated with decreased activity in medial temporal and anterior cingulate regions during episodic retrieval and decreased activity in the posterior parietal and anterior cingulate cortex during working memory performance. These results suggest that disruption of white matter tracts, especially within the PFC, may be a mechanism for age-related changes in memory functioning.