We examined the patterns and variability of recovery post-stroke in multiple behavioral domains. A large cohort of first time stroke patients with heterogeneous lesions was studied prospectively and longitudinally at 1-2 weeks, 3 months and one year post-injury with structural MRI to measure lesion anatomy and in-depth neuropsychological assessment. Impairment was described at all timepoints by a few clusters of correlated deficits. The time course and magnitude of recovery was similar across domains, with change scores largely proportional to the initial deficit and most recovery occurring within the first three months. Damage to specific white matter tracts produced poorer recovery over several domains: attention and superior longitudinal fasciculus II/III, language and posterior arcuate fasciculus, motor and corticospinal tract. Finally, after accounting for the severity of the initial deficit, language and visual memory recovery/outcome was worse with lower education, while the occurrence of multiple deficits negatively impacted attention recovery.