We examined the relationship between daily physical activity and everyday memory using an eight-day diary design with young, middle-aged, and older adults. Contrary to expectations, age differences were not reported in the frequency of memory failures and daily physical activity at the between-person level. Multilevel modeling, however, indicated that on days when adults engaged in leisure exercise (physical activity performed during leisure or free time) they reported fewer memory failures, and this was most apparent for older adults. Lagged analyses indicated that when leisure activity was reported on one day, fewer memory failures were reported the next day, and this was especially true for older adults. Thus, findings demonstrate that the benefits of physical activity for memory in later life are observable on a short-term daily basis.