We aimed to characterize the nature and magnitude of cognitive decline in a group of healthy older adults with high and low levels of amyloid-β (Aβ) and who were APOE ε4 carriers and non-carriers. Healthy older adults underwent positron emission tomography neuroimaging for Aβ, APOE genotyping, and cognitive and clinical assessment as part of their baseline assessment in the Australian Imaging, Biomarker, and Lifestyle study. Cognitive function and clinical ratings were reassessed 18 months later. Linear mixed model analyses adjusted for baseline cognitive function indicated that relative to healthy older adults with low Aβ, healthy older adults with high Aβ showed greater decline in episodic memory and language at 18 months. No decline on any measure of executive function, attention, or clinical rating was observed for healthy older adults with high Aβ levels. Compared to non-carriers, APOE ε4 carriers showed a greater decline only on the task of visual memory at the 18 month assessment. Importantly though, no interaction between APOE ε4 and Aβ was observed on any measure of cognitive function. The results of this study suggest that high Aβ load was associated with greater decline in episodic memory and language, that the magnitude of this decline was moderate and equivalent across both domains, and that APOE ε4 carriage did not moderate the relationship between Aβ and decline in memory and language functions.