We assessed the validity and reliability of self-report of eye disease in participants with unilateral vision loss (presenting visual acuity worse than 6/12 in the worse eye and equal to or better than 6/12 in the better eye) or bilateral vision loss (presenting visual acuity worse than 6/12 in the better eye) in Australia's National Eye Health Survey. In total, 1738 Indigenous Australians and 3098 non-Indigenous Australians were sampled from 30 sites. Participants underwent a questionnaire and self-reported their eye disease histories. A clinical examination identified whether participants had cataract, age-related macular degeneration, diabetic retinopathy and glaucoma. For those identified as having unilateral or bilateral vision loss (438 Indigenous Australians and 709 non-Indigenous Australians), self-reports were compared with examination results using validity and reliability measures. Reliability was poor for all four diseases (Kappa 0.06 to 0.37). Measures of validity of self-report were variable, with generally high specificities (93.7% to 99.2%) in all diseases except for cataract (63.9 to 73.1%) and low sensitivities for all diseases (7.6% in Indigenous Australians with diabetic retinopathy to 44.1% of non-Indigenous Australians with cataract). This study suggests that self-report is an unreliable population-based research tool for identifying eye disease in those with vision loss.