In polygynous, sexually dimorphic species, sexual selection should be stronger in males than in females. Although this prediction extends to the effects of early development on fitness, few studies have documented early determinants of lifetime reproductive success in a natural mammal population. In this paper, we describe factors affecting the reproductive success of male and female red deer (Cervus elaphus) on the island of Rum, Scotland. Birthweight was a significant determinant of total lifetime reproductive success in males, with heavier-born males being more successful than lighter ones. In contrast, birthweight did not affect female reproductive success. High population density and cold spring temperatures in the year of birth decreased several components of fitness in females, but did not affect the breeding success of males. The results confirm the prediction that selection on a sexually dimorphic trait should be greater in males than in females, and explain the differential maternal expenditure between sons and daughters observed in red deer. Differences between the sexes in the effects of environmental and phenotypic variation on fitness may generate differences in the amount of heritable genetic variation underlying traits such as birthweight.