Studies on the determinants of physical activity have traditionally focused on social factors and environmental barriers, but recent research has shown the additional importance of biological factors, including genetic variation. Here we review the major tenets of this research to arrive at three major conclusions: First, individual differences in physical activity traits are significantly influenced by genetic factors, but genetic contribution varies strongly over age, with heritability of leisure time exercise behavior ranging from 27% to 84% and heritability of sedentary behaviors ranging from 9% to 48%. Second, candidate gene approaches based on animal or human QTLs or on biological relevance (e.g., dopaminergic or cannabinoid activity in the brain, or exercise performance influencing muscle physiology) have not yet yielded the necessary evidence to specify the genetic mechanisms underlying the heritability of physical activity traits. Third, there is significant genetic modulation of the beneficial effects of daily physical activity patterns on strength and endurance improvements and on health-related parameters like body mass index. Further increases in our understanding of the genetic determinants of sedentary and exercise behaviors as well as the genetic modulation of their effects on fitness and health will be key to meaningful future intervention on these behaviors.