The degree to which individual variation in brain structure in humans is genetically or environmentally determined is as yet not well understood. We studied the brains of 54 monozygotic (33 male, 21 female) and 58 dizygotic (17 male, 20 female, 21 opposite sex) pairs of twins and 34 of their full siblings (19 male, 15 female) by means of high resolution magnetic resonance imaging scans. Structural equation modeling was used to quantify the genetic and environmental contributions to phenotypic (co)variance in whole brain, gray and white matter volume of the cerebrum, lateral ventricle volume and associated variables such as intracranial volume and height. Because the cerebral cortex makes up more that two-thirds of the brain mass and almost three-quarters of its synapses, our data predominantly concerns the telencephalon. Genetic factors accounted for most of the individual differences in whole brain (90%), gray (82%) and white (88%) matter volume. Individual differences in lateral ventricle volume were best explained by a model containing common (58%) and unique (42%) environmental factors, indicating genes to be of no or minor influence. In our sample, genetic or environmental influences were not different for males and females. The same genes influenced brain volumes and intracranial volume and almost completely explained their high phenotypic correlation. Genes influencing gray and white matter overlapped to a large extent and completely determined their phenotypic correlation. The high heritability estimates that were found indicate that brain volumes may be useful as intermediate phenotypes in behavioral genetic research.