Clonal hematopoiesis (CH) is characterized by the acquisition of a somatic mutation in a hematopoietic stem cell that results in a clonal expansion. These driver mutations can be single nucleotide variants in cancer driver genes or larger structural rearrangements called mosaic chromosomal alterations (mCAs). The factors that influence the variations in mCA fitness and ultimately result in different clonal expansion rates are not well-understood. We used the Passenger-Approximated Clonal Expansion Rate (PACER) method to estimate clonal expansion rate for 6,381 individuals in the NHLBI TOPMed cohort with gain, loss, and copy-neutral loss of heterozygosity mCAs. Our estimates of mCA fitness were correlated (R 2 = 0.49) with an alternative approach that estimated fitness of mCAs in the UK Biobank using a theoretical probability distribution. Individuals with lymphoid-associated mCAs had a significantly higher white blood cell count and faster clonal expansion rate. In a cross-sectional analysis, genome-wide association study of estimates of mCA expansion rate identified TCL1A , NRIP1 , and TERT locus variants as modulators of mCA clonal expansion rate.