Mosaic loss of Y (mLOY) is the most common somatic chromosomal alteration detected in human blood. The presence of mLOY is associated with altered blood cell counts and increased risk of Alzheimer’s disease, solid tumors, and other age-related diseases. We sought to gain a better understanding of genetic drivers and associated phenotypes of mLOY through analyses of whole genome sequencing of a large set of genetically diverse males from the Trans-Omics for Precision Medicine (TOPMed) program. This approach enabled us to identify differences in mLOY frequencies across populations defined by genetic similarity, revealing a higher frequency of mLOY in the European American (EA) ancestry group compared to those of Hispanic American (HA), African American (AA), and East Asian (EAS) ancestry. Further, we identified two genes ( CFHR1 and LRP6 ) that harbor multiple rare, putatively deleterious variants associated with mLOY susceptibility, show that subsets of human hematopoietic stem cells are enriched for activity of mLOY susceptibility variants, and that certain alleles on chromosome Y are more likely to be lost than others.