Colorectal neuroendocrine carcinomas, both small cell and large cell types, are highly aggressive tumors with poor prognosis compared with colorectal adenocarcinoma. The molecular drivers of neuroendocrine carcinoma are best defined in small cell lung cancer, which shows near-universal genomic alterations in TP53 and RB1. The genetics of colorectal neuroendocrine carcinoma remain poorly understood; recent studies demonstrated infrequent RB1 alterations and genetics closely resembling colorectal adenocarcinoma. To better define the molecular pathogenesis of colorectal neuroendocrine carcinoma, we performed capture-based next-generation sequencing on 25 cases and evaluated for expression of p53, Rb, p16, and high-risk human papillomavirus (HR-HPV) subtypes using immunohistochemistry, in situ hybridization, and polymerase chain reaction. Rb/E2F pathway dysregulation was identified in nearly all cases (23/25, 92%) and occurred via three distinct mechanisms. First, RB1 genomic alteration was present in 56% (14/25) of cases and was accompanied by Rb protein loss, high p16 expression, and absence of HR-HPV; these cases also had frequent genomic alterations in TP53, the PI3K/Ras and Wnt pathways, as well as in DNA repair genes, with 4/14 cases being hypermutated. Second, 16% (4/25) of cases, all left-sided, had TP53 alteration without RB1 alteration; half of these harbored high-level amplifications in CCNE1 and MYC or MYCN and arose in patients with ulcerative colitis. Finally, 28% (7/25) of cases, all rectal or anal, lacked genomic alterations in RB1 or TP53 but were positive for HR-HPV. Our data demonstrate that Rb/E2F pathway dysregulation is essential in the pathogenesis of colorectal neuroendocrine carcinoma, akin to neuroendocrine carcinomas in other anatomic sites. Moreover, colorectal neuroendocrine carcinomas stratify into three distinct molecular subgroups, which can be differentiated based on Rb protein and HR-HPV status. HR-HPV infection represents a distinct mechanism for Rb and p53 inactivation in cases lacking genomic alterations in either gene. Differential treatment strategies for hypermutated and HPV-driven cases could improve patient outcomes.