Multiple studies have shown loss of SARS-CoV-2 specific antibodies over time after infection, raising concern that humoral immunity against the virus is not durable. If immunity wanes quickly, millions of people may be at risk for reinfection after recovery from COVID-19. However, memory B cells (MBC) could provide durable humoral immunity even if serum neutralizing antibody titers decline. We performed multi-dimensional flow cytometric analysis of S protein receptor binding domain (S-RBD)-specific MBC in cohorts of ambulatory COVID-19 patients with mild disease, and hospitalized patients with moderate to severe disease, at a median of 54 (39-104) days after onset of symptoms. We detected S-RBD-specific class-switched MBC in 13 out of 14 participants, including 4 of the 5 participants with lowest plasma levels of anti-S-RBD IgG and neutralizing antibodies. Resting MBC (rMBC) made up the largest proportion of S-RBD-specific class-switched MBC in both cohorts. FCRL5, a marker of functional memory when expressed on rMBC, was dramatically upregulated on S-RBD-specific rMBC. These data indicate that most SARS-CoV-2-infected individuals develop S-RBD-specific, class-switched MBC that phenotypically resemble germinal center-derived B cells induced by effective vaccination against other pathogens, providing evidence for durable B cell-mediated immunity against SARS-CoV-2 after recovery from mild or severe COVID-19 disease.