The SARS-CoV-2 spike (S) glycoprotein contains an immunodominant receptor-binding domain (RBD) targeted by most neutralizing antibodies (Abs) in COVID-19 patient plasma. Little is known about neutralizing Abs binding to epitopes outside the RBD and their contribution to protection. Here, we describe 41 human monoclonal Abs (mAbs) derived from memory B cells, which recognize the SARS-CoV-2 S N-terminal domain (NTD) and show that a subset of them neutralize SARS-CoV-2 ultrapotently. We define an antigenic map of the SARS-CoV-2 NTD and identify a supersite (designated site i) recognized by all known NTD-specific neutralizing mAbs. These mAbs inhibit cell-to-cell fusion, activate effector functions, and protect Syrian hamsters from SARS-CoV-2 challenge, albeit selecting escape mutants in some animals. Indeed, several SARS-CoV-2 variants, including the B.1.1.7, B.1.351, and P.1 lineages, harbor frequent mutations within the NTD supersite, suggesting ongoing selective pressure and the importance of NTD-specific neutralizing mAbs for protective immunity and vaccine design.