We examine the interplay of symmetry and topological order in $2+1$ dimensional topological phases of matter. We present a definition of the \it topological symmetry \rm group, which characterizes the symmetry of the emergent topological quantum numbers of a topological phase, and we describe its relation with the microscopic symmetry of the underlying physical system. We derive a general framework to characterize and classify symmetry fractionalization in topological phases, including phases that are non-Abelian and symmetries that permute the quasiparticle types and/or are anti-unitary. We develop a theory of extrinsic defects (fluxes) associated with elements of the symmetry group, which provides a general classification of symmetry-enriched topological phases derived from a topological phase of matter $\mathcal{C}$ with symmetry group $G$. The algebraic theory of the defects, known as a $G$-crossed braided tensor category $\mathcal{C}_{G}^{\times}$, allows one to compute many properties, such as the number of topologically distinct types of defects associated with each group element, their fusion rules, quantum dimensions, zero modes, braiding exchange transformations, a generalized Verlinde formula for the defects, and modular transformations of the $G$-crossed extensions of topological phases. We also examine the promotion of the global symmetry to a local gauge invariance, wherein the extrinsic $G$-defects are turned into deconfined quasiparticle excitations, which results in a different topological phase $(\mathcal{C}_{G}^{\times})^{G}$. A number of instructive and/or physically relevant examples are studied in detail.
