While recent work has revealed how different inhibitory interneurons influence cortical responses to sensory stimuli, little is known about how their activity contributes to sensory perception. Here, we optogenetically stimulated different genetically defined interneurons (parvalbumin (PV), somatostatin (SST), vasoactive intestinal peptide (VIP)) in visual cortex (V1) of mice working at threshold in contrast increment or decrement detection tasks. The visual stimulus was paired with optogenetic stimulation to assess how enhancing V1 inhibitory neuron activity synchronously during cortical responses altered task performance. PV or SST activation impaired, while VIP stimulation improved, contrast increment detection. Notably, PV or SST stimulation also impaired contrast decrement detection, when opsin-evoked inhibition would exaggerate stimulus-evoked decrements in firing rate, and thus might improve performance. The impairment produced by PV or SST stimulation persisted throughout many weeks of testing. In contrast mice learned to reliably detect VIP activation in the absence of natural visual stimulation. Thus, different inhibitory signals make distinct contributions to visual contrast perception.