Primary sensory cortices have been demonstrated to process sensory input from non-preferred sensory modalities, e.g. primary visual cortex reacting to auditory stimulation, bringing their presumed sensory specificity into question. Whether this reflects processing of the non-preferred stimulus per se or originates from cross-modal associations is debated. Visual/auditory objects typically have strong reciprocal associations; hence, it is difficult to address this question in these modalities. Here, we attempt to dissociate between the two competing hypotheses of whether this form of activation in primary cortices is caused by unisensory processing or cross-modal associations by turning to the olfactory system where cross-modal associations are generally weaker. Using unisensory visual and auditory objects with odor associations ranging from none to strong, we show that the posterior piriform cortex, an area known to process odor objects, is activated by both sounds and pictures of objects. Critically, this activation is independent of the objects odor associations, thereby demonstrating that the activity is not due to cross-modal associations. Using a Floyd-Warshall algorithm, we further show that the amygdala mediate condition relevant information between the posterior piriform cortex and both the auditory and visual object-oriented cortices. Importantly, we replicate past findings of clear crossmodal processing in the visual and auditory systems. Our study demonstrates processing of non-olfactory input in olfactory cortices that is independent of cross-modal associations and contributes to a more nuanced view of modality specificity in olfactory, auditory, and visual cortices.