Oracle Bone Inscriptions (OBIs) represent the early pictographic writing system of the matured Chinese civilization, documenting the history of the Shang dynasty. Deciphering them holds significant importance for unraveling the origins of civilization. Recently, an increasing number of algorithms have been proposed to assist in deciphering OBIs. However, most of these efforts have focused on the character level, thus offering limited assistance. Considering the presence of many similar components within OBI characters, associating different OBI characters with the same component will facilitate OBI decipherment. In this paper, we therefore propose a component-level OBI retrieval task, i.e., using an OBI component to retrieve all OBI characters containing this component. We accordingly collect a dataset, termed OBI component 20, containing 10,257 OBIs, which is annotated by OBI experts. Then, we propose a dual-stream attention-based model and two types of triplets based on components and characters as anchors to model the relationships between components and characters. Specifically, these two types of triplets ensure that characters containing different components are further apart, while those containing the same component are closer to the corresponding component. Experimental results demonstrate the effectiveness of our proposed model.