The link between genomic structure and biological function is yet to be consolidated, it is, however, clear that physical manipulation of the genome, driven by the activity of a variety of proteins, is a crucial step. To understand the consequences of the physical forces underlying genome organization, we build a coarse-grained polymer model of the genome, featuring three fundamentally distinct classes of interactions: lengthwise compaction, i.e., compaction of chromosomes along its contour, self-adhesion among epigenetically similar genomic segments, and adhesion of chromosome segments to the nuclear envelope or lamina. We postulate that these three types of interactions sufficiently represent the concerted action of the different proteins organizing the genome architecture and show that an interplay among these interactions can recapitulate the architectural variants observed across the tree of life. The model elucidates how an interplay of forces arising from the three classes of genomic interactions can drive drastic, yet predictable, changes in the global genome architecture, and makes testable predictions. We posit that precise control over these interactions in vivo is key to the regulation of genome architecture.