As passive control techniques, both inerter device (ID) and negative stiffness device (NSD) produce forces that have a phase delay of π from common springs, i.e., negative stiffness behavior, when subjected to harmonic excitations. Thus, these two are both able to assist the motion of viscous damper in typical tuned inerter-based or negative stiffness-based absorbers, like tuned viscous mass damper (TVMD) or negative stiffness amplifying damper (NSAD), leading to a damping magnification effect. However, the negative stiffness value of NSD is frequency-independent; while that value of ID is frequency-dependent. In this regard, ID and NSD will perform differently due to structural frequency variation caused by soil-structure-interaction (SSI), which in turn further affects the energy dissipation capabilities and the seismic performance of TVMD and NSAD. Based on these effects, this paper systematically compares and discusses the optimal design and seismic performance of TVMD and NSAD from the point of SSI.