Low Earth orbit (LEO) satellite communication, as an essential technology in the 6 G era, still faces challenges such as high path loss, severe Doppler shifts, multi-path propagation, link budget, and limited satellite-borne resources. Recently, a novel proposed orthogonal time sequency multiplexing (OTSM) modulation that multiplexes information symbols in the delay-sequency (DS) domain performs well in high-mobility scenarios. DS-domain symbols can be transformed into the delay-time domain via the Walsh-Hadamard transform (WHT), which only includes addition and subtraction. It has been proven that OTSM can perform similarly to orthogonal time-frequency space (OTFS) with a much lower-complexity transceiver. In this paper, we derive the 2D quasi-convolution input-output (I/O) relationship of OTSM under general waveforms, reflecting the interaction between symbols and the channel. Next, we design an iterative detector for the ideal- waveform-based OTSM system based on the unitary approximate message passing (UAMP) algorithm. Specifically, based on our derived I/O relationship, we explore the structural characteristics of channels in the DS domain and design a specific unitary transformation matrix for implementing the UAMP framework, where the WHT is used to improve the computational efficiency of the detector further. Then, we extend our detection algorithm to the case of the rectangular- waveform-based OTSM system. Finally, numerical simulations demonstrate the performance advantages of our proposed detector in OTSM systems.