Accurate identifications of ligand binding sites (LBS) on protein structure is critical for understanding protein function and designing structure-based drug. As the previous pocket-centric methods are usually based on the investigation of pseudo surface points (PSPs) outside the protein structure, thus inherently cannot incorporate the local connectivity and global 3D geometrical information of the protein structure. In this paper, we propose a novel point clouds segmentation method, PointSite, for accurate identification of protein ligand binding atoms, which performs protein LBS identification at the atom-level in a protein-centric manner. Specifically, we first transfer the original 3D protein structure to point clouds and then conduct segmentation through Submanifold Sparse Convolution (SSC) based U-Net. With the fine-grained atom-level binding atoms representation and enhanced feature learning, PointSite can outperform previous methods in atom-IoU by a large margin. Furthermore, our segmented binding atoms can work as a filter on predictions achieved by previous pocket-centric approaches, which significantly decreases the false-positive of LBS candidates. Through cascaded filter and re-ranking aided by the segmented atoms, state-of-the-art performance can be achieved over various canonical benchmarks and CAMEO hard targets in terms of the commonly used DCA criteria. Our code is publicly available through https://github.com/PointSite.