Abstract How a certain ground state of complex physical systems emerges, especially in two-dimensional materials, is a fundamental question in condensed-matter physics. A particularly interesting case is systems belonging to the class of XY Hamiltonian where the magnetic order parameter of conventional nature is unstable in two-dimensional materials leading to a Berezinskii−Kosterlitz−Thouless transition. Here, we report how the XXZ-type antiferromagnetic order of a magnetic van der Waals material, NiPS 3 , behaves upon reducing the thickness and ultimately becomes unstable in the monolayer limit. Our experimental data are consistent with the findings based on renormalization-group theory that at low temperatures a two-dimensional XXZ system behaves like a two-dimensional XY one, which cannot have a long-range order at finite temperatures. This work provides the experimental examination of the XY magnetism in the atomically thin limit and opens opportunities of exploiting these fundamental theorems of magnetism using magnetic van der Waals materials.

Suppression of magnetic ordering in XXZ-type antiferromagnetic monolayer NiPS3