The increase of image information brings the need for secure storage and management, and people are used to uploading images to cloud servers for storage, but the issue of privacy management and protection has become a great challenge because images may contain some sensitive information. To solve this problem, this paper proposes a novel secure and efficient fine-grained TPE scheme (FG-TPE), specifically, the image pixels are firstly divided into blocks, and multiple rounds of neighboring pixel substitution and permutation fine-grained encryption operations are performed in each block to achieve obfuscated protection of sensitive feature information of the image. Then, the state transfer process of image pixel encryption is reduction to the adversarial detection in a stochastic environment, and the optimal encryption rounds bounds are found by Kalman filtering method. Finally, experiments conducted on two face datasets show that, in qualitative and quantitative comparisons, the average encryption time is decreased remarkably, improved encryption efficiency, and the ciphertext expansion rate is reduced by 19.6% on average, possessing a better image spatiality when compared to the state-of-the-art approaches. Excellent resistance to AI restoration performance has been achieved with only 16 × 16 divided block encryption, and face detection recognition has been fully defended against 32 × 32 divided block encryption, achieving a balance between privacy security and usability management of image information.