Abstract The importance of polarization-sensitive optical coherence tomography (PS-OCT) has been increasingly recognized in human brain imaging. Despite the recent progress of PS-OCT in revealing white matter architecture and orientation, quantification of fine-scale fiber tracts in the human brain cortex has been a challenging problem, due to a low birefringence in the gray matter. In this study, we investigated the effect of refractive index matching by 2,2’-thiodiethanol (TDE) immersion on the improvement of PS-OCT measurements in ex vivo human brain tissue. We obtain the cortical fiber orientation maps in the gray matter, which reveals the radial fibers in the gyrus, the U-fibers along the sulcus, as well as distinct layers of fiber axes exhibiting laminar organization. Further analysis shows that index matching reduces the noise in axis orientation measurements by 56% and 39%, in white and gray matter, respectively. Index matching also enables precise measurements of apparent birefringence, which was underestimated in the white matter by 82% but overestimated in the gray matter by 16% prior to TDE immersion. Mathematical simulations show that the improvements are primarily attributed to the reduction in the tissue scattering coefficient, leading to an enhanced signal-to-noise ratio in deeper tissue regions, which could not be achieved by conventional noise reduction methods.