Abstract Dentate nuclei (DNs) segmentation is helpful for assessing their potential involvement in neurological diseases. Once DNs have been segmented, it becomes possible to investigate whether DNs they are microstructurally affected, through analysis of quantitative MRI parameters, such as the ones derived from diffusion weighted imaging (DWI). This study, therefore, aimed to develop a fully automated segmentation method using the non-DWI (b0) images from a DWI dataset to obtain DN masks inherently registered with parameter maps. Three different automatic methods were applied to healthy subjects in order to segment the DNs: registration to SUIT (a spatially unbiased atlas template of the cerebellum and brainstem), OPAL (Optimized Patch Match for Label fusion) and CNN (Convolutional Neural Network). DNs manual segmentation was considered the gold standard. Results show that the segmentation obtained with SUIT has an average Dice Similarity Coefficient (DSC) of 0.4907±0.0793 between the automatic SUIT masks and the gold standard. A comparison with manual masks was also performed for OPAL (DSC = 0.7624 ± 0.1786) and CNN (DSC = 0.8658 ± 0.0255), showing a better performance when using CNN. OPAL and CNN were optimised on heathy subjects’ data with high spatial resolution from the Human Connectome Project. The three methods were further used to segment the DNs of a subset of subjects affected by Temporal Lobe Epilepsy (TLE). This subset was derived from a 3T MRI research study which included DWI data acquired with a coarser resolution. In TLE dataset, SUIT performed similarly to using the HCP dataset, with a DSC = 0.4145 ± 0.1023. Using TLE data, OPAL performed worse than using HCP data: after changing the probability threshold the DSC was 0.4522 ± 0.1178. CNN was able to extract the DNs using the TLE data without need for retraining and with a good DSC = 0.7368 ± 0.0799. Statistical comparison of quantitative parameters derived from DWI analysis, as well as volumes of each DN, revealed altered and lateralised changes in TLE patients compared to healthy controls. The proposed CNN is therefore a viable option for accurate extraction of DNs from b0 images of DWI data with different resolutions and acquired at different sites.

Automatic segmentation of dentate nuclei for microstructure assessment: example of application to temporal lobe epilepsy patients