Objective To compare cortical dipole fitting spatial accuracy between the widely used yet highly simplified 3-layer and modern more realistic 5-layer BEM-FMM models with and without adaptive mesh refinement (AMR) methods. Methods We generate simulated noiseless 256-channel EEG data from 5-layer (7-compartment) meshes of 15 subjects from the Connectome Young Adult dataset. For each subject, we test four dipole positions, three sets of conductivity values, and two types of head segmentation. We use the boundary element method (BEM) with fast multipole method (FMM) acceleration, with or without (AMR), for forward modeling. Dipole fitting is carried out with the FieldTrip MATLAB toolbox. Results The average position error (across all tested dipoles, subjects, and models) is ∼4 mm, with a standard deviation of ∼2 mm. The orientation error is ∼20° on average, with a standard deviation of ∼15°. Without AMR, the numerical inaccuracies produce a larger disagreement between the 3- and 5-layer models, with an average position error of ∼8 mm (6 mm standard deviation), and an orientation error of 28° (28° standard deviation). Conclusions The low-resolution 3-layer models provide excellent accuracy in dipole localization. On the other hand, dipole orientation is retrieved less accurately. Therefore, certain applications may require more realistic models for practical source reconstruction. AMR is a critical component for improving the accuracy of forward EEG computations using a high-resolution 5-layer volume conduction model. Significance Improving EEG source reconstruction accuracy is important for several clinical applications, including epilepsy and other seizure-inducing conditions.