Endocytosis is a conserved process that mediates the internalization of nutrients and plasma membrane components, including receptors, for sorting to endosomes and the vacuole (lysosome). We combined systematic yeast genetics, high-content screening, and neural network-based image analysis of single cells to screen for genes that influence the morphology of four main endocytic compartments: coat proteins, actin patches, late endosome, and vacuole. This unbiased approach identified 17 mutant phenotypes and ~1600 genes whose perturbation affected at least one of the four compartments. Numerous mutants were associated with multiple phenotypes, indicating that morphological pleiotropy is often seen within the endocytic pathway. Morphological profiles based on the 17 aberrant phenotypes were highly correlated for functionally related genes, enabling prediction of gene function. Incomplete penetrance was prevalent, and single-cell analysis enabled exploration of the mechanisms underlying cellular heterogeneity, which include replicative age, organelle inheritance, and stress response.