In the modern human genome, elevated Neanderthal ancestry is found at genes affecting keratin filaments, suggesting that gene flow with Neanderthals helped modern humans to adapt to non-African environments; deficiencies of Neanderthal ancestry are also found, particularly on the X chromosome and in genes expressed highly in testes, suggesting that some Neanderthal mutations were not tolerated on a modern human genetic background as they reduced male fertility. The modern human genome contains traces of Neanderthal ancestry. But is Neanderthal DNA distributed uniformly across the human genome, or is it concentrated more in some parts than in others? Sriram Sankararaman et al. show that parts of the human genome enriched for genes affecting keratin filaments (in hair, for example) also contain a relatively high concentration of Neanderthal DNA, suggesting that this DNA helped modern humans adapt to the chillier non-African environment. On the downside, many Neanderthal-derived alleles are associated with disease risk. Other parts of the human genome contain a deficiency of Neanderthal alleles, implying their active removal in evolution. Among the 'lost' genes are a number expressed in the testis and on the X chromosome, implying that Neanderthal DNA reduced human fertility when moved to a modern human genetic background. Genomic studies have shown that Neanderthals interbred with modern humans, and that non-Africans today are the products of this mixture1,2. The antiquity of Neanderthal gene flow into modern humans means that genomic regions that derive from Neanderthals in any one human today are usually less than a hundred kilobases in size. However, Neanderthal haplotypes are also distinctive enough that several studies have been able to detect Neanderthal ancestry at specific loci1,3,4,5,6,7,8. We systematically infer Neanderthal haplotypes in the genomes of 1,004 present-day humans9. Regions that harbour a high frequency of Neanderthal alleles are enriched for genes affecting keratin filaments, suggesting that Neanderthal alleles may have helped modern humans to adapt to non-African environments. We identify multiple Neanderthal-derived alleles that confer risk for disease, suggesting that Neanderthal alleles continue to shape human biology. An unexpected finding is that regions with reduced Neanderthal ancestry are enriched in genes, implying selection to remove genetic material derived from Neanderthals. Genes that are more highly expressed in testes than in any other tissue are especially reduced in Neanderthal ancestry, and there is an approximately fivefold reduction of Neanderthal ancestry on the X chromosome, which is known from studies of diverse species to be especially dense in male hybrid sterility genes10,11,12. These results suggest that part of the explanation for genomic regions of reduced Neanderthal ancestry is Neanderthal alleles that caused decreased fertility in males when moved to a modern human genetic background.