In C. elegans nematodes, components of liquid-like germ granules were shown to be required for transgenerational small RNA inheritance. Surprisingly, we show here that mutants with defective germ granules (pptr-1, meg-3/4, pgl-1) can nevertheless inherit potent small RNA-based silencing responses, but some of the mutants lose this ability after many generations of homozygosity. Animals mutated in pptr-1, which is required for stabilization of P granules in the early embryo, display extremely strong heritable RNAi responses, which last for tens of generations, long after the responses in wild type animals peter out. The phenotype of mutants defective in the core germ granules proteins MEG-3 and MEG-4, depends on the genotype of the ancestors: Mutants that derive from maternal lineages that had functional MEG-3 and MEG-4 proteins exhibit enhanced RNAi inheritance for multiple generations. While functional ancestral meg-3/4 alleles correct, and even potentiates the ability of mutant descendants to inherit RNAi, defects in germ granules functions can be memorized as well; Wild type descendants that derive from lineages of mutants show impaired RNAi inheritance for many (>16) generations, although their germ granules are intact. Importantly, while P granules are maternally deposited, wild type progeny derived from meg-3/4 male mutants also show reduced RNAi inheritance. Unlike germ granules, small RNAs are inherited also from the sperm. Moreover, we find that the transgenerational effects that depend on the ancestral germ granules require the argonaute protein HRDE-1, which carries heritable small RNAs in the germline. Indeed, small RNA sequencing reveals imbalanced levels of many endogenous small RNAs in germ granules mutants. Strikingly, we find that hrde-1;meg-3/4 triple mutants inherit RNAi, although hrde-1 was previously thought to be essential for heritable silencing. We propose that germ granules sort and shape the RNA pool, and that small RNA inheritance memorizes this activity for multiple generations.