Do you wash with a grainular detergent or liquid? And do you dry the towels in the dryer?
Grainular detergents can leave behind residue on your towels and drying them in the dryer will take away some of their absorbency.
To me, these are just things to live with. You're going to see some...