Well what you are most likely also seeing are the deeper swirls.
Light upper stuff comes out easy regardless of hardness. Deep stuff takes more work, it's a simple fact.
I know first hand to get really bad paint perfect in the compounding phase, it's not uncommon to take a second pass at the...