I’m of the opinion (actually, I just started to think about this) that swirl marks are not necessarily those concentric scratches we observed, but they may be related to some optical phenomena. If that is the case, they will manifest as the clear coat thickness compares with (a multiple value of) the wavelength (say 550 nm) of the visible light we use to illuminate the surface. Removing the swirl marks will then be equivalent to changing the thickness of the clear coat film in the order of few (or hundreds) of nanometers. So, if you polish the clear coat for too long you’ll be “removing” swirls in one pass, then, add them back … remove… add them periodically with subsequent passes. Therefore, apply enough swirl removers to get rid of them, and stop after achieving your objective. BTW, I’m not a professional detailer; hopefully, others, with the right expertise, will advise accordingly.