if the metal polish won't remove the swirls, you could try the steel wool and going in one direction with long strokes, then follow it with metal polish and a mini ball attached on a drill.
I have wheels with polished lips and have used sandpaper (1000, 2000), super fine steel wool, then going...