I'm sure the clay is ok. Sometimes you can perform a process on your car, and not have it deliver the expected results. Sometimes you can polish a car to remove swirls, and yet the swirls are not removed. This could be due to the product you are using, the technique you are using, or that the swirls are too deep.
To get back to your problem, you may have some bonded contamination that the claying didn't remove. Usually smeary stuff like that is some sort of tree sap or other organic fallout. It may be that it was too "stuck" or too round and your clay glided over it. Maybe you used to fine a clay, maybe too much lube to "bite", maybe this stuff is too tough, maybe you didn't do enough passes, maybe not enough pressure on the clay--lot of variables.
Sometimes sappy deposits like that can be easily removed with a solvent like IPA, sometimes an APC (dilute both to be safe). Can you feel these spots with your fingernail? Can you scrape them off (carefully)?