If my garage floor was as clean as my paint, I would just pick it up and continue my work.
If it was dirtier, I would slice off the part that touched the ground with a razor blade and use that for wheels and such. (This I have done)
If one is truly of the opinion that a piece of dirt from the ground in the clay does no more damage than the grit in the paint, it is hard for me to understand why they bother to clay at all.
I use a small piece of clay, about 1 ounce, and I try not to drop it.
I did drop a 4 ounce piece of clay once, and I realized my mistake.
I drop clay 3% of the time or less. I clay to make my paint perfect, and if I drop it and pick up grit, it can make my paint worse than by not claying.
As long as you keep in mind you are trying to make your paint it's best, you will "grasp" the practice of throwing away dropped clay.