Im sorry to hear you had less then perfect results...

Yeah, if your gonna use dirt, you definantly need a soft dirt, rocks are not the best thing for a surface.

I (still, and probobly always will) use dirt to clean rust off most everything, hubcaps, bike wheels, table legs, the list goes on and on. However, in your case, it sounds that the rust has been there so long that it has eaten into the metal so deep that no amount of rubbing will ever get the rust off.
Unfortunantly, in such cases the only thing to do, is use one of three methods.
1. Navel Jelly or other type of "Rust Treatment" that destroys rust. Followed by either rubbing, sealing, filling, and painting.
2. Sand Blasting
3. Simply coating with a sealer (primer, clear enamel, both are good) so no more can be eaten away. Then paint.
From your picture, it did not appear to be this bad. Sorry to have made you waste your time.

I hope they turn out nice!