Thanks guys for the input, and the timing is just perfect as last weekend I redid this but am still not sure if it will hold.
My fix at the moment was to use the other pololu mount as you get two and to tap a slightly bigger grub screw into it, but the cotter pin idea would of been much better and I have plenty here from bicycle breaks.
If that fails then I now have some other ideas mostly gleamed from using the search term "6mm shaft coupling" that brought up items like
this aswell as other items used in CNC`s.
Infact as an experiment I think I will order
this right now just to see how suitable it is.
@GoYouSalukis I really like that bearing so may well go for it!, and Le Chuck my bearing is just a generic lazy susan from the bay and would work just fine if it was flat - just not at an angle.
Not sure what a sleuth bearing is and my google skills are failing me.
On a side note the VEX site from the turntable link has some fun looking toys in it.