You should use the crimp that's designed for the mating part. My panel shares a mix of both and I make the loom to suit. They should be a bit of a pain to pull apart if they are a good fit.
As far as bare crimps are concerned, as long as they are all snug, and they aren't touching other terminals you'll be fine. Because we're only talking about 5v here, if a crimp comes off and touches another, worst case is it will cause strange behaviour. But having one drag itself across your keyboard encoder, now that's a different matter...