sometimes it goes smoothly and I can go around the hole without problems, but sometimes "something" happens and the file gets locked on the metal and I loose all control of the tool. Sometimes I cannot even place the file onto a certain part of the hole because it will not "engage" cleanly. Did I explain correctly?
this can happen with these style of tools if they are not driven fast or powerfully enough.
they are designed to be used in a die grinder, which spin in the neighborhood of 20,000 - 30,000 rpm. often with cheap electric die grinders, or air powered die grinders operated at less than optimal flow, you get too much slowdown when you apply the tool. (no high rpm end power) when it slows you get more power, but it's slower and the tool tends to dig in to the work piece.
you will notice if you look close your metal will have a slight oscillation in depth, this is what your tool is catching onto causing a vibration where no work is getting done, since the tool catches and lifts off the workpiece. when you try over and over and you get the same oscillation...
the way to fix it, is to "hit" the area hard and fast, removing the offending material on the work piece. i don't mean gouge it or lay into it, just get your tool up to speed and give it a hard bzzt! just a fraction of a second. you might need to do it 3 or 4 times if the metal of your workpiece is tough.
though very efficient, you can avoid this issue completely by using a different tool. a stone based grinder:

is allot slower, but less prone to issues such as this.
also a sandpaper type grinding wheel:

can be used for metal/wood/plastic whatever... a good all around tool.