I think the guy kinda forgot what the joke was. His original concept -- a full-size guy part of an official midget-fighting sport -- is really funny. But once he got into his thing the joke wasn't about midgets at all, we were supposed to laugh at how stupid his karate moves were which, unfortunately, they weren't very funny.
You win some, you lose some, I suppose.