The JPAC only passes video. The jumpers select what video it's willing to pass. This is to avoid inadvertently sending video through to your monitor that it can't handle (which can, under some circumstances, break the monitor). The JPAC can't do any video format conversions - you have to provide video compatible with the monitor.
The stock XBox software is definitely going to run at 15kHz (standard res). Common hacks also support 31kHz (VGA - compatible with most computer monitors). 25kHz (medium res) is really only used in the arcade monitor world, so you'll definitely have to configure it do to that, if it even supports it. You'll definitely need non-standard firmware on your XBox to do this. It won't work with a normal TV or computer monitor at this point. It will only work on medium res arcade monitors (and the uber rare, typically old, computer monitors supporting 25kHz).
The image looks like exactly what I'd expect from trying to put standard res (15k) on a medium res (25k) monitor.
I have no idea if XBMC supports 25k video. I doubt it. The hardware may not even be capable of it, though it probably is. A suitable resolution would be something like 512x384 if you have that as an option.