Personally, I would build it in such a way that would allow me first to see how good it was using the TV speakers as is! Then if it is crap, you can always follow the other route. Just seems crazy to me to jump straight into something more complicated before seeing whether or not it suits your needs!
I had a touchscreen monitor that had built in speakers. However, I didn't take my own advice. I built some crappy old PC speakers into the cab. Later, when I ended up testing the monitor speakers for something else, I realised just how good the sound was through them and that it was so much better than what I had set up!