Your signal will have to go through additional switching in the hub before going to your PC, however that additional time will be much shorter than you can comprehend. I wouldn't worry about.
Control inputs are very, very, very VERY low bandwidth so you wouldn't need to worry about overloading a single USB port with lots of connected devices on hubs so long as they wern't storage devices.