what about if all the devices are plugged in before you start up the computer? Then what's the order?
Depends. Want me to "go long" again?

What should happen (is spec'ed to happen): Every device holds its possition relative to each other, just like if unplugged/replugged while computer on.
What really happens:

If devices already installed: acts the same as if computer was on except:
If you have the problem with switching ports like I do, you'll have pretty more the same problem I discribed before, except it sometimes will also happen if multiple devices numbered below it are replugged or unplugged at bootup, even if the buggy device is left in. Can't nail exact cause down.
If you switch the same model devices' ports, the device number (player) may or may not switch depending if the driver looks for each device's ID# or not, sort of like when on.
USB hubs
might add a twist if before shutdown the joy was installed without a hub, & before the hub (thus earlier in the USB lookup list), but on startup the joy plugged into the hub. Depends on the hub driver, as hubs should be found before other device, but I've had numbering change from moving a device like discribed.
For my sanity, I try to install, unplug, and replug USB devices only while the computer is on, unless the install manual says do it while the computer is off (never, AFAIK). And if I have a problem with USB: unplug, watch for hourglass mouse pointer, replug is one of the basic tests I do now. Not that I
never unplug while off, but I try not to.
If the device model/driver is not installed yet:
It gets placed last, like if computer on.
If multiple devices not installed at startup:

<sound="thhhhhpp"> Don't do this if you want control of how they are numbered.
If the device was installed and then unplugged before shutdown, and then plugged in before startup:
Well, I'm not sure. I try not to do this, as I've had renumbering from doing this, before, but now that I think about it, I'm not sure if it was because of my buggy driver.