i have a 360, and the following music games:
rock band
rock band 2
Guitar Hero 3
Guitar Hero World Tour
Guitar Hero Metallica
Rock Band 2 is the only one that gets played...the gameplay is fantastic, the set list is pretty good, and the ammount of songs available is insane. The selection of downloadable content for guitar hero is laughable at best. I do have to agree that the guitar hero guitars are much, much better than the rock band guitars. i havent used the guitar hero drums, i have the original rock band drums, and they work just fine. the rock band 2 drums are nicer because they are wireless and quieter than the original ones. as for compatibility, cross compatibility is pretty much all there on the 360. as far as i know, all of the instruments work in the rockband games, and i think they all work in the newer guitar hero games.(world tour and newer). the rock band guitar doesnt work with guitar hero 3.
with rock band, there are new downloadable songs every week...there are over 300 available to download right now, not including the tracks included on the disks. like javeryh said, you can import all of the original rockband tracks for $5, which adds alot of songs also.
i am a metal head, so i do like guitar hero because they have a generally heavier track list. however, i dont too often play any of these games by myself..they are usually when we have people over..the track list in rockband has a much more broad range of music, which is good because not everyone loves metal. plus, there are some good heavy songs in rock band available...Lamb of God FTW!!!
now i do have to say the Guitar Hero Metallica has the highest percentage of awesome tracks of any of them, but it is a relatively small set compared to rock band, so it isnt really fair.