I'm not a "Linux guy", rather someone who just understands how widespread it's installed base is, and owns several devices which use it.
The main issue holding it back for widespread desktop use is the high level of fragmentation. Hardware support is much better nowadays than it was years ago, but it still doesn't offer users the "plug and go" experience that Windows does. Computer users are just that...users. They aren't code junkies, and don't think like programmers. RTFM doesn't help this massive group, because the manual might as well be written in Jabutí.
Anyone who has played with replacing firmware on a phone, knows exactly the challenges associated with Linux. While some talented folks are able to tweak stock Android installations for specific hardware, there are almost always portions of the hardware which either end up not working correctly, or even not working at all. Desktops are similar, but in some ways worse. At least the phone guys have a closed box environment when dealing with a specific phone model. A desktop can literally have millions of variations, depending on the hardware attached. To make matters worse, even if your hardware comes with the required drivers (it usually won't), the average "user" needs to jump through hoops which are outside of their comfort level, and those hoops aren't even the same for all distributions. Sure, you can limit the hardware you use to a specific distributions "supported hardware", but that is a limitation which few desire to have associated with the use of their desktop.
Then there is the issue of application support. There are some amazing free applications for Linux. There's also a lot of open-source stuff attempted to be used by others with a different hardware base than the individual who coded it. Again, this is great for the linux coding crowd, as they are able to tweak the code to do what they want it to, possibly even growing the application to support more hardware to the benefit of those less technically inclined. The problem is that this is an "organic" process. It's much like being hungry now, and having only some cash, vegetable seeds and a pot of dirt. Eventually you might have some food with the latter two, but you'll likely starve before that time. To take the analogy further, it's like having seeds which grow at a pace controlled by someone else, unless you learn how to control that pace yourself. The Linux community views this as a means to bring novices into the realm of coding, a philosophy, if you will, and it it does do that to some extent. But the vast majority will just use their cash to buy "food" elsewhere. i.e. Windows.
One example I have seen over and over again is something as simple as generic HID joysticks. In Windows, they just work, as variations in report structures are already recognized and accounted for and/or they can be calibrated in the common software interface in the OS. And, every application uses the same calls to make sure it it is receiving the calibrated data. Linux, on the other hand, uses external utilities for joystick calibration (JSCAL, IIRC) and unless the application is specifically written to use the values it creates, or the kernal is modified, your joystick may not work properly. If you have to deal with issues like this for a dead simple HID joystick, I'd hate to think what else was in store for other hardware lacking specific support.
Linux smacks of MS-DOS with Windows 3.1 layered atop. It's much more powerful than that old combo, but the need to often resort to the command line, something "users" loath and Windows has done everything it can to avoid happening, still has it in that realm.
Linux claims to be free, but if you place any value at all on your time, it's not, unless you operate strictly within the parameters of the distribution, or have a good friend like Nitrogen_Widget
Again, I don't claim to know much about the OS, so some of my observations may not be 100% (i.e. feel free to correct) but these are my observations from my incidental contact with it. I'm also not your average "user", and even I don't find myself to be too excited about rolling up my sleeves and becoming "good" at it. But I'm happy it's out there, and use it often (some version of it is on just about every portable device or "smart" appliance out there.)