Why would there be lag? Is there lag when you flip on your light? At any rate, no there is no lag. Cars have had partial drive-by-wire for years with Antilock Brakes and more and more often we're getting dynamic stability control that adjusts the speed of the wheels to compensate for slipping on wet roads or for taken corners too fast. These adjustments are made hundreds of times per second, and all without the driver knowing it.
As to the big question, that's easy. Because whether or not someone should be driving has very little influence over whether they will be driving. And even most good drivers are probably safer in a dive by wire car.
I can see, perhaps, a possibility of making ourselves dumber, in the same way that calculators have probably had a negative impact on our ability to do basic math in our heads, or to do division even on paper. A person who learned to drive in a car with ABS doesn't learn how to properly brake suddenly in a car without ABS, for example. But overall our roads will be safer, just like overall calculators have advanced our ability to perform mathematics.