Spending four million bucks pushing the boundaries on current capabilities to cut out 20% of your market is stupid.
From a buisness standpoint, perhaps. From an artistic standpoint, no.
Id is a company of programmers. They are founded by and run by programmers. They think like programmers. And for programmers playing with hardware is one of the more interesting things to do in game development. And programmers like nothing more than interesting things to do.
They push the limits of the artform of videogames in their own way. Yes, they might have made more money on Doom3 if it had not been as ambitious. But then again, since the gameplay wasn't that great to begin with, it sold primarily on its ambition and the atmosphere that the ambition created. So maybe it would have sold even less if it hadn't been ambitious.
Id is a company of programmers, and it shows. Blizzard is a company of artists, and it shows. Programmers get in the way of artists, telling them what tools they can use and so forth. Best to choose levels of hardware that there are mature tools for and tell the programmers to go to hell.
Valve is a company of game designers. Everything feeds back into that, programming and art alike. Again, they want to do cool stuff, but they're going to push things in a direction that feeds into gameplay. The physics in the Source engine/HL2. The Apeture Science Hand-Held Portal Device in, well, Portal. The Director AI in Left 4 Dead. And so on.
Linear play and no longer on my HD.
Yes, but you bought them. That's all they care about, from a business perspective.
As far as business is concerned, infinite replay value is a bad thing. Because if you're still playing game X, you don't have the time to put into playing the new game Y that they just put out. It's the WoW problem: WoW itself depresses PC game sales because time spent playing WoW is time not spent playing other games. And money spent playing WoW is money not spent playing other games. WoW is a bargain at $15 a month if it can replace all of your gaming, and your normal gaming habit is one $50 game every month. Of course, that's money that other developers don't get.