I've had extensive business relationships with research scientists at both XOM and SLB.  In my experience big oil makes a point of only employing top talent in engineering and science.  However my colleagues were often frustrated by their management's conservative approach to adopting new technology and emphasis on practical results that impacted the bottom line in the near-term.  They were constantly in fear of the budget consequences of the next dip in the price of oil as R&D is one of the first areas that big oil cuts when the profits dry up.  I agree that this culture has tended to stifle real innovation, instead making big oil R&D departments procurers of technology that others have paid to develop to a fairly high technical readiness level.  I've seen big oil swallow up 100 person companies and fairly hefty patent portfolios without a one line blurb on their web site; sometimes the technology ends up down a well or in an exploration ship; sometimes it doesn't produce $ soon enough and gets flushed never to be seen again.

Most of the real innovation occurs in universities, gov labs, or much smaller dynamic start-ups, which is why government R&D spending, progressive small business R&D tax treatment and especially availability of private venture capital are far more important to solving our energy supply problems than big oil R&D expenditures.

Agree. When I was visiting Amoco, they showed us a big fancy research building but a small semiconductor lab with just a few researchers working it. No way could they even compete with, for example, the Coordinated Science Lab down at U of Illinois, where you could find people like Holonyak, Morkoc, and the elder statesman Bardeen inventing and perfecting all sorts of new solid-state technologies.
Wow, great summary.

This is virtually identical to what happens in the life sciences, like plant and animal research.

The really basic research is done at universities not business.  Business has leveraged decades of basic research into products but is now running out of areas to capitalize on.  No one really wants to do basic R&D because of cost and timelines.  My impression is that policy makers think that giving tax breaks to big multinationals and drug companies will spur the R in R&D.  My experience is that it only cranks up the D side and fosters lots of mergers to leverage other peoples intellectual property that took decades to build.  

We need dollars going to people to just do Research without any expected product in 2 years or less.  That knowledge often leads to new products, but only after 15-20 years of other complimentary work getting done.  Then the area is mature and a flurry of products can be made.  It appears no one is patient enough for research to pay off anymore.

Our great mis-leader wonders why one should waste money re-searcherizing thru stuff you already done searched through. Clearly, the R part of the R&D is a complete waste of political capital.