The reason for oil to shift back and forth between contango and backwardation is it is highly volatile (politically I mean) and current supply is tight.  When I say that it is tight there is little flex in the market. A single political event or statement psyches out the oil future market (ie Osama releases an audio tape or a terrorist attack). Weather conditions like the previous hurricane season changed prices at the pumps well before any supply difficulties were realized.
  Other future markets like wheat for example are not the subject of debate and scrutiny, and therefore less volatile.  I think when supply is on the decline though there will be one last "flip" to contango.  But that shift still has a few or several years to go.  I work on a drill ship, in a fleet of such ships.  My company is making record profits.  Wells are pumped dry every day but new ones are drilled in deeper and deeper water.

This economic shift in theory would be gradual (see adam smith) it is the political and popular culture that drives the oil futures.

Bingo.  This is a classic case of something we want to measure (the price of oil futures contracts) being heavily influenced by things (natural catastrophes, politics) that are impossible to quantify.

I expect we'll see much greater volatility in the near future, in part because the Bad People have likely figured out how easy it is to cause the West economic pain by staging a high profile (even if unsuccessful) attack on the oil infrastructure.

This this piece on the former Saudi oil minister seems germane to all of this as well...
Until we're all independent of oil?? Is that what he said? When does he expect that to happen--after the dieoff?
The problem with futures is that they are traded NOW.

If I expect resources to be scarce in, say, one to five years, I will be willing to pay more for it NOW. As traders say, it's already in the price.

If the world believes we are past peak, they would be willing to pay 10$ gasoline prices NOW - simply because it will be harder to get in 5 years. That means that the price that it will take to produce the stuff in 5 years are already considered NOW and do not need to be put into the futures' prices which are due in five years. The futures prices will therefore remain a sort of running average of past prices MINUS what you would get by putting your money into bonds.

That is why we see a long period of constant futures in the 1990s for instance. Price went up and down, but the average was mostly the same - thus the futures (five years' difference, for instance) remained mostly the same.

If you see the futures price as "present expectations of the price in the future", you will be led down the wrong path. PRESENT PRICES already have the present expectations of future availability priced in.

If you try to use the future's price to tell us something about the future, you fall into the thought trap that Greenspan fell into a while back - that PO doesn't exist because the future markets don't show it. As he made his comments (not quite two years ago, if I remember correctly), oil was priced somewhere under 40$.

I have to disagree with you on this, as this entire discussion of contango and backwardation is a discussion of exatly when and why futures prices DO NOT remain a "running average of past prices MINUS what you would get by putting your money into bonds."  Futures prices do, in fact, represent the market's expectation of the future equillibrium of supply and demand (price)--if they only represented today's price minus carry cost and bond yield, then they would always be in a formulaic backwardation, and this is definitely not what the price data actually shows.  How, for example, would this "running average minus" maxim account for futures prices being higher in December 2007 than the spot price today?  Just my opinion, but I think that some of the other commentary on this thread has done an excellent job of explaining the real mix between short-term risk and long term supply issues...
Yeah, sorry. Of course there's a premium for futures in the short term (about a year) - otherwise you would be in a deflationary world - buy tomorrow at a cheaper price than today, then you always buy 3-12 months in advance and sell at termin, month for month - a nice arbitrage...

But after that, it's a question of "investment" (speculation) and what's going to give you the most return on your money. In that respect, all investments must be equal. That's why the comparison with the bonds comes in.

Sorry again. I meant "sell at delivery date". German slipped in there somehow.

Anyway, the economist would say there's no free lunch - neither in the short term or in the long term, and the typical curve (Feb. 2005) demonstrates this wisdom.