Even before the devastating hurricanes of last summer, world oil markets had been subject to a degree of strain not experienced for a generation. Oil prices had been persistently edging higher since 2002 as increases in global oil consumption progressively absorbed the buffer of several million barrels a day in excess capacity that stood between production and demand. Today world oil production stands at about 85 million barrels a day, and little excess capacity remains. Just how much excess capacity, and of what quality oil, is a matter of debate. But no matter what the precise answer, the buffer between supply and demand is much too small to absorb shutdowns of even a small part of the world's production. Moreover, growing threats of violence to oilfields, pipelines, storage facilities, and refineries, especially in the Middle East, have increased the private demand to hold oil inventories worldwide. Oil users judge they need to be prepared for the possibility that at some point a raid will succeed, with a devastating impact on supply.

For most of the history of oil, its producers and consumers determined its price. Only those who could physically store large quantities of oil had the ability to trade. But important advances in finance have opened the market to a much larger number of participants. There has been a major upsurge in over-the-counter trading of oil futures and other commodity derivatives. Thus, when in the last couple of years it became apparent that the world's oil industry was not investing enough to expand crude-oil production capacity quickly enough to meet rising demand, increasing numbers of hedge funds and other institutional investors began bidding for oil. They accumulated it in substantial net long positions in crude oil futures, largely in the over-the-counter market. These net long futures contracts, in effect, constituted a bet that oil prices would rise. The sellers of those contracts to investors, when all of the offsetting claims are considered, are of necessity the present owners of the billions of barrels of private inventories of oil held throughout the world - namely, the producers and consumers.

Even though inventories of oil have risen significantly in recent years, persistent upward price movements have made it apparent that the rise in investors' ownership claims to the world's oil inventories has likely exceeded the inventory increase. This implies a reduction in the unencumbered inventory holdings of producers and consumers. In other words, some part of the oil in the world's storage tanks and pipelines is spoken for by investors. The extent of the surge in participation by financial institutions in claims on real barrels of oil is reflected in the near tripling of the notional value of commodity derivatives (excluding precious metals) during the four quarters of 2005 reported by U.S. commercial banks. Most of those contracts are for oil. The accumulation of net long positions in oil on the New York Mercantile Exchange by non-commercial traders , which is to say by investors, has exhibited a similar pattern.

The new participants, investors and speculators, to the world's two trillion dollar-a-year oil market are hastening the adjustment process that has become so urgent with the virtual elimination of the world supply buffer. With the demand from the investment community, oil prices have moved up sooner than they would have otherwise. In addition, there has been a large increase in oil inventories. In response to higher prices, producers have increased production dramatically and some consumption has been scaled back. Even though crude oil productive capacity is still inadequate, it too has risen significantly over the past two years in response to price.

Hypothetically, if we still had the 10 million barrels a day of spare capacity that existed two decades ago, neither surges in demand nor temporary shutdowns of output from violence, hurricanes or unscheduled maintenance would be having much, if any, impact on price. Returning to such a level of spare capacity appears wholly out of reach for the foreseeable future, however. This is not because there is any shortage of oil in the ground. The problem is that aside from Saudi-Aramco, few, if any, of national oil companies which own most of the world's proved oil reserves are investing enough of their surging cash flow to convert the reserves into crude oil productive capacity. Only Saudi-Aramco appears sufficiently concerned, at least publicly, that high oil prices will reduce the long term demand for oil, which could significantly diminish the value of Saudi Arabia's - or indeed, any country's -oil reserves.

Although outlays on productive capacity are rising, the significant proportion of oil revenues held as financial assets suggests that many governments perceive that the benefits of investing in additional capacity to meet rising world oil demand are limited. Moreover, much oil revenue has been diverted to meet the perceived high-priority needs of rapidly growing populations. Unless those policies, political institutions, and attitudes change, it is difficult to envision a rate of reinvestment by these economies adequate to meet rising world oil demand. Some members of the Organization of Petroleum Exporting Countries (OPEC) have recently announced expansion plans. But how firm such plans are, is difficult to judge. They and other nations have rebuffed offers by international oil companies to help tap their reserves. Opportunities to expand oil production elsewhere are limited to a few regions, notably the former Soviet Union.

Besides feared shortfalls in crude oil capacity, the adequacy of world refining capacity has become worrisome as well. Over the past decade, crude oil production has risen faster than refining capacity. A continuation of this trend would soon make lack of refining capacity the binding constraint on growth in oil use. This may already be happening in certain grades of oil, given the growing mismatch between the heavier and more sour content of world crude oil production and the rising world demand for lighter, sweeter petroleum products.

There is thus a special need to add adequate coking and desulphurization capacity to convert the average gravity and sulphur content of much of the world's crude oil to the lighter and sweeter needs of product markets, which are increasingly dominated by transportation fuels that must meet ever more stringent environmental requirements. Yet the expansion and modernization of world refineries are lagging. For example, no new refinery has been built in the United States since 1976. The consequence of lagging modernization is reflected in a significant widening of the price spread between the higher-priced light sweet crudes such as Brent which are easier to refine and the heavier crudes such as Maya, which are not.

To be sure, refining capacity does continue to expand, albeit too gradually, and oil exploration and development is continuing, even in industrial countries. Conversion of the vast Athabasca oil sands reserves in Alberta to productive capacity, while slow, has made this unconventional source of oil highly competitive at current market prices. However, despite improved technology and high prices, additions to proved reserves in the developed world have not kept pace with production; so those reserves are being depleted.

The history of world petroleum is one of a rapidly growing industry in which producers have sought to provide consumers with stable prices to foster the growth of demand. In the first decade of the 20th century, pricing power was firmly in the hands of Americans. Even after the breakup of the Standard Oil monopoly in 1911, pricing power remained with the United States--first with the U.S. oil companies and later with the Texas Railroad Commission, which would raise limits on output to suppress price spikes and cut output to prevent sharp price declines.

Indeed, as late as the 1950s, crude oil production in the United States (more than 40% of which was in Texas) still accounted for more than half of the world total. In 1951, excess Texas crude was poured into the market to contain the impact on oil prices of the nationalization of Iranian oil. Excess American oil was again released to the market to counter the price pressures induced by the Suez crisis of 1956 and the Arab-Israeli War of 1967.

American oil's historical role ended in 1971, when rising world demand finally exceeded the excess crude oil capacity of the United States. At that point, the marginal pricing of oil abruptly shifted--at first to a few large Middle East producers and later to market forces broader than they, or anyone, can contain.

To capitalize on their newly acquired pricing power in the early 1970s, many producing nations, especially in the Middle East, nationalized their oil companies. The full magnitude of the pricing power of the nationalized companies became evident in the aftermath of the oil embargo of 1973. During that period, posted crude oil prices at Ras Tanura, Saudi Arabia, rose to more than $11.00 per barrel, far above the $1.80 per barrel that had been unchanged from 1961 to 1970. The further surge in oil prices that accompanied the Iranian Revolution in 1979 eventually drove up prices to $39 per barrel by February 1981. That translates to $76 per barrel in today's prices.

The higher prices of the 1970s abruptly ended the extraordinary growth of U.S. and world consumption of oil and the increased intensity of its use which were hallmarks of the decades following World War II. Since the more than tenfold increase in crude oil prices between 1972 and 1981, world oil consumption per real dollar equivalent of global gross domestic product (GDP) has declined by approximately one-third.

In the United States, between 1945 and 1973, consumption of petroleum products rose at a startling average annual rate of 4-1/2 percent, well in excess of growth of our real GDP. However, between 1973 and 2006, U.S. oil consumption grew, on average, at only 1/2 percent per year, far short of the rise in real GDP. In consequence, the ratio of U.S. oil consumption to GDP fell by half.

Much of the decline in the ratio of oil use to real GDP in the United States has resulted from growth in the proportion of GDP composed of services, high-tech goods, and other less oil-intensive industries. The remainder of the decline is due to improved energy conservation: greater home insulation, better gasoline mileage, more efficient machinery, and streamlined production processes. These ongoing trends seem to have intensified of late with the sharp, recent increases in oil prices.

To date, it is difficult to find serious erosion in world economic activity as a consequence of sharply higher oil prices. Indeed, we have just experienced one of the strongest global economic expansions since the end of World War II. The United States, especially, has been able to absorb the huge implicit tax of rising oil prices so far. However, recent data indicate we may finally be experiencing some impact.

Clearly, if the current almost non-existent supply buffer were significantly increased through a step-up in supply or a stepdown in consumption, oil prices would fall, perhaps sharply. This would likely occur even if there were no decrease in the threat to oil facilities from attacks or hurricanes. A large enough buffer could absorb such contingencies with modest impact on price.

But for good reason, holders of claims to the existing private inventories of oil apparently do not foresee a likelihood of change sufficient to alter the current outlook. This does not mean that oil prices will necessarily move higher, however. All of the concerns about future contingencies are already discounted in today's spot price. It will require a change in the outlook one way or the other to move crude oil prices. History tells us that will happen - often.

The U.S. economy has been able to absorb the huge impact of rising oil prices with little consequence to date because it has become far more flexible over the past three decades owing to deregulation and globalization. Growing protectionism would undermine that flexibility and make our nation increasingly vulnerable to the vagaries of the oil market.

Current oil prices over time should lower to some extent our worrisome dependence on petroleum. Still higher oil prices will inevitably move vehicle transportation to hybrids, and despite the inconvenience, plug-in hybrids. Corn ethanol, though valuable, can play only a limited role, because its ability to displace gasoline is modest at best. But cellulosic ethanol, should it fulfill its promise, would help to wean us of our petroleum dependence, as could clean coal and nuclear power. With those developments, oil in the years ahead will remain an important element of our energy future, but it need no longer be the dominant player.