As you implied, the problem is the number of wells that we have to drill. For example, Texas natural gas production, from gas wells, in 2006 was the highest we have seen since 1981, but it took more than twice as many gas wells in 2006 to produce the same amount of gas that we produced in 1981:

http://www.rrc.state.tx.us/divisions/og/statistics/production/ogisgpwc.html

I think that the fourth quarter of 2007 was the real start of the permanent energy boom/crisis. Whether it is a boom or a crisis depends on what side of the producer/consumer equation that one is located on. The industry can and will make money from exploiting smaller conventional oil and gas fields and from exploiting unonventional resources, but the question is whether we can increase our total net energy output, especially with personnel and equipment constraints.

I think Liebig's Law of the Minimum is likely to be a big issue also. These facilities are out in the middle of nowhere. Food and water need to be trucked in, I am guessing. Replacement parts will often come from overseas. I wonder how long the whole complex operation can be held together once oil supplies begin to decline, and we run into increased financial difficulties.