"I was really thinking that the processes of nature are optimized for energy collection and transfer over both distance (microns to kilometers) and time scales"

This opinnion, unless a humoristic entry, would be greatly ignorant of how evolution works. Plants produce hard cellulose walls and any number of nasty chemistry for a simple purpose: they are trying to make it harder for herbivors to eat and to digest them. There is, of course, a co-evolution going on between plants and herbivors. By large the plants are winning.  We are the perfect example of just that: we have essentially given up to compete on that level of the game and are have resolved to eating the herbivors that haven't quit the plant battle, yet.

Absolute energy efficiency is, in most eco-systems, not a problem or we would have seen plants with photochemical mechanisms far superior to photosynthesis. After all... a silicon solar cell beats photosynthesis by a factor of 20! But it seems to be harder for evolution to switch from the local extremum that photosynthesis represents to a totally different chemical electron transfer and storage mechnism than to keep improving other survival mechanisms (like toxins). As long as everyone competes using the same photosynthesis engine, there is very little selection pressure from that side.

As a result... we can not expect to get much more energy out of plants than we already are (except that we don't use most of it efficiently). We could get two orders of magnitude more energy out of solar cells, though.

"This opinnion, unless a humoristic entry, would be greatly ignorant of how evolution works. Plants produce hard cellulose walls and any number of nasty chemistry for a simple purpose: they are trying to make it harder for herbivors to eat and to digest them."

Talk about the pot calling the kettle black. The suggestion that there is some purpose in the way evolution works, that evolution did something in order to accomplish something else, suggests a fundamental misunderstanding of the process.

The first paragraph in this comment is the funniest thing I've read on this site in a long time.

Yes, there is a co-evolution going on between plants and herbivores. Looked at the level of populations, rather than individuals, there is some give and take, but the relationship is largely a symbiotic one. Herbivores distribute seeds, either by their intestinal tract, or by their coats. The same goes for pollen. That act would be a good trade from the grass' perspective, but there's another important benefit that the herbivore brings: The creation and maintenance of habitat.

Perennial grasses (and clovers) that co-evolve with herbivores must be able to withstand being cropped every couple of years. Plants that can't deal with this cropping as well, can't compete in such an environment. This fact is why the plains were covered in perennial grasses (and clovers) but not trees. If it weren't for the herbivores, trees would have taken over much more of the plains. If the plants are winning, they are winning with the help of the herbivores.

But your last two paragraphs are just about bang on. I won't quibble the details because it would muddy an very insightful point.

Perennial grasses (and clovers) that co-evolve with herbivores must be able to withstand being cropped every couple of years

take it even further- perennial grasses and clovers depend on being cropped regularly for their survival.  At least in the midwest, if a field is not regularly cut down by burning, mowing or herbivorous chewing, it will be replaced by forest in short order.  This is part of the natural symbiosis between grazing animals and perennial grasses.  

If it weren't for the herbivores, trees would have taken over much more of the plains.

I think you're forgetting the important role of prairie fires.