Technically, Earth is a diathermic system. Earth does exchange energy with its surroundings as heat (energy) is radiated into space. However, as you correctly point out, Earth does not exchange mass with its surroundings.

The truth is that it does not matter a whole lot about what KIND OF SYSTEM Earth is. What matters is a little thing called equilibrium. The burning of fossil fuels is upsetting the thermal/environmental equilibrium that previous generations were used to. Before humans numbered into the millions/billions, the temperature/environmental conditions of Earth changed abruptly only when something like several volcanoes erupted, solar flares increased the radiation flux, meteors struck Earth, or some other very punctuated event occurred. Earth reached an equilibrium after events like these because they were often few and far between. When man started populating the earth in large numbers, burning fossil fuels, and destroying plants that sequestered the incoming solar radiation, Earth was slowly poisoned and could not recover, in the same way that Hanson's disease slowly spreads though the human body or in the same way mercury or arsenic build up until death. Mankind is simply overwhelming the natural tendency of Earth to equilibrate to conditions that support man and other forms of life.

Earth has got a fever of sorts. Just as a fever to a person is a response to an invasion of disease, global warming is a response to infection from humanity. The fever of Earth will only break when the infection is finally subdued through natural disasters (e.g., drought, starvation, tornadoes, earthquakes, or disease)...unless mankind kills it first.