We do know there's a crapload of oil in the ground that can never be touched. In fact, I think that even the easiest to evacuate fields are still left with about 1/3rd left just because it can't be extracted with a net energy gain. I'm not positive on that though
Yeah that's how I understand it as well.

Quoting the Crash Course I mentioned above:
But something interesting happens at the halfway mark. Where oil gushed out under pressure at first, the back half usually has to get laboriously pumped out of the ground at higher cost, obviously. Where every barrel of oil was cheaper to extract on the way up, the reverse is true on the way down. Each barrel becomes more costly in terms of time, money, and energy to extract. Eventually, it costs more to extract a barrel of oil than it is worth, and that’s when an oil field is abandoned.
...
Here’s an interesting aside. Suppose we wanted to become “independent from imported oil” and decided to replace those 10 million imported barrels with some other form of energy. Those 10 million barrels represent the same power equivalent as 750 nuclear power plants. Considering the issues we have with the 104 we have operating right now, I think it’s safe to say nuclear power is not a realistic candidate for reducing oil imports. Well then, how much would we have to increase our solar wind and biomass energy production? There, we’d have to increase our currently installed base by a factor of 2,000. Not 2000%. Two-thousand times as much.