Unless you have good reasons not to, I'd definitely recommend starting with the OSGeo4W installer, which can install multiple different versions of GDAL and their relevant Python bindings. It works great and dramatically simplifies the Windows deployment story. Specifically, you'll want to install pkg-gdal-python, which is within 'Libs' in the installer tree.
Have you logged off or rebooted your server since setting the environment variables?
The changes only come into effect once you do this.
I often now just set these in my Python scripts manually to avoid conflicts / reliancy on Windows variables. You could try something similar in your script or IDLE:
import os
os.environ['PATH'] = "C:\\Program Files\\GDAL" + ';' + os.environ['PATH']
os.environ['GDAL_DRIVER_PATH'] = "C:\\Program Files\\GDAL\\gdalplugins"
os.environ['GDAL_DATA'] = "C:\\Program Files\\GDAL\\gdal-data"
Assuming this still fails..
The PostGIS driver requires the libpq.dll (the postgres client library). You can test the GDAL/OGR dll with depends.exe to see if it is missing.
I would have assumed it had been compiled with this as the Info Page lists the PostGIS driver. It is present in a recent MapServer download I took from the same site.
However it is worth checking, and making sure it is in your PATH variable.
After checking the Info Page again it notes GDAL was built with the postgresql-8.3.7 client library. As you are using PostgreSQL 9.04-1 this could be a problem.
In my MapServer download there is a:
gdal\plugins-optional\ogr_PG.dll
I believe this is required for using the PostGIS plugin with OGR, so moving this to your PATH could well allow you to use the driver.
As you say in your comments this does not seem to be included in the package. To resolve the issue, download a MapServer package, which does contain this file. Once I added this to the C:\Program Files\GDAL\gdalplugins folder with the exact same installs as you it worked:
Best Answer
I'd recommend installing GDAL using OSGeo4W, which will set up an environment where all these tools will work nicely together.
You may have some luck using the new package management tool for Windows, Chocolately, but I haven't tried it yet for GDAL (it worked really well to install NPM and other tools for me, though, and seems to be the sensible option these days).