The service parameter in the QGIS/PostGIS dialog should be empty.
And I spent a bit of time killing the user account "postgres" in windows and removing the names of the databases which are created in the windows registry.
After that I could repeatedly fail the dialog by having something in the service parameter of the QGIS Postgis connection dialog.
You should verify that you can connect using psql. Try psql -U username -h localhost dbname. It should prompt for a password then connect. Run SELECT postgis_version(); to verify that PostGIS is active in the database.
If you can connect but SELECT postgis_version() reports an error, PostGIS isn't installed in the database:
ERROR: function postgis_version() does not exist
LINE 1: SELECT postgis_version();
Other solution:
SELECT * FROM Postgis_lib_version();
If you want also to see the version of PostGIS and the version number of the libraries GEOS and Proj4 just change lib to Full.
If you get the above error, then presuming you're running PostGIS 2.0 on PostgreSQL 9.1 or above, connect as user postgres and run CREATE EXTENSION postgis;. Eg:
psql -U postgres -h localhost dbname -c 'CREATE EXTENSION postgis;'
You might also need to install some of the extras, like the PostGIS topology support:
CREATE EXTENSION postgis_topology;
or the legacy support script, which isn't packaged as an extension and must be sourced:
psql -U postgres -h localhost dbname -f /path/to/postgis-2.0/legacy.sql
See the PostGIS documentation.
If you can't connect to the DB, it's probably a pg_hba.conf issue, failure to create the DB or user, etc. Hard to say without error messages and log contents.
Best Answer
After wasting several hours it turns out that QGIS cannot connect to an empty database, and Postgres.app ships with one.
Solution was to initialize PostGIS in the database in psql with
create extension postgis;and then connection was successful.