our problem was similar to yours. I had several issues, that aren't addressed here, and also found some limitations in the software. I can’t say that I’m 100% sure it’s set up correctly, but it’s working from the fully qualified domain name. To use SSL properly through the reverse proxy the connection to your ArcGIS Server must also be using a secure connection. Essentially when using SSL you’ll want secure not only from the reverse-proxy, but also to the ArcGIS Server machine.
I started with changing IIS to require https for the ArcGIS Server machine. For IIS 7 on your ArcGIS Server machine setting up SSL is fairly straight forward. This link takes you to esri's website for instructions on setting up the SSL through IIS. They recommend a signed certificate, but we used a self-signed certificate on the ArcGIS Server machine. This saves having to pay a company like Verisign twice (once for the reverse proxy, and the other for the internal ArcGIS machine). Internal users that consume the services through an internal link will get a security warning. However, external users that follow the FQDN will not get a security warning, assuming your certificate on the reverse-proxy is a certified signed certificate. Once the SSL is setup on your ArcGIS Server machine, and has been tested, make changes to the httpd.conf file to reflect the https. (See code snip below.)
When setting up the reverse-proxy I found what, I think, is a better document to follow. This document was written for 9.2, 9.3, and 9.3.1, but still works for 10. While talking with esri support I asked why this document was 15 pages versus the other 3 page document for 10. Their short answer was that not everything was needed from the 15 page document. However, the 15 page document seemed to make more sense to me.
Once I had IIS on the ArcGIS Server machine requiring SSL I set the proxy passes in Apache to read https://... I also followed the instructions to use case-insensitive found in the 15 page document. Example:
ProxyPassMatch (?i)^/[arcgis_instance]/rest/services/(.*) https://[ip_address]/[arcgis_instance]/rest/services/$1
ProxyPassReverse /[arcgis_instance]/rest/services/ https://[ip_address]/[arcgis_instance]/rest/services/
Our instance on our ArcGIS Server was set to use port 8181 instead of 80. DON’T DO THIS! This is where there is a limitation in the software working with SSL. Our IT Department was using port 80 for some reason when we first set ArcGIS Server up on that server, and told us to use a different port. We had to re-install our instance of ArcGIS Server. The port 80 install in ArcGIS Server works alongside with port 443, so you won’t have to re-install your instance if set up to use port 80.
Edit your rest.config file to reflect the correct port and https. Example:
<SoapUrl>https://[fqdn]/[instance_name]/services</SoapUrl>
<SoapSslUrl>https://[fqdn]/[instance_name]/services</SoapSslUrl>
<SslPort>443</SslPort>
<ReverseProxyPort>80</ReverseProxyPort>
<ReverseProxySslPort>443</ReverseProxySslPort>
If using the .NET wizard for .NET applications, edit the ApplicationBuilderConfig.xml found in C:\inetpub\wwwroot[instance name]\Manager\App_Data. Change the <DefaultHttpScheme> to https:// instead of http://.
One other thing we changed, though I don't know if this did anything, was we inserted SSLProxyEngine On just before the proxy passes.
I hope this helps. If you think I'm missing anything or have any questions please let me know, I'll be glad to try and help.
Best Answer
The issue may be that your trying to access an older version that isn't compatible, administrative connections can only be made with ArcGIS Server and ArcGIS Desktop products that are on the same version.
ESRI has a a technical article on an error message that may be relevant to your issue here