Here's how we ended up doing it. It was a several day process, and as ujjwalesri suggested, a fail-over as a standard configuration is the way to go. However, we weren't lucky enough to have a fail-over. For the most part this worked, but we encountered a couple issues.
After installation of ArcGIS Server and post install, etc. we copied over the files from the server/user/cfg folder. Paste these into the same location on the new server. (This assumes that the new server is accessing the mxd/msd through a network and that those documents are not local to the server.)
Next, copy the applications from the inetpub/wwwroot folder. Again, paste these into the inetpub/wwwroot folder on the new server.
Edit the Sites XML document found in the inetpub/wwwroot/ArcGIS/Manager/App Data (ArcGIS may be replaced by your named instance.) The Host tag will be edited within the sites XML document.
IIS 7 was used on the new server, whereas IIS 6 was used on the old server. This was an issue only in the sense that I haven't used IIS 7, and it is very different. Find the applications that were just pasted into the wwwroot folder in the IIS 7 manager, under default website, and right-click and select Convert to Application.
Next, change all references to the old server in the code found in the inetpub/wwwroot/[application name]. I found the server references in numerous files throughout the folders. By editing each service in the ArcGIS Server Manager, and selecting the correct output directories, etc., the CFG files were updated appropriately.
Eventually esri sent us these instructions...a little late:
Server Configuration Files
And
HowTo: Move a ArcGIS Server .NET Web application to a new Web server machine
I hope this helps someone. It can be done, it's just a major pain trying to find all the references to server names.
The problem in this case was rather stupid:)
The cache mentioned above was residing in a folder, that was located in another folder, that was located in another folder. So despite the fact that I registered the main folder as a cache folder for the server, it couldn't read inside all these folders.
So we created a new folder, named it exactly as the map service and moved there only the folders containing the bundle files.
Bottom line: don't store you cache in a cascade of folders.
Best Answer
The company I work for (Latitude Geographics) makes a product called Geocortex Optimizer that can do this for you.
It analyzes both your ArcGIS Server logs and your IIS logs to determine what parts of your maps are being requested:
That's the results of some stress testing we did on it, so it's not as representative of the final result. (Normally you only get really "hot" zones over urban centers)
If you want more information, use the "Request a demo" function on our web site.