your question is too big to be answered in one post and I think it is not clear enough.
I assume currently you have some predefined images on the server for different areas and depending on what user want to see you load one image or another.
To be more dynamic (like this http://openlayers.org/dev/examples/wms.html) one possibility is to configure a WMS server.
The idea of a WMS is you have some sources of data: shapefiles, images, etc and "add" them to the WMS server. (See: http://workshops.opengeo.org/geoserver-intro/overview/wms.html)
For example, you can have a 1600x1000 Europe image georefernced in GeoTIFF format and add it to the WMS server. Then any request to the WMS server requesting for a bounding box of Spain will produce the server load the image, cut the piece for spain and return a PNG image for that part.
On the client side OpenLayers has a WMS layer wich you can use for this. Simply point to your WMS server.
Ok, this is the solution using a WMS server. As you can see everytime you open your OpenLayers web page some request are madeing against your WMS server, it computes the images and returns it. If you move the map, a new request with the new BBox is made.
In summary, the WMS works a lot and becauyse this appears applications like geowebcache or tilecache. The first time a BBOX request is made it is passed to the WMS server and the result image is stored. For the subsequent requests the cache system will return the stored image avoiding make work the WMS server.
Another solution to serve images, used for big providers liek Bing or google, is to create the imagery, store as images and them simply serve that images when requested. It is because the google maps looks like a set of square tiles.
(See: http://msdn.microsoft.com/en-us/library/bb259689.aspx and http://www.maptiler.org/google-maps-coordinates-tile-bounds-projection)
Supposing you are generating a tile pyramid for the whole world, the idea is to create one image 256x256 for the whole world (level 0), then four images of 256x256 each one for the whole world again (level 1), then 16 images of 256x256 for the whole world (level 2), etc
This will result on a set of images on your disk stores like: LEVEL/X/Y.png
This is the TMS or newest OGC spec WMTS. Again OpenLayers offers a TMS or XYZ layer to access this kind of layers.
The complex part to cook a TMS pyramid or configured a WMS is to have the ingredients, that is the data. If you have some data with good degree of resolution yo have some "levels" then you have the more important part.
Because you have worked with Java I recommend you start using GeoServer. It is a Java WMS, WFS, WCS, ... server, easy to install and configure.
There are good tutorial and documents out there. Here are a couple:
Hope this can clarify you a bit.
Best Answer
Have a look at GeoTools. This is a Java geospatial toolkit maintained through OsGeo.