If the five images have the same aspect ratio, it's not difficult to solve the problem: let's say that the rectangles have base a and height a/r (with possibly different a each). Let's say we want to leave 12pt of blank space between the two halves and 3pt of space between the small figures.
We have thus to compute dimensions x and y, the bases of the big rectangle and of the small ones respectively, such that
x + 2y = \textwidth-12pt-3pt
x/r = 2y/r + 3pt
A bit of linear algebra will tell that
x = (\textwidth - 15pt + 3ptr)/2
y = (\textwidth - 15pt - 3ptr)/4
Let's say that r=2; then
\documentclass[a4paper]{article}
\usepackage{graphicx,subfig}
\begin{document}
\begin{figure}
\centering
\subfloat[]{\includegraphics[width=\dimexpr(\textwidth-15pt+3pt*2)/2\relax]{fig1}}%
\hfill
\subfloat[]{\vbox{\offinterlineskip\halign{#\hskip3pt&#\cr
\includegraphics[width=\dimexpr(\textwidth-15pt-3pt*2)/4\relax]{fig2}&
\includegraphics[width=\dimexpr(\textwidth-15pt-3pt*2)/4\relax]{fig3}\cr
\noalign{\vskip3pt}
\includegraphics[width=\dimexpr(\textwidth-15pt-3pt*2)/4\relax]{fig4}&
\includegraphics[width=\dimexpr(\textwidth-15pt-3pt*2)/4\relax]{fig5}\cr
}}}
\caption{Test images}
\end{figure}
\end{document}
If the package subcaption is used instead of subfig, change \subfloat[] into \subcaptionbox{} (thanks to Axel Sommerfeldt).
I've used \halign to get a more precise control on the spacing.
If the figure on the left has a different aspect ratio than the ones on the right, it's not difficult either.
\halign is the TeX primitive on which tabular is built. There's too much to say about it; the main thing is the structure:
\halign{<template1>&<template2>&...\cr
<body>
}
The tokens up to the first \cr define the templates for the columns; each is separated from the following by &. A template consists of three parts:
<before>#<after>
<before> means tokens that are inserted at the start of each cell, # stands for the actual cell entry in the body, and similarly <after> means tokens that are inserted after the entry. For example a template for "centered cells" is
\hfil#\hfil
In our case the templates are trivial: #\hskip3pt and #; since all images will have the same width it's not necessary to specify anything else.
Notice that \halign, by default, doesn't insert intercolumn space; to be precise it inserts \tabalign glue, but talking about this would take too far away. The default value of the parameter is zero, anyway.
<body> is similar to the body of a tabular environment, but rows must be terminated by \cr and spaces before & and \cr are not ignored; those after them are. Also the final row must be terminated by \cr.
Between rows one can specify spacing (and other things) via \noalign.
To get precise control on spacing, I've said \offinterlineskip before \halign so that TeX will butt all rows together without interline glue. LaTeX does the same, actually. This is done inside a \vbox, so that the setting will be in force only there.
This seems to work but I don't really understand why. Note that I don't have your images so I had to replace them and the white doesn't show up as well on the examples from mwe. However, you can see the captions if you look closely...
\documentclass{article}
\usepackage{graphicx,caption,subcaption,tikz}
\usetikzlibrary{backgrounds}
\usetikzlibrary{calc}
\DeclareCaptionFormat{overlay}{\gdef\capoverlay{#1#2#3\par}}
\DeclareCaptionStyle{overlay}{format=overlay}
\DeclareCaptionFormat{suboverlay}{\gdef\subcapoverlay{(\thesubfigure) #3\par}}
\DeclareCaptionStyle{suboverlay}{format=suboverlay}
\newcommand{\captionOverlay}[1]{%
\captionsetup{format=overlay}
\caption{}%
\begin{tikzpicture}
\node [inner sep=0] (image) {#1};
\draw node [white] {\capoverlay};
\end{tikzpicture}%
}
\captionsetup[subfigure]{format=suboverlay}
\newcommand{\subcaptionOverlay}[1]{%
\subcaption{}%
\begin{tikzpicture}
\node [inner sep=0] (image) {#1};
\draw node [white] {\subcapoverlay};
\end{tikzpicture}%
}
\begin{document}
\begin{figure}[htpb]
\centering
\begin{subfigure}[b]{.3\textwidth}
\centering
\subcaptionOverlay{\includegraphics[width=\textwidth]{example-image-a}}
\label{subA}
\end{subfigure}
\begin{subfigure}[b]{0.3\textwidth}
\centering
\subcaptionOverlay{\includegraphics[width=\textwidth]{example-image-a}}
\label{subB}
\end{subfigure}
\caption{}
\label{mainFig}
\end{figure}
\begin{figure}[htpb]
\centering
\captionOverlay{\includegraphics[width=\textwidth]{example-image-a}}
\label{normalFig}
\end{figure}
\end{document}
Best Answer
The
subfigureenvironment from thesubcaptionpackage takes all the same arguments as aminipage.The
minipageenvironment adds a small amount of horizontal space at the end of the environment, which you can remove by using a%after\end{minipage}, or in this case\end{subfigure}.All I've done in the MWE below is replace
~with%; you are correct about your description of\textwidth, it uses the parent value.