Here is a very rough answer.
The Gromov-Witten invariants show up in a few a priori different
contexts within string theory. Let me focus on one particular place they show up that is
directly related to conventional physics, as opposed to topological
quantum field theory.
Type IIA string theory is formulated on a spacetime "background"
which is, in the simplest setup, just a Lorentzian 10-manifold. The
equations of motion of the theory require (at least in their leading
approximation) that the metric on this 10-manifold should be Ricci-flat.
A popular thing to do is to take this 10-manifold of the form
X x R^{3,1}, where X is a compact Calabi-Yau threefold.
We can simplify matters by taking X to be very small ---
smaller than the Compton wavelength of any of the particles we are able
to create. (Remember that in quantum mechanics particles have a
wavelike character, with wavelength inversely related to their energy;
since we only have limited energy available to us, we can't make
particles with arbitrarily short wavelength.) A little more precisely,
let's take X such that the first nonzero eigenvalue of the Laplacian is
larger than the energy scale we can access.
In this case we low-energy
observers will not be able to detect X directly in any experiments. To
us, spacetime will appear to be R^{3,1}.
What will be the physics we see on this R^{3,1}? We will see
various different species of particle. Each species of particle that
we see corresponds to some zero-mode of the Laplacian of X.
In particular, there are particles corresponding to classes in H^{1,1}(X).
The genus 0 Gromov-Witten invariants are giving
information about the interactions between these particles. (So if you want to calculate what will come out when you
shoot two of these particles at each other, one of the inputs to that calculation
would be the genus 0 Gromov-Witten invariants.) The higher genus Gromov-Witten
invariants are giving information about interactions which involve these particles
together with other particles related to the gravitational interaction.
This is a great question I wish I understood the answer to better.
I know two vague answers, one based on derived algebraic geometry and one based on string theory.
The first answer, that Costello explained to me and I most likely misrepeat,
is the following. The B-model on a CY X as an extended TFT can be defined in terms of
DAG: we consider the worldsheet $\Sigma$ as merely a topological space or simplicial set (this is a reflection of the lack of instanton corrections in the B-model), and consider the mapping space $X^\Sigma$ in the DAG sense. For example for $\Sigma=S^1$ this is the derived loop space (odd tangent bundle) of $X$.. In this language it's very easy to say what the theory assigns to 0- and 1-manifolds: to a point we assign coherent sheaves on $X$, to a 1-manifold cobordism we assign the functor given by push-pull of sheaves between obvious maps of mapping spaces (see e.g. the last section here). For example for $S^1$ we recover Hochschild homology of $X$. Now for 2-manifold bordisms we want to define natural operations by push-pull of functions, but for that we need a measure -- and the claim is the Calabi-Yau structure (together with the appropriate DAG version of Grothendieck-Serre duality, which Kevin said Lurie provides) gives exactly this integration...
Anyway that gives a tentative answer to your question: the B-model assigns to a surface $\Sigma$ the "volume" of the mapping space $X^\Sigma$, defined in terms of the CY form.
More concretely, you chop up $\Sigma$ into pieces, and use the natural operations on Hochschild homology, such as trace pairing and identification with Hochschild cohomology (and hence pair-of-pants multiplication).. of course this last sentence is just saying "use the Frobenius algebra structure on what you assigned to the circle" so doesn't really address your question - the key is to interpret the volume of $X^\Sigma$ correctly.
The second answer from string theory says that while genus 0 defines a Frobenius manifold you shouldn't consider other genera individually, but as a generating series -- i.e. the genus is paired with the (topological) string coupling constant, and together defines a single object, the topological string partition function, which you should try to interpret rather than term by term. (This is also the topic of Costello's paper on the partition function). BTW for genus one there is a concrete answer in terms of Ray-Singer torsion, but I don't think that extends obviously to higher genus.
As to how to interpret it, that's the topic of the famous BCOV paper - i.e. the Kodaira-Spencer theory of gravity. For one thing, the partition function is determined recursively by the holomorphic anomaly equation, though I don't understand that as "explaining" the higher genus contributions. But in any case there's a Chern-Simons type theory quantizing the deformation theory of the Calabi-Yau, built out of the Kodaira-Spencer dgla in a simple looking way, and that's what the B-model is calculating.
A very inspiring POV on this is due to Witten, who interprets the entire partition function as the wave function in a standard geometric quantization picture for the middle cohomology of the CY (or more suggestively, of the moduli of CYs). This is also behind the Givental quantization formalism for the higher genus A-model, where the issue is not defining the invariants
but finding a way to calculate them.
Anyway I don't know a totally satisfactory mathematical formalism for the meaning of this partition function (and have tried to get it from many people), so would love to hear any thoughts. But the strong message from physics is that we should try to interpret this entire partition function - in particular it is this function which appears in a million different guises under various dualities (eg in gauge theory, as solution to quantum integrable systems, etc etc...)
Best Answer
Donaldson-Thomas invariants in mathematics are a virtual count of sheaves (or possibly objects in the derived category of sheaves) on a Calabi-Yau threefold. In physics, sheaves (and more generally objects in the derived category) are considered as models for D-branes in the topological B-model and Donaldson-Thomas invariants are counts of the BPS states of various D-branes systems. For example, the "classical" DT invariants that are considered by MNOP count ideal sheaves of subschemes supported on curves and points. You will hear physicists refer to such invariants as "counting the states of a system with D0 and D2 branes bound to a single D6 brane". The single D6 brane here is the structure sheaf $\mathcal{O}_X$ and the D0 and D2 branes form the structure sheaf $\mathcal{O}_C$ of the subscheme $C$ (which is supported on curves and points) and the term "bound to" refers to the map $\mathcal{O}_X \to \mathcal{O}_C$ because they are replacing the ideal sheaf with the above two-term complex (which are equivalent in the derived category. Note that the $k$ in D$k$-brane refers to the (real) dimension of the support.
There is a discussion of the meaning of the motivic DT invariants in physics in the paper "Refined, Motivic, and Quantum" by Dimofte and Gukov (http://arxiv.org/pdf/0904.1420) where the basic claim is that the motivic invariants and the "refined" BPS state counts are the same. "Refined" here refers to the way you count BPS states. BPS states are certain kinds of representations of the super-Poincare algebra and "counting" means just finding the dimension of these representations (I think that little book on super-symmetry by Dan Freed has a good mathematical discussion of this). Sitting inside the super-Poincare algebra is a copy of $\mathfrak{sl}_2\oplus\mathfrak{sl}_2$ and normally one looks at the action of the diagonal $\mathfrak{sl}_2$ on the space of BPS representations and finds the dimensions of the irreducibles, for the "refined" count, you look at both copies of $\mathfrak{sl}_2$. The generating function for the dimensions of these representations thus gets an extra variable which is suppose to correspond to the Lefschetz motive $\mathbb{L}$ in the motivic invariants.
As for the DT/GW correspondence, I'm afraid that I don't really understand the physicist's explanations. There is a few paragraphs in MNOP (presumably written by Nekrasov) about it and I think that physicists regard it as well understood, but I haven't found something that I can understand. Let me know if you do.