This is a great question I wish I understood the answer to better.
I know two vague answers, one based on derived algebraic geometry and one based on string theory.
The first answer, that Costello explained to me and I most likely misrepeat,
is the following. The B-model on a CY X as an extended TFT can be defined in terms of
DAG: we consider the worldsheet $\Sigma$ as merely a topological space or simplicial set (this is a reflection of the lack of instanton corrections in the B-model), and consider the mapping space $X^\Sigma$ in the DAG sense. For example for $\Sigma=S^1$ this is the derived loop space (odd tangent bundle) of $X$.. In this language it's very easy to say what the theory assigns to 0- and 1-manifolds: to a point we assign coherent sheaves on $X$, to a 1-manifold cobordism we assign the functor given by push-pull of sheaves between obvious maps of mapping spaces (see e.g. the last section here). For example for $S^1$ we recover Hochschild homology of $X$. Now for 2-manifold bordisms we want to define natural operations by push-pull of functions, but for that we need a measure -- and the claim is the Calabi-Yau structure (together with the appropriate DAG version of Grothendieck-Serre duality, which Kevin said Lurie provides) gives exactly this integration...
Anyway that gives a tentative answer to your question: the B-model assigns to a surface $\Sigma$ the "volume" of the mapping space $X^\Sigma$, defined in terms of the CY form.
More concretely, you chop up $\Sigma$ into pieces, and use the natural operations on Hochschild homology, such as trace pairing and identification with Hochschild cohomology (and hence pair-of-pants multiplication).. of course this last sentence is just saying "use the Frobenius algebra structure on what you assigned to the circle" so doesn't really address your question - the key is to interpret the volume of $X^\Sigma$ correctly.
The second answer from string theory says that while genus 0 defines a Frobenius manifold you shouldn't consider other genera individually, but as a generating series -- i.e. the genus is paired with the (topological) string coupling constant, and together defines a single object, the topological string partition function, which you should try to interpret rather than term by term. (This is also the topic of Costello's paper on the partition function). BTW for genus one there is a concrete answer in terms of Ray-Singer torsion, but I don't think that extends obviously to higher genus.
As to how to interpret it, that's the topic of the famous BCOV paper - i.e. the Kodaira-Spencer theory of gravity. For one thing, the partition function is determined recursively by the holomorphic anomaly equation, though I don't understand that as "explaining" the higher genus contributions. But in any case there's a Chern-Simons type theory quantizing the deformation theory of the Calabi-Yau, built out of the Kodaira-Spencer dgla in a simple looking way, and that's what the B-model is calculating.
A very inspiring POV on this is due to Witten, who interprets the entire partition function as the wave function in a standard geometric quantization picture for the middle cohomology of the CY (or more suggestively, of the moduli of CYs). This is also behind the Givental quantization formalism for the higher genus A-model, where the issue is not defining the invariants
but finding a way to calculate them.
Anyway I don't know a totally satisfactory mathematical formalism for the meaning of this partition function (and have tried to get it from many people), so would love to hear any thoughts. But the strong message from physics is that we should try to interpret this entire partition function - in particular it is this function which appears in a million different guises under various dualities (eg in gauge theory, as solution to quantum integrable systems, etc etc...)
For fixed integers $g,n$, any projective scheme $X$ over a field $k$, and a linear map $\beta:\operatorname{Pic}(X)\to\mathbb Z$, the space $\overline{M}_{g,n}(X,\beta)$ of stable maps is well defined as an Artin stack with finite stabilizer, no matter the characteristic of $k$. You can even replace $k$ by $\mathbb Z$ if you like.
Now if $X$ is a smooth projective scheme over $R=\mathbb Z[1/N]$ for some integer $N$, then $\overline{M}_{g,n}(X,\beta) \times_R \mathbb Z/p\mathbb Z$ is a Deligne-Mumford stack for almost all primes $p$. For such $p$, $\overline{M}_{g,n}(X,\beta) \times_R \mathbb Z/p\mathbb Z$ has a virtual fundamental cycle, and so you have well-defined Gromov-Witten invariants. This holds for all but finitely many $p$. Nothing about $\mathbb C$ here, that is my point, the construction is purely algebraic and very general.
It is when you say "Hodge structures" then you better work over $\mathbb C$, unless you mean $p$-Hodge structures.
As far as mirror symmetry in characteristic $p$, much of it is again characteristic-free. For example Batyrev's combinatorial mirror symmetry for Calabi-Yau hypersurfaces in toric varieties is simply the duality between reflexive polytopes. You can do that in any characteristic, indeed over $\mathbb Z$ if you like.
Best Answer
Here is a very rough answer.
The Gromov-Witten invariants show up in a few a priori different contexts within string theory. Let me focus on one particular place they show up that is directly related to conventional physics, as opposed to topological quantum field theory.
Type IIA string theory is formulated on a spacetime "background" which is, in the simplest setup, just a Lorentzian 10-manifold. The equations of motion of the theory require (at least in their leading approximation) that the metric on this 10-manifold should be Ricci-flat.
A popular thing to do is to take this 10-manifold of the form X x R^{3,1}, where X is a compact Calabi-Yau threefold.
We can simplify matters by taking X to be very small --- smaller than the Compton wavelength of any of the particles we are able to create. (Remember that in quantum mechanics particles have a wavelike character, with wavelength inversely related to their energy; since we only have limited energy available to us, we can't make particles with arbitrarily short wavelength.) A little more precisely, let's take X such that the first nonzero eigenvalue of the Laplacian is larger than the energy scale we can access.
In this case we low-energy observers will not be able to detect X directly in any experiments. To us, spacetime will appear to be R^{3,1}. What will be the physics we see on this R^{3,1}? We will see various different species of particle. Each species of particle that we see corresponds to some zero-mode of the Laplacian of X. In particular, there are particles corresponding to classes in H^{1,1}(X).
The genus 0 Gromov-Witten invariants are giving information about the interactions between these particles. (So if you want to calculate what will come out when you shoot two of these particles at each other, one of the inputs to that calculation would be the genus 0 Gromov-Witten invariants.) The higher genus Gromov-Witten invariants are giving information about interactions which involve these particles together with other particles related to the gravitational interaction.