This is a great question I wish I understood the answer to better.
I know two vague answers, one based on derived algebraic geometry and one based on string theory.
The first answer, that Costello explained to me and I most likely misrepeat,
is the following. The B-model on a CY X as an extended TFT can be defined in terms of
DAG: we consider the worldsheet $\Sigma$ as merely a topological space or simplicial set (this is a reflection of the lack of instanton corrections in the B-model), and consider the mapping space $X^\Sigma$ in the DAG sense. For example for $\Sigma=S^1$ this is the derived loop space (odd tangent bundle) of $X$.. In this language it's very easy to say what the theory assigns to 0- and 1-manifolds: to a point we assign coherent sheaves on $X$, to a 1-manifold cobordism we assign the functor given by push-pull of sheaves between obvious maps of mapping spaces (see e.g. the last section here). For example for $S^1$ we recover Hochschild homology of $X$. Now for 2-manifold bordisms we want to define natural operations by push-pull of functions, but for that we need a measure -- and the claim is the Calabi-Yau structure (together with the appropriate DAG version of Grothendieck-Serre duality, which Kevin said Lurie provides) gives exactly this integration...
Anyway that gives a tentative answer to your question: the B-model assigns to a surface $\Sigma$ the "volume" of the mapping space $X^\Sigma$, defined in terms of the CY form.
More concretely, you chop up $\Sigma$ into pieces, and use the natural operations on Hochschild homology, such as trace pairing and identification with Hochschild cohomology (and hence pair-of-pants multiplication).. of course this last sentence is just saying "use the Frobenius algebra structure on what you assigned to the circle" so doesn't really address your question - the key is to interpret the volume of $X^\Sigma$ correctly.
The second answer from string theory says that while genus 0 defines a Frobenius manifold you shouldn't consider other genera individually, but as a generating series -- i.e. the genus is paired with the (topological) string coupling constant, and together defines a single object, the topological string partition function, which you should try to interpret rather than term by term. (This is also the topic of Costello's paper on the partition function). BTW for genus one there is a concrete answer in terms of Ray-Singer torsion, but I don't think that extends obviously to higher genus.
As to how to interpret it, that's the topic of the famous BCOV paper - i.e. the Kodaira-Spencer theory of gravity. For one thing, the partition function is determined recursively by the holomorphic anomaly equation, though I don't understand that as "explaining" the higher genus contributions. But in any case there's a Chern-Simons type theory quantizing the deformation theory of the Calabi-Yau, built out of the Kodaira-Spencer dgla in a simple looking way, and that's what the B-model is calculating.
A very inspiring POV on this is due to Witten, who interprets the entire partition function as the wave function in a standard geometric quantization picture for the middle cohomology of the CY (or more suggestively, of the moduli of CYs). This is also behind the Givental quantization formalism for the higher genus A-model, where the issue is not defining the invariants
but finding a way to calculate them.
Anyway I don't know a totally satisfactory mathematical formalism for the meaning of this partition function (and have tried to get it from many people), so would love to hear any thoughts. But the strong message from physics is that we should try to interpret this entire partition function - in particular it is this function which appears in a million different guises under various dualities (eg in gauge theory, as solution to quantum integrable systems, etc etc...)
For fixed integers $g,n$, any projective scheme $X$ over a field $k$, and a linear map $\beta:\operatorname{Pic}(X)\to\mathbb Z$, the space $\overline{M}_{g,n}(X,\beta)$ of stable maps is well defined as an Artin stack with finite stabilizer, no matter the characteristic of $k$. You can even replace $k$ by $\mathbb Z$ if you like.
Now if $X$ is a smooth projective scheme over $R=\mathbb Z[1/N]$ for some integer $N$, then $\overline{M}_{g,n}(X,\beta) \times_R \mathbb Z/p\mathbb Z$ is a Deligne-Mumford stack for almost all primes $p$. For such $p$, $\overline{M}_{g,n}(X,\beta) \times_R \mathbb Z/p\mathbb Z$ has a virtual fundamental cycle, and so you have well-defined Gromov-Witten invariants. This holds for all but finitely many $p$. Nothing about $\mathbb C$ here, that is my point, the construction is purely algebraic and very general.
It is when you say "Hodge structures" then you better work over $\mathbb C$, unless you mean $p$-Hodge structures.
As far as mirror symmetry in characteristic $p$, much of it is again characteristic-free. For example Batyrev's combinatorial mirror symmetry for Calabi-Yau hypersurfaces in toric varieties is simply the duality between reflexive polytopes. You can do that in any characteristic, indeed over $\mathbb Z$ if you like.
Best Answer
The better source for the $J$-function now would be Dubrovin-Zhang http://arxiv.org/abs/math/9808048. The idea of that is simple - the part of the genus=1 corellators comes from the three-points corellators only. From the point of view of the Frobenius manifold - from the algebra structure of it.
The advantage of the cohomology-valued function is very simple. Everything Givental writes looks to ge thought of having certein singularity theory behind. In this case it is just natural.