This is a great question I wish I understood the answer to better.
I know two vague answers, one based on derived algebraic geometry and one based on string theory.
The first answer, that Costello explained to me and I most likely misrepeat,
is the following. The B-model on a CY X as an extended TFT can be defined in terms of
DAG: we consider the worldsheet $\Sigma$ as merely a topological space or simplicial set (this is a reflection of the lack of instanton corrections in the B-model), and consider the mapping space $X^\Sigma$ in the DAG sense. For example for $\Sigma=S^1$ this is the derived loop space (odd tangent bundle) of $X$.. In this language it's very easy to say what the theory assigns to 0- and 1-manifolds: to a point we assign coherent sheaves on $X$, to a 1-manifold cobordism we assign the functor given by push-pull of sheaves between obvious maps of mapping spaces (see e.g. the last section here). For example for $S^1$ we recover Hochschild homology of $X$. Now for 2-manifold bordisms we want to define natural operations by push-pull of functions, but for that we need a measure -- and the claim is the Calabi-Yau structure (together with the appropriate DAG version of Grothendieck-Serre duality, which Kevin said Lurie provides) gives exactly this integration...
Anyway that gives a tentative answer to your question: the B-model assigns to a surface $\Sigma$ the "volume" of the mapping space $X^\Sigma$, defined in terms of the CY form.
More concretely, you chop up $\Sigma$ into pieces, and use the natural operations on Hochschild homology, such as trace pairing and identification with Hochschild cohomology (and hence pair-of-pants multiplication).. of course this last sentence is just saying "use the Frobenius algebra structure on what you assigned to the circle" so doesn't really address your question - the key is to interpret the volume of $X^\Sigma$ correctly.
The second answer from string theory says that while genus 0 defines a Frobenius manifold you shouldn't consider other genera individually, but as a generating series -- i.e. the genus is paired with the (topological) string coupling constant, and together defines a single object, the topological string partition function, which you should try to interpret rather than term by term. (This is also the topic of Costello's paper on the partition function). BTW for genus one there is a concrete answer in terms of Ray-Singer torsion, but I don't think that extends obviously to higher genus.
As to how to interpret it, that's the topic of the famous BCOV paper - i.e. the Kodaira-Spencer theory of gravity. For one thing, the partition function is determined recursively by the holomorphic anomaly equation, though I don't understand that as "explaining" the higher genus contributions. But in any case there's a Chern-Simons type theory quantizing the deformation theory of the Calabi-Yau, built out of the Kodaira-Spencer dgla in a simple looking way, and that's what the B-model is calculating.
A very inspiring POV on this is due to Witten, who interprets the entire partition function as the wave function in a standard geometric quantization picture for the middle cohomology of the CY (or more suggestively, of the moduli of CYs). This is also behind the Givental quantization formalism for the higher genus A-model, where the issue is not defining the invariants
but finding a way to calculate them.
Anyway I don't know a totally satisfactory mathematical formalism for the meaning of this partition function (and have tried to get it from many people), so would love to hear any thoughts. But the strong message from physics is that we should try to interpret this entire partition function - in particular it is this function which appears in a million different guises under various dualities (eg in gauge theory, as solution to quantum integrable systems, etc etc...)
I know that you are thinking firmly about the integrable world, but I thought it worth adding that for symplectic manifolds, there is no obvious generalisation of Gromov-Witten theory to higher dimensional subvarieties. This is because to define "holomorphic" you use a non-integrable almost complex structure and non-integrability means that there are no higher dimensional holomorphic objects. The fact that there are holomorphic curves can be thought of as an instance of the fact that all almost complex structures over 2-manifolds are automatically integrable. (E.g., since there are no (2,0)-forms, the space where the Nijenhuis tensor should live is zero.)
Best Answer
OK, I guess, your first question is addressed to me. The answer is: fixed point localization. In my paper "Elliptic Gromov-Witten invariants and the mirror conjecture", a formula is found for the genus-1 (no descendants) potential of a semisimple target. It is a theorem, discovered and proved by fixed point localization when a torus acts on the target with isolated fixed points, and the GW-invariants are understood as equivariant ones. Since the answer is expressed in genus-0 data making sense for any semisimple Frobenius manifold, the conjectural extension to all such manifold is immediate. (The conjecture was proved by Dubrovin-Zhang in the sense that they showed my formula being the only candidate that would satisfy Getzler's relation.) The paper of mine you are asking about, "Semisimple Frobenius structures at higher genus", does exactly the same that the elliptic paper, but for higher genus GW-invariants, first without, and then with gravitational descendants.
After the fact, there is a more satisfying description of how that formula could have been invented. Dubrovin's connection of a semisimple Frobenius manifod allows for an asymptotical fundamental solution (which looks like the complete stationary phase asymptotics of oscillating integrals on the mirror theory). It's construction ("the $R$-matrix") is contained in the key lemma in that elliptic paper I've mentioned. Another way to interpret this solution is to say - in terms of overruled Lagrangian cones in symplectic loop spaces as the objects that describe genus-0 theory in lieu of Frobenius structures - that the overruled Lagrangian cone of a semisimple Frobenius manifold is isomorphic to the Cartesian product of several such cones corresponding to the one-point target space, and moreover, the isomorphism is accomplished by transformations from the twisted loop group: $$ L = M (L_{pt}\times \cdots \times L_{pt}).$$ The "mysterious" conjectural higher genus formula simply says that the same relation persists for the total descendant potentials of higher genus theory:$$D \sim \hat{M} (D_{pt}\otimes\cdots\otimes D_{pt}),$$ where the elements of the loop group are quantized, and the equality is replaced by proportionality up to a non-zero "central constant".