As Richard Thomas has written (we paraphrase just slightly), mathematical physicists Vafa and Witten introduced new "invariants" of four-dimensional spaces in a paper:
A Strong Coupling Test of S-Duality (1994),
Cumrun Vafa, Edward Witten
https://arxiv.org/abs/hep-th/9408074
Nucl.Phys.B431:3-77,1994
These invariants "count" solutions of a certain equation (the N=4 supersymmetric Yang-Mills equations) over the four dimensional space, and should tell us (both physicists and mathematicians) something about the space. There is one for every integer charge of the Yang-Mills field.
Motivated by a generalization of electromagnetic duality in string theory, Vafa and Witten predicted that on a fixed space, one could put all these invariants together in a generating series (a Taylor series or Fourier series, with coefficients the Vafa-Witten invariants) and get a very special function called a "modular form". In particular, the invariants should have hidden symmetries that mean that only a finite number of them determine all the rest.
If I was told correctly, until now mathematicians have been unable to make sense of how this Vafa-Witten "counting" should be done without getting infinity.
question 1: So what are Vafa-Witten invariants meant for mathematicians in your research fields or subfields? (questionable since the mathematical "counting" so far involves getting infinity.)
question 2: Are there similar invariants describing "topologically twisted maximally supersymmetric 5d Yang-Mills theory"?
Best Answer
I'm not trying to give an answer, I just want to make a few remarks.
question 2: Are there similar invariants describing "topologically twisted maximally supersymmetric 5d Yang-Mills theory"? No. A crucial aspect of the Vafa-Witten computation is supersymmetry. For supersymmetry to exist, a globally covariant section of the spinor bundle (the quare root of the canonical bundle) is needed; in other words, the manifold should be a local model of some even-dimensional Calabi-Yau manifold.
Five dimensional supersymmetric theories exist. But always embedded on higher dimensional spaces.
question 1: So what are Vafa-Witten invariants meant for mathematicians in your research fields or subfields? (questionable since the mathematical "counting" so far involves getting infinity.)
You can think on the Vafa-Witten theory as a lower dimensional analog of the Donaldson-Thomas theory. Indeed, the cases explicitly analized in his famous paper were choosen because on those cases, the Vafa-Witten partition function localizes to the Euler characteristic of the moduli space of instantons for the relevant twisted version of $N=4$ super Yang-Mill theory. In mathematical terms that means that aforementioned invariants provide an enummerative theory of ideal sheaves on a given 4-manifold with spin structure.
For an explicit computation of the Vafa-Witten invariants as "lower dimensional analogue" of the DT ones see: Crystals and intersecting branes.