To explain one aspect of the context of that interaction: Wigner was a very senior faculty member, had won a Nobel Prize, and was in his 60s. Shimura was a very young full professor.
Another aspect: Wigner's 1939 paper on the representation theory of (in effect) $SO(2,1)$, written to address issues of quantum mechanics, was the first substantive result on representation theory of non-abelian, non-compact groups. The 1947 paper by V. Bargmann, a physicist, was the second. No further progress on this until the 1950s when Harish-Chandra, a student of the physicist Dirac, began his systematic study of repn theory of semi-simple and reductive (Lie, and eventually, p-adic) groups.
Meanwhile, Shimura had almost single-handedly resurrected the arithmetic and algebraic-geometry aspects of holomorphic modular forms on higher rank groups, although Klingen and Hel Braun (a student of Siegel) had been ("quietly"?) working on the complex-analytic aspects, and Klingen's c. 1960 discussion of special values of (abelian) L-functions over number fields was very arithmetical. Perhaps Shimura's most special "early" contribution was to the possibility of expressing Hasse-Weil zeta functions of "Shimura curves" (as they are now known, a class generalizing "modular curves") as Mellin transforms of automorphic forms of some sort, etc.
Even the plausibility, of the Taniyama-Shimura conjecture would not be acknowledged by Weil until the mid-1960's, after his work on converse theorems. People then, and until the Wiles-Taylor work and others' in the mid-to-late 1990s, I think thought RH would be proven before Taniyama-Shimura. No one had any idea about RH, but they had even fewer [sic] ideas about Taniyama-Shimura.
Wigner would not have known about Weil's conjectures, nor the nascent algebraic geometry required to put them in any context. Shimura might not have believed that Harish-Chandra's repn theory, beginning with Wigner's result, would, as explained by Gelfand and his school, and Selberg, and taken up by Langlands et al, provide an over-arching context for not-necessarily-holomorphic automorphic forms, if not their "arithmetic".
The other "human" aspects of the situation we can imagine easily...
But, even beyond the human-foible aspect, it is absolutely not surprising that Shimura was not in awe of Wigner, and that Wigner had no reason to care much about Shimura's work.
Witten's interest was quite a few years later, after Shimura's, Selberg's, Harish-Chandra's, Langlands', and many others' work had made clear that the special objects studied in "number theory" strongly resembled the special objects of parts of physics. Not to mention that Witten is more of a "visionary" than many of us. And won a Fields Medal, so maybe he's a good mathematician, too? :)
From my personal viewpoint, apart from those historical observations, I note that the specific mathematics on arXiv that seems relevant to my concerns, second after "number theory", is the "math-ph" section.
As an example, the van Hove (et al) differential equations that (I hear...) model something about graviton interactions, are precisely the same genre of differential equations "in automorphic forms" that appear in various spectral-theoretic scenarios, going back to Anton Good's papers in the early 1980's, and continuing in various peoples' work today. Steven Miller at Rutgers, a guy who "does" automorphic forms, has actively collaborated with that physics groups, for example.
Indeed, Rudnick, Ueberschar, Marklof, and their collaborators often say that they are doing "mathematical physics", and are in "physics institutes", ... but their work looks to me like a study of number-theoretic aspects of harmonic analysis... which would extend to be "automorphic forms", if taken on to more difficult cases.
And, finally, probably autobiographies do not reliably involve scholarly reconsideration of much of anything at all, as they are reminiscences... so scientific accuracy is by far not guaranteed.
It is true there is a lot of high level math in physics, some of which is done by physicists and some by mathematicians. For example the professor I am working with works with (and writes papers with) physicists very often, but the work he does is really math because it comes from a field called algebraic geometry which depends highly on pure math fields like commutative algebra and topology.
What you major in ultimately depends what you are most interested in. I am writing this from a pure math perspective, and I switched into math (from physics) because I liked the abstraction and the rigor with which everything is done. In a pure math degree almost everything you do is proofs. This is where you will take courses like number theory (or topology, group theory, analysis etc).
That being said my friends who remained in physics say that higher level physics has a lot more proofs than in the first couple of years, but they are not as abstract as the sort of proofs you would find in a mathematics course. The math courses you take with a physics degree are mostly calculus, linear algebra, differential equations. From what my friends say often the more advanced math in a physics degree is learned in physics courses (so its by nature an application) with just the introduction to the subjects in the math courses. In my university there is also an option to take a mathematical physics degree which would also allow you to take some analysis, which is more like the courses one would take in a math degree.
I unfortunately cannot say much for the kind of math you would learn in an applied math degree. The math courses you would take are similar to those you would take in a physics degree; however you would take higher level courses in these areas. From what I gather, you would learn much more of the theory behind these subject areas, which you don't necessarily get to do in a physics degree. I leave it to an applied mathematician to say more.
Despite my long answer, all in all I think the biggest differences between the math learned in math and physics, is that the math in math is very abstract, proofy, and often taught without any non-math applications. The math in physics is done always with a sense of application, but the mathematical rigor is not always taught. I would recommend taking some courses in both areas to see what interests you more.
Best Answer
I don't know if this is an option for you, but I would suggest you do both. At my university (Utrecht University in the Netherlands) there is a combined mathematics/physics course. Because it is heavier than the separate mathematics and physics courses, it is a good preparation for a Ph.D. scholarship in either discipline. Moreover, when you realize that mathematics is the superior discipline, you can always drop physics ;-)