If you really know nothing about physics I suggest you begin with any text book on physics for undergrad. Easy to read, it will introduce the main usual suspects. After, you'll ask again :)
I am not sure that jumping from nothing to quantum mechanics, or even worse quantum fields theory, would be wise, like jumping from nothing in math to algebraic geometry or K-Theory.
After that, it depends of course at what level of mathematical physics you want to stop. I will illustrate this with some examples:
Question: What is the "mass" of an isolated dynamical system?
Math Answer: It is the class of cohomology of the action of the group of Galilee, measuring the lack of equivariance of the moment map, on a symplectic manifold representing the isolated dynamical system.
Another question: Why in general relativity $E = mc^2$?
Math Answer: Because the group of Poincaré has no cohomology
Another, other question: What is the theorem of decomposition of motions around the center of gravity?
Math Answer: Let $(M,\omega)$ be a symplectic manifold with an hamiltonian action of the group of Galilee, if the "mass" of the system is not zero (in the sense above) then $M$ is the symplectic product or $({\bf R}^6, {\rm can})$, representing the motions of the center of gravity, by another symplectic manifold $(M_0,\omega_0)$, representing the motions around the center of gravity. The group of Galillee acting naturally on $\bf R^6$ and $SO(3) \times {\bf R}$ on $M_0$.
Another, other, other question: What are the constants of motions?
Math Answer: Let $(M,\omega)$ be a pre-symplectic manifold with an hamiltonian action of a Lie group $G$, then the moment map is constant on the characteristics of $\omega$, that is the integral manifolds of the vector distribution $x \mapsto \ker(\omega_x)$.
These answers are the mathematical versions of physics classical constructions, but it would be very difficult to appreciate them if you have no pedestrian introduction of physics. You may enjoy also Aristotles' book "Physics", as a first dish, just for tasting the flavor of physics :)
After that, you will be able to appreciate also quantum mechanics, but this is another question.
Addendum
Just before entering in the modern world of physics I would suggest few basic lectures for the winter evenings, near the fireplace (I'm sorry I write them down in french because I read them in french).
• Platon, Timée, trad. Émile Chambry.
• Aristote, La Physique, Éd. J. Vrin.
• Maïmonide, Le Guide des Égarés, Éd. Maisonneuve & Larose. (the part about time as an accident of motion, accident of the thing. Very deep and modern thoughts).
• Giordano Bruno, Le Banquet des Cendres, Éd. L’éclat.
• Galileo Galilei, Dialogue sur les Deux Grands Systèmes du Monde, Éd. Points.
• Albert Einstein, La Relativité, Éd. Payot.
• Joseph-Louis Lagrange, Mécanique Analytique, Éd. Blanchard.
• Felix Klein, Le Programme d’Erlangen, Éd. Gauthier-Villars.
• Jean-Marie Souriau, Structure des Systèmes Dynamiques, Éd. Dunod.
• Victor Guillement & Shlomo Sternberg, Geometric Asymptotics, AMS Math Books
• François DeGandt Force and Geometry in Newton Principia.
If you allow such a comprehensive reference to re-introduce basic mathematics, then either as a layman or a working mathematician your prayers are answered by the following (he even prefaces by saying that his intended layman-audience must have some mathematical sophistication):
The Road to Reality: A Complete Guide to the Laws of the Universe, by Roger Penrose
Now let's try to break down the subjects.
Classical Mechanics:
1) Mathematical Methods of Classical Mechanics, by Arnold
2) A Mathematical Introduction to Fluid Mechanics, by Chorin-Marsden
Quantum Mechanics:
1) Mathematical Foundations of Quantum Mechanics, by Mackey
2) The Theory of Groups and Quantum Mechanics, by Weyl
General Relativity:
1) General Relativity for Mathematicians, by Sachs-Wu
2) The Large Scale Structure of Space-Time, by Hawking-Ellis
Electrodynamics:
1) Electromagnetic Theory and Computation: A Topological Approach, by Gross-Kotiuga
2) On the Mathematical Foundations of Electrical Circuit Theory, by Smale
3) This is a plug for Gauge theory:
3a) On Some Recent Developments in Yang-Mills Theory, by Bott
3b) On Some Recent Interactions Between Mathematics and Physics, by Bott
3c) Concept of Nonintegrable Phase Factors and Global Formulation of Gauge Fields, by Wu-Yang
3d) From Superconductors and Four-Manifolds to Weak Interactions, by Witten
Standard Model:
The Algebra of Grand Unified Theories, by Baez-Huerta
Quantum Field Theory and String Theory:
1) Quantum Physics: A Functional Integral Point of View, by Jaffe-Glimm
2) Geometry and Quantum Field Theory, 1994 IAS lectures
3) Quantum Fields and Strings: A Course for Mathematicians, 1996 IAS lectures
Best Answer
The short but ahistorical answer is that topological string theories turn out to be examples of $(\infty,1)$-categories. The mathematical formulation of this statement is in Lurie's classification of topological field theories http://www.math.harvard.edu/~lurie/papers/cobordism.pdf (building on work of Atiyah, Segal, Getzler, Costello, Baez-Dolan, Kontsevich and probably a bunch more I'm forgetting.)
The content of this statement is that when you write down the axioms for a topological string theory, the collection of "boundary conditions" or "D-branes" look like the collection of objects in an $(\infty,1)$ category.
Of course, you can ask why the derived category of coherent sheaves. Historically, the answer to that is that it is very easy to write down a boundary condition for a holomorphic vector bundle in the topological B-model. It's not a huge leap from there to coherent sheaves, and if you start mumbling words like tachyon condensation, you can get to the derived category with a fair bit of hand waving.
That's from the physics side of things. On the math side, Kontsevich got there first, possibly by noting that the space of closed string states in the B-model ($H^\bullet(\wedge^\bullet TX)$) is exactly the Hochschild coohomology of the derived category of coherent sheaves. He then followed up by associating the (still not yet defined?) Fukaya category with the A-model and conjecturing that mirror symmetry is an equivalence of the two (with some Hodge structure goodies thrown in). Subsequently, it looks like you have to add in some things called coisotropic branes to cover all your bases, but the basic idea is right.
Kontsevich formulated all this in terms of $A_\infty$ categories which in the Lurie language turn into $(\infty,1)$ categories which are just TQFTs in disguise. So, Kontsevich's homological mirror symmetry is then the statement that two TQFTs are the same, just like mirror symmetry in string theory.
From the physics side of things, this was all a bit of a mess, but we now understand that the derived category really arises via Block's construction of the derived category (I'm being intentionally vague as to which version of the derived category) as arising from integrable super-connections of graded smooth vector bundles http://www.math.upenn.edu/~blockj/papers/BottVolume.pdf. You can see this explicitly in the physics from a few sources, particularly Kapustin, Rozansky and Saulina, and Herbst, Hori and Page, but I'm rather fond of my own contribution http://arxiv.org/abs/0808.0168.