Many topics in linear algebra suffer from the issue in the
question. For example:
In linear algebra, one often sees the determinant of a
matrix defined by some ungodly formula, often even with
special diagrams and mnemonics given for how to compute it
in the 3x3 case, say.
det(A) = some horrible mess of a formula
Even relatively sophisticated people will insist that
det(A) is the sum over permutations, etc. with a sign for
the parity, etc. Students trapped in this way of thinking
do not understand the determinant.
The right definition is that det(A) is the volume of the
image of the unit cube after applying the transformation
determined by A. From this alone, everything follows. One
sees immediately the importance of det(A)=0, the reason why
elementary operations have the corresponding determinant,
why diagonal and triangular matrices have their
determinants.
Even matrix multiplication, if defined by the usual
formula, seems arbitrary and even crazy, without some
background understanding of why the definition is that way.
The larger point here is that although the question asked about having a single wrong definition, really the problem is that a limiting perspective can infect one's entire approach to a subject. Theorems,
questions, exercises, examples as well as definitions can be coming
from an incorrect view of a subject!
Too often, (undergraduate) linear algebra is taught as a
subject about static objects---matrices sitting there,
having complicated formulas associated with them and
complex procedures carried out with the, often for no
immediately discernible reason. From this perspective, many
matrix rules seem completely arbitrary.
The right way to teach and to understand linear algebra is as a fully dynamic
subject. The purpose is to understand transformations of
space. It is exciting! We want to stretch space, skew it,
reflect it, rotate it around. How can we represent these
transformations? If they are linear, then we are led to
consider the action on unit basis vectors, so we are led
naturally to matrices. Multiplying matrices should mean
composing the transformations, and from this one derives
the multiplication rules. All the usual topics in
elementary linear algebra have deep connection with
essentially geometric concepts connected with the
corresponding transformations.
Best Answer
One of the most important contemporary mathematical concepts without a rigorous definition is quantum field theory (and related concepts, such as Feynman path integrals).
Note: As noted in the comments below, there is a branch of pure mathematics --- constructive field theory --- devoted to making rigorous sense of this problem via analytic methods. I should add that there is also a lot of research devoted to understanding various aspects of field theory via (higher) categorical points of view. But (as far as I understand), there remain important and interesting computations that physicists can make using quantum field theoretic methods which can't yet be put on a rigorous mathematical basis.