Take two vector spaces $V$ and $W$ over a field $F$. One may form the tensor product $V\otimes W$ and it fulfills an universal property. Elements of $V\otimes W$ are called tensors and they are linear combinations of elementary tensors $v\otimes w$, the elementary tensors generate $V\otimes W$.
People from physics think of a tensor as a generalization of scalars, vectors, and matrices, I think and I have seen them tensoring matrices with matrices as entries with matrices and so on.
What does this mean and what has it to do with the definition from above? What "is" a tensor?
Best Answer
I have heard it said that tensor products are the hardest thing in mathematics. Of course that's not really true, but certainly a fluent understanding of how to work with tensor products is one of the dividing lines in your education from basic to advanced mathematics.
Disclaimer: What I will discuss here are tensor products in the sense of linear algebra, so only tensor products of individual vector spaces rather than tensor fields (which is what the physicists mean by tensor product).
For a long time I could not understand how the physicists could work with tensors by thinking about them as "quantities that transform in a certain way under a change in coordinates". The only way I could come to terms with them is by their characterization as something that satisfies a universal mapping property. Do not think about what tensors (elements of a tensor product space) are but rather what the whole construction of a tensor product space can do for you. It's sort of like quotient groups (only harder), where if you focus all your energy on trying to understand cosets you kind of miss the point of quotient groups. What makes tensor product spaces harder to come to terms with than quotient groups is that most elements of a tensor product space are not elementary tensors $v \otimes w$ but only sums of these things.
The whole (mathematical) point of tensor products of vector spaces is to linearize bilinear maps. A bilinear map is a function $V \times W \rightarrow U$ among $F$-vector spaces $V, W$, and $U$ which is linear in each coordinate when the other one is kept fixed. There are tons of bilinear maps in mathematics, and if we can turn them into linear maps then we can use constructions related to linear maps on them. The tensor product $V \otimes_F W$ of two $F$-vector spaces provides the most extreme space, so to speak, which is a domain for the linearization of all bilinear maps out of $V \times W$ into all vector spaces (over $F$). It is a particular vector space together with a particular bilinear map $V \times W \rightarrow V \otimes_F W$ such that any bilinear map out of $V \times W$ into any vector space naturally (!) gets converted into a linear map out of this new space $V \otimes_F W$. Some notes I wrote on tensor products for an algebra course are at http://www.math.uconn.edu/~kconrad/blurbs/linmultialg/tensorprod.pdf, and in it I address questions like "what does $v \otimes w$ mean?" and "what does it mean to say $$ v_1 \otimes w_1 + \cdots + v_k \otimes w_k = v_1' \otimes w_1' + \cdots + v_k' \otimes w_k'?" $$ Right from the start I allow tensor products of modules over a ring, not just vector spaces over a field. There are some aspects of tensor products which appear in the wider module context that don't show up for vector spaces (particularly since modules need not have bases). So you might want to skip over, say, tensor products involving ${\mathbf Z}/m{\mathbf Z}$ over $\mathbf Z$ on a first pass if you don't know about modules.
As for the question of how tensor products generalize scalars, vector spaces, and matrices, this comes from the natural (!) isomorphisms $$ F \otimes_F F \cong F, \ \ \ F \otimes_F V \cong V, \ \ V \otimes_F V^* \cong {\rm Hom}_F(V,V). $$ On the left side of each isomorphism is a tensor product of $F$-vector spaces, and on the right side are spaces of scalars, vectors, and matrices. In the link I wrote above, see Theorems 4.3, 4.5, and 5.9. You can also tensor two matrices as a particular example in a tensor product of two spaces of linear maps. Spaces of linear maps are vector spaces (with some extra structure to them), so they admit tensor products as well (with some extra features).
Returning to the physicist's definition of tensors as quantities that transform by a rule, what they always forget to say is "transform by a multilinear rule". I discuss the transition between tensor products from the viewpoint of mathematicians and physicists in section 6 of a second set of notes at http://www.math.uconn.edu/~kconrad/blurbs/linmultialg/tensorprod2.pdf.