I have scoured all the answers on this website but I still cannot understand why $a\cdot b = |a||b|\cos\theta$, if the dot product is interpreted as the amount of one vector, say $a$, in the same direction as the other, say $b$, then why do we scale it by multiplying it with the magnitude of the vector on which it projects? why do we need $|b|$?
Why is the vector dot product scaled
vectors
Related Solutions
Write $a \cdot b = ( |a|) ( |b| \cos \theta)$
Then consider 
The projection of B onto A is of length $|b| \cos \theta$, it is the portion of the green line up to the dotted line perpendicular to it (or I suppose beyond it). Then the dot product is the just magnitude of these two vectors, $a$ and $b\cos \theta$, multiplied.
Note: I have interchanged $a$ and $A$, $b$ and $B$ due to the diagram and your question.
Have a look in the preview of this book, Section 1.2 has a good explanation of the dot product:
Let $\vec{\mathbf a} = (x_1, y_1) = (a \cos \alpha, a \sin \alpha)$
Let $\vec{\mathbf b} = (x_2, y_2) = (b \cos \beta, b \sin \beta)$
Then $\theta = |\beta - \alpha|$
By definition,
\begin{align} \vec{\mathbf a} \circ \vec{\mathbf b} &= x_1x_2 + y_1y_2 \\ &= ab(\cos \alpha \cos \beta + \sin \alpha \sin \beta) \\ &= ab \cos(\beta - \alpha)\\ &= ab \cos \theta \end{align}
(Note $\cos(\theta) = \cos(-\theta)$)
Related Question
- [Math] Why is vectors A.B expressed in the form of $a_xb_x +a_yb_y +a_zb_z$ in the dot product
- [Math] Distance between point and plane – why use the dot product
- [Math] Why is the dot product of perpendicular vectors zero
- Vector Spaces – Why Isn’t the Dot Product of Two Identical Vectors Equal to the Magnitude?
Best Answer
To understand how the dot product is defined, it's better to first look at why the dot product is defined. The idea of the dot product is to have some operation which takes in two vectors (say $a$ and $b$), and returns a single (meaningful) value. As you suggest, if we were merely interested in the length of the projection of $a$ onto $b$ (or, as you put it, the amount of one vector goes in the same direction as the other), then we would indeed want to use $|a|\cos\theta$. But there are several problems with this ''definition''.
Indeed, let's say we did define such a product: say $a\star b=|a|\cos\theta$. Observe this ''product'' has the following problems:
We do not have $a\star b=b\star a$. In other words, the product is not commutative.
We do not have $a\star (b+c)=a\star b+a\star c$, it is not distributive.
We do not have $a\star (xb)=x(a\star b)$ for $x\in \mathbb{R}$, it does not respect scalar multiplication.
That is why defining $a\cdot b=|a||b|\cos\theta$ is a much more natural definition. It satisfies the properties we want a usual product to satisfy, and hence tells us more about the vectors $a$ and $b$.