Calculating trigonometric functions
If you want to code your own trig functions on a computer, here's some math that's useful to know.
(Co)secants and (co)tangents
Of the six commonly-taught trig functions, four of them can be defined in terms of the other two.
$$\sec x = \frac{1}{\cos x}$$
$$\csc x = \frac{1}{\sin x}$$
$$\tan x = \frac{\sin x}{\cos x}$$
$$\cot x = \frac{\cos x}{\sin x}$$
So, if you can code accurate sin and cos functions, you can trivially implement sec, csc, tan, and cot using the above formulas. So from now on, I'm just going to pretend those last functions don't exist and focus on sin and cos.
Argument reduction
sin and cos are periodic, repeating their values every 360° or 2π radians. So if, for example, you ever need to know $\sin(12345°)$, you can calculate it as $\sin(12345° \operatorname{mod} 360°) = \sin 105°$.
There's also a kind of pseudo-periodicity with half and quarter turns.
$$\sin(x \pm 180°) = -\sin x$$
$$\cos(x \pm 180°) = -\cos x$$
$$\sin(x \pm 90°) = \pm \cos x$$
$$\cos(x \pm 90°) = \mp \sin x$$
Using these identities, you can take any angle, no matter how large, and reduce the problem to a “reference angle” in Quadrant 1.
Trig identities
Recall the sum/difference formulas:
$$\sin(x \pm y) = \sin x \cos y \pm \cos x \sin y$$
$$\cos(x \pm y) = \cos x \cos y \mp \sin x \sin y$$
These can help calculate unknown function values in terms of known function values. For example,
$$\sin(15°) = \sin(45° - 30°) = \sin 45° \cos 30° - \cos 45° \sin 30° = \frac{\sqrt 2}{2}\frac{\sqrt 3}{2} - \frac{\sqrt 2}{2}\frac{1}{2} = \frac{\sqrt 6 - \sqrt 2}{4}$$
$$\cos(15°) = \cos(45° - 30°) = \cos 45° \cos 30° + \sin 45° \sin 30° = \frac{\sqrt 2}{2} \frac{\sqrt 3}{2} + \frac{\sqrt 2}{2} \frac{1}{2} = \frac{\sqrt 6 + \sqrt 2}{4}$$
These identities are especially powerful when combined with the half-angle formulas.
$$\sin(\frac{x}{2}) = \sqrt{\frac{1 - \cos(x)}{2}}$$
$$\cos(\frac{x}{2}) = \sqrt{\frac{1 + \cos(x)}{2}}$$
(I'm omitting the ± symbols here because I assume you've already followed the previous section's advice to reduce your angles to Quadrant 1.)
With these, having found the trig function values for 15°, we can repeatedly bisect the angle and find the values for 7.5°, 3.75°, 1.875°, 0.9375°, 0.46875°, 0.234375°, 0.1171875°, etc.
In combination with the sum/difference identities, we can calculate the trig functions for any angle that's a binary fraction of 15°. For example,
$$17.109375° = 15° + \frac{15°}{8} + \frac{15°}{64}$$
Taylor series
There are well-known infinite series for $\sin$ and $\cos$ (when their argument angle is expressed in radians):
$$\sin x = x - \frac{x^3}{6} + \frac{x^5}{120} - \frac{x^7}{5040} + \cdots = \sum_{k=0}^\infty \frac{(-1)^k x^{2k+1}}{(2k+1)!}$$
$$\cos x = 1 - \frac{x^2}{2} + \frac{x^4}{24} - \frac{x^6}{720} + \frac{x^8}{40320} - \cdots = \sum_{k=0}^\infty \frac{(-1)^k x^{2k}}{(2k+1)!}$$
Deriving these formulas is probably too advanced for you at this point, but for now, just note that they exist.
Obviously, you can't actually sum an infinite series by computer, but for small enough $|x|$, you can just take the first few terms and get a pretty nice approximation.
Precomputed tables
If your algorithm for sin and cos is slow (e.g., because you live in the pre-calculator era and are doing all the arithmetic by hand), you can calculate them for a subset of angles (e.g., whole degrees) once, and publish them in a table somewhere.
Interpolation
Suppose that you want to calculate $\sin(17°)$, but all you know is that $\sin(15°) \approx 0.258819$ and $\sin(30°) = 0.5$.
Then just use the formula $y = \frac{y_2 - y_1}{x_2 - x_1}(x - x_1) + y_1$ for a line between two points — in this case $(15, 0.258819)$ and $(30, 0.5)$ — and with a little bit of algebra, you can estimate $\sin(17°) \approx 0.290977$. The true value is approximately $0.292372$, so it's an error of 0.001395, which may or may not be good enough for your needs.
But the narrower you can make the interval on which you need to approximate the function, the better your accuracy gets.
Calculating inverse trig functions
Once you have a good implementation of the trig functions, calculating the inverse trig functions becomes a root-finding problem. For example, let's use the simple bisection method to estimate $\arctan(0.75)$. IOW, find $x$ such that $\tan(x) = 0.75$.
The angle is positive, so $0° \le x \le 90°$.
$\tan 45° = 1$. We know that $\tan$ is increasing on this interval, so $x$ must be lower: $0° \le x \le 45°$
$\tan 22.5° \approx 0.414214$. Go higher ($22.5° \le x \le 45°$).
$\tan 33.75° \approx 0.668179$. Go higher ($33.75° \le x \le 45°$).
$\tan 41.25° \approx 0.876976$. Go lower ($33.75° \le x \le 41.25°$).
With each iteration, the interval where $x$ could lie is cut in half. Once you've got a small-enough interval, return the midpoint of it as the value of arctan.
More efficient algorithms to calculate arctan exist, but this one should be easy to understand.
Let's see.
Of course, $x\mapsto x^2\cdot\sin(1/x)$ will never be exactly 'like' a sine wave. However, as $x\to 0$, there are some asymptotic relations which hold.
First of all, when looking at the graph, you notice the amplitude is essentially constant. Let's fix a small number $a$ (for you, $a=1/k$). Near $a$, the difference between $x^2$ and $a^2$, if $x=a+h$, is $2ah+h^2$ which, for $a$ and $h$ small, is essentially negligible. Especially since this is being multiplied by a number bounded in $[-1,1]$, so the error can't blow up at all.
Therefore, near $a$, I approximate $x^2\sin(1/x)\approx a^2\sin(1/x)$ with reasonably good accuracy. What do we do about the $1/x$ term?
Again, it is good to examine how this changes. For $x=a+h$, we have: $$\frac{1}{a+h}-\frac{1}{a}=-\frac{h}{a(a+h)}=-\frac{h}{a^2}\cdot\frac{1}{1+h/a}$$
So, when $|h|\ll a$, this difference is essentially $-h/a^2$. So I notice: $$\sin(1/x)\approx\sin\left(\frac{1}{a}-\frac{h}{a^2}\right)$$Now this is looking linear in $h$. Consider the function: $$x\mapsto\sin\left(\frac{1}{a}-\frac{x-a}{a^2}\right)=\sin\left(\frac{2}{a}-\frac{x}{a^2}\right)$$Near $x=a$, this will roughly equal $\sin(1/x)$.
We then approximate the curve with: $$x\mapsto a^2\cdot\sin\left(\frac{2}{a}-\frac{x}{a^2}\right)$$
If you let $a=-0.001$ in your graph, you will notice very good results. It will only get better as $a\to0$.
See the results for $a=-10^{-4}$.
Best Answer
This behavior is a result of the definition of $\arctan$. Because $\tan$ is not a one-to-one function, it doesn't have an inverse. So we define $\arctan(y)$ to be the unique value $x$ in the interval $(-\frac{\pi}{2},\frac{\pi}{2})$ such that $\tan(x) = y$. Thus, the function $\arctan(\tan(x))+\frac{\pi}{2}$ reduces $x$ modulo $\pi$.
We can relate this to your function with the identity $\cot(x) = \tan(\frac{\pi}{2}-x)$, which explains the negative slope.
The modular arithmetic–like behavior that you noticed arises from this interaction between $\arctan$ and $\tan$: Because there are infinitely many $x$ such that $\tan(x)=y$, $\arctan$ chooses a unique representative, which is equivalent to reducing $\text{mod}\ \pi$. It's really just reflective of the (arbitrary) way we define $\arctan$.