That's set theory. I honestly recommend that you study set theory, as nowdays all mathematics are built on sets. Just pick any introductory book to set theory, that ill be (imo) enough.
Now, what you may having difficulties with, is the interpretation of those things, let's take an example from that page:
Given the set of N observations Y = {yi ; i = 1, . . . , N } we want to identify which
observations belong to the same object.
This basically mean that $Y$ is a set containing observations, more simple (not so formal) translation could be: "We have a bunch of observations".
We are looking for a partition ω of observations from Y into several
trajectories Yk ⊂ Y (subsets of Y ) such that each trajectory collects
observations believed to come from a single person.
This now means that $\omega$ (defined in the next parragraph) is a partition from a space of partitions, and you're looking for subsets, that they call $Y_k$ that is believed to belong to the same person. Now the translation: "We are looking for observations that we believe belong to the same person, and we are going to separate the observations that belong to every person in different groups (sets)".
A valid partition expresses the set of all observations as an
exhaustive union of trajectories: Y = Y1 ∪ . . . ∪ YKω , where Kω is
the number of objects proposed by a partition ω. The trajectories must
be mutually exclusive: Y ∩ Yk = ∅, when = k.
And last, they say a valid $\omega$ must have trajectories that contain (alltogether) the initial set $Y$, and that this trayectories' intersection must be empty. The translation: "The partition must have all subsets of observations, that means we must have formed groups that cover all observations, and two different subsets of observations cannot share a member, that would be obviously absurd, if we had two belonging to person A, and another two belonging to person B, and one of those two was repeated, then A and B would be the same person"
That goes on for the whole article. The final parragraph means:
We have a bunch of observations. We are looking for observations that
we believe belong to the same person, and we are going to separate the
observations that belong to every person in different groups (sets),
that separation is called "partition". The partition must have all
subsets of observations, that means we must have formed groups that
cover all observations, and two different subsets of observations
cannot share a member, that would be obviously absurd, if we had two
belonging to person A, and another two belonging to person B, and one
of those two was repeated, then A and B would be the same person.
This notation is used because it doesn't let space for ambiguity, and is more rigouruos, so besides studying set theory, which is basic, you should train you interpretation of these articles (just read more and try to do what I've done here)
BTW, I don't think there's any book about mathematical notation for computer science (it would really surprise me), because such thing doesn't exist, that's just mathematical/formal notation.
The problem is that you cannot learn mathematical notation as though it were a programming language with a single, well-defined, fixed syntax where particular grammatical constructs always have the same meaning. It's much more like a natural language: a collection of rules and conventions, some inviolate, others less so, with lots of idioms some of which are mutually incompatible, and lots of variation between "dialects" (by which I mean, conventions within various fields). That's why you get the advice in the other answers: There is no reference manual and no formal specification. Just keep reading and writing the language and allow yourself to absorb it through practice. Here, let me give some examples to convince you.
You ask what $f_n$ means devoid of context. Well, sometimes it is the $n$th function in a sequence of functions $f_1,f_2,\ldots$. Sometimes it is the $n$th entry of an $m$-dimensional vector $\mathbf f=(f_1,f_2,\ldots,f_m)$. Sometimes it's the normal component of a force, as opposed to the tangential component which might be called $f_t$.
You might think that at least $f^n$ will always be $f$ to the $n$th power, but that's not always true either. Sometimes we put an index at the top because we're already using indices at the bottom to mean something else — so $f_i^n$ might be the value at the $i$th grid cell at time $n$. Usually $\sin^nx$ means $(\sin x)^n$ but usually $\log^n x$ means $\underbrace{\log\log\cdots\log}_{\text{$n$ times}}\, x$.
Why this apparently miserable state of affairs? Because mathematical notation is actually an extremely efficient method for communicating ideas between people, and people are, with a little bit of practice, quite adept at determining with high accuracy the intended meaning of informal, ad-hoc, underspecified, potentially ambiguous signals. When doing mathematics, we don't worry about shaping our thoughts to fit the rigid syntax of our language, like we do when programming. Instead, we freely shape the syntax to fit our thoughts. If that means it is impossible to read mathematics without knowing what it means, so be it; it only needs to be easy to parse by the intended reader, who is usually a mathematically literate human being. And said reader surely knows that the context in which $f_n$ appears is about, say, sequences of functions, in which case $f_n$ almost certainly means the $n$th function in the sequence.
See also the fourth section ("Mathematical syntax") of Jeremy Kun's essay "Why there is no Hitchhiker’s Guide to Mathematics for Programmers".
(Re. CBenni's comment: Suppose someone asks "What is the meaning of f[n] in programming?" If you're programming in the C family, it means the nth element of the array f. If you're programming in Haskell or ML, it means the function f applied to the list [n], whose only element is n. If you're programming in Mathematica, it means the function f applied to n. The meaning of $f_n$ in mathematics is similar.)
Best Answer
If it's truly important to describe the algorithm in mathematical notation, look to Haskell for inspiration. Many Haskell statements can be translated directly into mathematical notation. For example, the definition
is equivalent to the mathematical statements
$$\begin{align*}fac(0) &= 1\\fac(n) &= n\ fac(n - 1)\ (\operatorname{if} n \ne 0).\end{align*}$$
In practice, however, what you really want is usually to write algorithms precisely, with mathematical terminology. In order to accomplish this, it is essential to practice doing so, and to ask other people for feedback. You can't learn to play the piano by reading books about it, nor can you learn piano while wearing earmuffs. Look at examples, too; every time you look at an example and think "oh, what a good idea!", you've learned something.
Really, I don't know of any better ways to learn this. Think of an algorithm, and try to write it down in a way that a mathematician would understand. Ask a mathematician if they understand. If not, figure out why. Repeat.