I really enjoyed reading Forman Acton's Numerical Methods that Usually Work. He writes with a lot of personality and presents interesting problems. I don't know that this is the best book to learn the field from because (1) it was written in 1970 and last revised in 1990 and (2) I am not an expert, so I simply don't know how to evaluate it. But it sounds like these might not be major obstacles from your perspective.
The starting point for the literature is the survey by Jeff Lagarias ten years ago:
math/0309224 The 3x+1 problem: An annotated bibliography (1963--1999) (sorted by author). Jeffrey C. Lagarias. math.NT (math.DS).
https://arxiv.org/abs/math/0309224
Other papers from an arxiv Front search for Collatz and 3x+1:
math/0608208 The 3x+1 Problem: An Annotated Bibliography, II (2000-2009). Jeffrey C. Lagarias. math.NT (math.DS) https://arxiv.org/abs/math/0608208
arXiv:0910.1944 Stochastic Models for the 3x+1 and 5x+1 Problems. Alex V. Kontorovich, Jeffrey C. Lagarias. in: The Ultimate Challenge: The 3x+1 problem, Amer. Math. Soc.: Providence 2010, pp. 131--188. math.NT. https://arxiv.org/abs/0910.1944
math/0509175 Benford's law for the $3x+1$ function. Jeffrey C. Lagarias, K. Soundararajan. J. London Math. Soc. 74 (2006), 289--303. math.NT (math.PR). https://arxiv.org/abs/math/0509175
math/0412003 Benford's Law, Values of L-functions and the 3x+1 Problem. Alex V. Kontorovich, Steven J. Miller. Acta Arithmetica 120 (2005), no. 3, 269-297. math.NT (math.PR). https://arxiv.org/abs/math/0412003
and
https://arxiv.org/abs/math/0411140
https://arxiv.org/abs/math/0205002
https://arxiv.org/abs/math/0201102
https://arxiv.org/abs/math/0204170
There are many un-serious papers on this subject at the same site, I selected the above from papers that I read or authors whose other papers I have seen.
There is also a celebrated theorem by John H Conway in which he shows that the natural generalization of the Collatz problem, where different linear functions are applied to $n$ in several arithmetic progressions (partitioning all integers), can simulate a computer. A consequence is that it is undecidable to determine if such an iteration loops when started from $1$.
Best Answer
That's set theory. I honestly recommend that you study set theory, as nowdays all mathematics are built on sets. Just pick any introductory book to set theory, that ill be (imo) enough.
Now, what you may having difficulties with, is the interpretation of those things, let's take an example from that page:
This basically mean that $Y$ is a set containing observations, more simple (not so formal) translation could be: "We have a bunch of observations".
This now means that $\omega$ (defined in the next parragraph) is a partition from a space of partitions, and you're looking for subsets, that they call $Y_k$ that is believed to belong to the same person. Now the translation: "We are looking for observations that we believe belong to the same person, and we are going to separate the observations that belong to every person in different groups (sets)".
And last, they say a valid $\omega$ must have trajectories that contain (alltogether) the initial set $Y$, and that this trayectories' intersection must be empty. The translation: "The partition must have all subsets of observations, that means we must have formed groups that cover all observations, and two different subsets of observations cannot share a member, that would be obviously absurd, if we had two belonging to person A, and another two belonging to person B, and one of those two was repeated, then A and B would be the same person"
That goes on for the whole article. The final parragraph means:
This notation is used because it doesn't let space for ambiguity, and is more rigouruos, so besides studying set theory, which is basic, you should train you interpretation of these articles (just read more and try to do what I've done here)
BTW, I don't think there's any book about mathematical notation for computer science (it would really surprise me), because such thing doesn't exist, that's just mathematical/formal notation.