Question
Let $\mathbb{N} = \{0, 1, 2, \dots \}$, $\mathbb{N}^+ = \mathbb{N} – \{0\}$, $m \in \mathbb{N}^+$, $[m] := \{0, 1, \dots, m-1\}$ and define the function $f:[m]\to[m]$ as below
$$f(i) = (a + i \, b) \bmod m,$$
where $a, \, b \in \mathbb{N}$. I want to show that $\big( f(0), f(1), \dots, f(m-1) \big)$ is a permutation of $(0, 1, \dots, m-1)$ provided that $b$ and $m$ are relatively prime.
Motivation
In open address hash tables, hash functions are of the form
$$h:U \times \{0, 1, \dots, m-1\} \to \{0, 1, \dots, m-1\}, \tag{1}$$
where $U$ is the set of all possible keys. It is required that the sequence of probes defined as
$$p(k) := \big(h(k, 0), \, h(k, 1), \dots, \, h(k, m-1)\big) \tag{2}$$
generates a permutation of $(0,1,\dots,m-1)$ for every key $k \in U$, which guarantees that all of the hash table slots will be visited in a probe sequence. One way to get around this is by double hashing. In this method, the hash function is defined as below
$$h(k,i):=f(k) + i \, g(k) \bmod m, \tag{3}$$
where $f$ and $g$ are functions of the form
$$f:U \to \{0, 1, \dots, m-1\}, \qquad g:U \to \{0, 1, \dots, m-1\}.$$
In order to have the permutation property, we have the following theorem.
Theorem. If $g(k)$ and $m$ are relatively prime then $p(k)$ is a permutation of $(0,1,\dots,m-1)$.
Best Answer
This is not true in general, since you haven't put any restrictions on the divisibility properties of $a, b$, and $m$. For example, if $a= 3$ and $b = m$ then $f$ is not a permutation since it just maps everything to $3$.
However, it will be true if $b$ is assumed coprime to $m$. Let's see why.
First, the final step of adding $a$ and then modding by $m$ obviously gives a permutation, and so you only need to check that multiplying by $b$ gives a permutation.
Since $b$ and $m$ are coprime, there are integers $c$ and $d$ such that $cb + dm = 1$. That means that any arbitrary class $f$ modulo $m$ is the same as $(cb + dm)f = b\cdot(cf)$ mod $m$. So multiplication by $b$ is a permutation and multiplication by $c$ is the inverse of it.