To elaborate on Martin's comment: let $x_i = ab^i a^{-1}$. Then what you have said is that $x_1 x_4 = x_5$ in what you are calling $F(S)$ (what you really mean is $\langle S\rangle$, the subgroup of $F_2$ generated by $S$). But $x_1 x_4$ is a reduced word in the $x_i$'s, so it shouldn't 'collapse to' a different word. Since it does, $\langle S\rangle$ is not free on $S$.
EDIT: Since what you were mainly asking about was whether your line of reasoning is correct, I'll elaborate a little more. When you define $F(S)$ to be the free group on $S$, then you are treating $S$ as an abstract set, rather than as a subset of $F_2$. This is perfectly valid in some sense, but it's not going to give you a subgroup of $F_2$, so it's not helpful.
What you want to do is to find, for each $k\in \mathbb{N}$, a subset $X_k$ of $F_2$ such that the subgroup generated by $X_k$ is free on $X_k$.
You're actually very close to the right idea, though. The standard way of doing this does involve taking conjugates, but remember that you don't want any 'collapsing' to occur except when you have something like $x_i x_i^{-1}$, so you'll need to do something a little differently.
Well, since there's another answer with an outline of a proof, and where to look etc., I'll add a partial spoiler for this way, but please don't look until you've tried.
Instead of setting $x_i = ab^i a^{-1}$, try $x_i = a^i b a^{-i}$. Now all that pesky 'collapsing' shouldn't be a problem. So if $S_k = \langle x_i \mid 1\leq i\leq k\rangle$, you should be able to show that $S_k\cong F_k$.
In algebra, a "free X" (where X is the name of an algebraic structure, such as "group", "semigroup", "magma", etc.) is basically an algebraic structure in which two expressions are equal if and only if the definition of X says that they must be.
For example, the simplest kind of algebraic structure is a magma: a set $S$ equipped with an arbitrary binary operator $\cdot$, which is only required to be closed (i.e. that if $a$ and $b$ are members of $S$, then $a \cdot b$ exists and is a member of $S$). A magma is not required to satisfy any algebraic identities except the trivial one (that two identically written expressions are equal), and a free magma is one that indeed doesn't — two expressions in a free magma (using only the given generators as variables) are equal if and only if they're written in exactly the same way.
Formally, a free magma can be defined as follows:
- The generators are (distinct) elements of the magma.
- For any two elements $a$ and $b$ of the magma, the expression $(a \cdot b)$ is an element of the magma.
- No two expressions constructed in this way are equal, unless they are written identically. In other words, the expression $(a \cdot b)$ is never equal to a generator, and is only equal to the expression $(c \cdot d)$ when $a = c$ and $b = d$.
- The magma contains no other elements.
(And yes, this definition indeed implies that any (non-empty) free magma must have infinitely many elements.)
Similarly, a semigroup is a magma that obeys the associative law $(a \cdot b) \cdot c = a \cdot (b \cdot c)$, and a free semigroup can be constructed by taking a free magma and identifying any two expressions that can be made identical by applying the associative law.
It turns out that, because of the associative property, expressions in a semigroup can always be written unambiguously without parentheses, simply as ordered sequences of elements separated by the binary operator: any method of reinserting the parentheses will yield equivalent results under the associative law. Thus, another way of constructing a free semigroup is as the set of all finite-length $n$-tuples of the generators, with concatenation as the semigroup operator:
- For all generators $x$, the 1-tuple $(x)$ is an element of the semigroup.
- For any two tuples $a = (a_1, \dots, a_m)$ and $b = (b_1, \dots, b_n)$ in the semigroup, the concatenated tuple $a \cdot b = (a_1, \dots, a_m, b_1, \dots, b_n)$ is an element of the semigroup.
- No two tuples in the semigroup are equal, unless they have the same length and exactly the same elements.
- The semigroup contains no other elements.
It's common to also include the empty tuple $\epsilon = ()$, which then acts as an identity element, in the free semigroup. Technically, though, what we then have is a free monoid, a monoid being a semigroup with an identity element $\epsilon$ that satisfies the additional law $a \cdot \epsilon = \epsilon \cdot a = a$ for all $a$.
Finally, a group is a monoid with inverses: that is, every element $x$ of a group has an inverse element $x^{-1}$ such that $x \cdot x^{-1} = x^{-1} \cdot x = \epsilon$. Again, a free group can be obtained from a free monoid by adding inverses for the generators, and identifying any expressions that can be made identical by repeatedly removing pairs of adjacent elements that are inverses of each other.
Similarly, we might e.g. consider free abelian groups, constructed by taking a free group and identifying any expressions that can be made identical by applying the commutative law $a \cdot b = b \cdot a$ to reorder their elements (and removing inverse pairs); these turn out to be representable as (finitely supported) functions from the set of generators to the integers, where the value of the function giving the number of times that generator appears in the expression (with inverses counted as $-1$ appearance).
Or we could skip groups entirely, and instead consider e.g. free commutative monoids; these can be represented in the same way as free groups, except that, since we don't have inverses, no generator can appear a negative number of times in an expression. (A free commutative semigroup is the same as a free commutative monoid, except that it has no identity.)
Best Answer
Firstly, just in case you don't know, given a set $S$, the free group $F(S)$ satisfies the following universal property mentioned by @PaulPlummer in the comments. Given a group $G$ and a morphism of sets (i.e. a function) $f\colon S \to G$, there exists a group homomorphism $\phi_f\colon F(S) \to G$ that extends the map $f$. I mean this in the sense that if $i\colon S \to F(S)$ is inclusion, then $\phi_f\circ i = f$ as a set map $S \to G$. Moreover the map $\phi_f$ is unique. I'd draw you a pretty diagram, but it wants a diagonal arrow.
If this is new to you, try proving it!
Now, given a function $f\colon S \to S'$, we want to show that there exists a group homomorphism $F(f) \colon F(S) \to F(S')$ and that the assignment $f \mapsto F(f)$ is functorial. To use the above property, we need a set map $S \to F(S')$. Let $i_{S'} \colon S' \to F(S')$ be the inclusion. We'll use $i_{S'}\circ f$; I claim that the resulting group homomorphism $F(f) := \phi_{i_{S'}\circ f}$ works. Its existence is afforded to us by the universal property.
To show functoriality, we need to show that the identity function $\operatorname{id}_S\colon S \to S$, is sent to the identity homomorphism $F(\operatorname{id_S}) = \operatorname{id}_{F(S)}$, and that composition is respected, i.e. $F(g\circ f) = F(g)\circ F(f)$.
Both properties are straightforward exercises and left to you. Note the usefulness of the uniqueness statement above!