In principle, we could make up arbitrarily many distinct quantifiers. $\forall x\colon \phi(x)$ and $\exists x\colon \phi(x)$ make certain statements about the class $C:=\{\,x\mid \phi(x)\,\}$: One says that $C$ is the all-class (or that its complement is empty), the other says that $C$ is not empty. And $\exists!$ say that $C$ is a singleton. We could introduce quantifiers for quite arbitrary statements about $C$, but the most natural would perhaps be about cardinality of $C$: Stating that $C$ or its complement has at least $n$, at most $n$, more than $n$, less than $n$, or exactly $n$ elements for a fixed finite or infinitie cardinality $n$ come to mind.
Of these, those with finite cardinalities are readily constructed from $\exists$ and $\forall$ (just like we do with $\exists!$).
However, any quantifiers about infinite cardinalities are problematic in a first order theory (i.e., when we can only "speak" about the objects of our universe, but not about sets of objects of our universe): There are no suitable rules of inference accompanying them. A proof of $\exists x,\phi(x)$ can consist of constructing a sing $a$ with $\phi(a)$. What should a proof of $\exists^\infty x,\phi(x)$ with $\exists^\infty$ intended to mean "there are infinitely many" look like? By exhibiting infinitely many $a_n$ with $\phi(a_n)$? You cannot do that in a first-order theory per se.
Nevertheless, outside first order formalism, some such quantifiers are very common: A very useful one is "almost all" which in the context of natural numbers is understood to mean "all but finitely many". "Almost all" $n$ have property $\phi$ is then a shortcut for $\exists n_0\in\Bbb N\colon \forall n\in \Bbb N\colon n>n_0\to \phi(n)$, a construct very common in the introduction of limits.
Note how the very properties of the set of natural numbers allow us to express "all but finitely many". (In other contexts, such as measure theory, we use "almost all" with a different meaning, namely "up to a set of measure $0$")
As often happens in logic (and was suggested by the comment thread), this question is easier to answer than it is to state. I'll first give an answer, and then explain how I'm interpreting the question precisely.
Answer
Yes, $\exists!$ is so definable. Here's a set of defining formulas for $\exists!$ which don't use any other quantifiers:
$(a)$: $(\mathscr{Q}x.U(x))\rightarrow [[U(a)\wedge U(b)]\rightarrow [V(a)\vee\neg V(b)]]$.
$(b)$: $\neg\mathscr{Q}x.\perp$.
$(c)$: $\mathscr{Q}x. x=a$.
Intuitively, $(a)$ says that $\mathscr{Q}$ doesn't contain any sets with more than one element, $(b)$ says that $\mathscr{Q}$ doesn't contain the empty set, and $(c)$ says that $\mathscr{Q}$ does contain every set with one element.
(We could have replaced "$V(a)\vee\neg V(b)$" in $(a)$ with "$a=b$," but I think it's cute that we don't need equality there, even if equality is needed in $(c)$.)
Question
OK, now how do we phrase the question precisely?
Alex Kruckman raised the key issue - that once we fix a single structure $\mathcal{M}$, we run into trouble when we try to think about the non-definable subsets of $\mathcal{M}$. However, there is a saving grace here: all the quantifiers you consider are "set-based," that is, their behavior over a structure $\mathcal{M}$ is entirely determined by the underlying set $M$ of $\mathcal{M}$. To be precise, a set-based $n$-ary quantifier is a (class) function assigning to each set $X$ a family of subsets of $X^n$, which is stable under permutations of $X$.
This lets us shift from trying to describe a quantifier over a single structure to describing its behavior over all structures at once. For example, we can describe the definability of $\forall$ as follows:
Suppose $\mathscr{Q}$ is a set-based quantifier such that (for a unary relation symbol $U$ and constant symbol $c$) each of the following sentences is a validity according to the "$\mathscr{Q}$-semantics:"
$\mathscr{Q}.\top$.
$(\mathscr{Q}x.U(x))\rightarrow U(c)$.
Then $\mathscr{Q}$ is just $\forall$.
The proof is exactly the intuitive argument. Suppose $\mathscr{Q}$ is a set-based quantifier satisfying $1$ and $2$ above, $\mathcal{M}$ is any structure with underlying set $M$, and $\varphi$ is a formula in the language of $\mathcal{M}$. If $\mathcal{M}\models\neg \mathscr{Q}x.\varphi(x)$ then we must not have $\varphi$ hold on all of $\mathcal{M}$, by $1$; on the other hand, if $X\subseteq M$ is "$\mathscr{Q}$-big" then considering the expansion of $\mathcal{M}$ by a predicate naming $X$ we get $X=M$ by $2$. So $\mathscr{Q}=\forall$ as hoped.
More generally, given any set-based quantifier $\mathscr{Q}$ let $\mathsf{QFL}[\mathscr{Q}]$ be quantifier-free logic augmented by $\mathscr{Q}$. Basically, we take $\mathsf{FOL}$, remove the quantifiers, and add $\mathscr{Q}$. Your question can (I claim!) be precisely asked as follows:
Suppose $\mathscr{Q}$ is a set-based quantifier such that $\mathsf{QFL}[\mathscr{Q}]$ and $\mathsf{QFL}[\exists!]$ have the same validities. Must $\mathscr{Q}=\exists!$ be true? If so, is there a finite set $\Phi$ of $\mathsf{QFL}[\exists!]$-validities such that whenever $\Phi$ is also a set of $\mathsf{QFL}[\mathscr{Q}]$-validities we must have $\mathscr{Q}=\exists!$ as above?
If you buy this phrasing, then it's easy to check that the answer I gave above does in fact work: if $\mathscr{Q}$ is a set-based quantifier such that $(i),(ii),(iii)$ are each $\mathsf{QFL}[\mathscr{Q}]$-validities, then $\mathscr{Q}$ is in fact $\exists!$.
Coda
First of all, note that "set-based" is not a standard term. I don't think there is one. Sorry about that. Second, and relatedly, not all quantifiers are set-based, with one example being the quantifier $\mathfrak{S}x.\varphi(x)\equiv$ "$\{x:\varphi(x)\}$ meets every substructure of the universe."
Third, and most substantively, we can keep this story going! The above amounts to showing that $\exists,\forall$, and $\exists!$ are each "definable" over $\mathsf{QFL}$ in a particular sense. It turns out that the quantifier $\exists^\infty$ is "definable" over $\mathsf{FOL}$ in the same sense but is not "definable" over $\mathsf{QFL}$, so there is a meaningful hierarchy of quantifiers in terms of steps needed to define them. Not much is known to me about this - for example, is there any quantifier that takes three steps to define? - but see this MO post (and note that the $\mathcal{L}_0$ used there is this post's $\mathsf{QFL}$).
Best Answer
There may be different definitions in different textbooks; in Shoenfield's classic "Mathematical Logic" the definition of a formula in prenex normal form (of which a universal formula is a special case in which only universal quantifiers and no existential quantifiers are permitted in the prefix) explicitly permits for an empty quantifier prefix.
Normally, unless specified otherwise, $n = 0$ is permitted in such definitions; and for the sake of the present notion it shouldn't hurt: The point of a universal formula is to have all quantifiers at the front and all quantifiers converted to universal ones; if there are no quantifiers altogether, it still matches that requirement.
An analogous argument can be made for existential formulas.
In any case, I don't see how this would render the claim in the linked question false. A quantifier-free formula $\alpha$ is, in the general case, indeed not logically equivalent to an existentially quantified one: $$\alpha \not \equiv \exists x \alpha$$
Remember that two formulas are equivalent if they are true under all the same structures and variable assignments. But since the truth value of a formula with an free variable may vary between different variable assignments, while the truth value of a formula with the variable bound is constant within the same structure and hence if it is true under one assignment it is true under all of them, there may be variable assignments that satisfy $\exists x \alpha$ but not $\alpha$.
Of course, a universal formula without universal quantifiers is equivalent to an existential formula without existential quantifiers, because then you have syntactically identical formulas: $$\alpha \equiv \alpha$$