This posting is not a complete answer. It only discusses the first axiomatization in the OP.
The first axiomatization correspond to the usual notion of boundary defined as $\overline{A}\cap\overline{X\setminus A}$. To see this, suppose $(X,\tau)$ is a topological space, that is $X$ is a non empty set and $\tau$ a collection of subsets of a set $X$ such that
- for any $\mathcal{U}\subset \tau$, $\bigcup \mathcal{U}\in \tau$,
- for any finite collection $\mathcal{J}\subset\tau$ $\bigcap\mathcal{J}\in \tau$ (if $\mathcal{J}=\emptyset$, it is understood that $\bigcap\mathcal{J}=X$).
Define $F$ to be a closed set iff $X\setminus F\in \tau$; for any set $A\subset X$, the closure $\overline{A}$ of $A$ is the intersection of all closed subsets of $X$ that contain $A$; the interior $A^o$ of $A$ is the union of all sets in $\tau$ that are contained in $A$; the boundary $\partial A$ of $A$ is defined as
$\overline{A}\cap\overline{X\setminus A}$.
It follows easily that $(X\setminus\overline{A})=(X\setminus A)^o$ for all $A\subset X$ and from this,
- $\partial A=\partial(X\setminus A)=\overline{A}\setminus A^o$;
- $\partial(A\cup B)\subset \big(\partial A\big)\cup\big(\partial B\big)$;
- $\partial (A\cap B)\subset \big(\partial A\big)\cup\big(\partial B\big)$.
These properties of $\partial$ in turn imply that
- $\overline{A}=A^o\cup\partial{A}\subset A\cup\partial{A}\subset\overline{A}$.
- $\partial(\overline{A})=\overline{\overline{A}}\cap\overline{X\setminus\overline{A}}= \overline{A}\cap\overline{(X\setminus A)^o}\subset\overline{A}\cap\overline{X\setminus A}=\partial A$
- $\partial(\partial A)=\partial(\overline{A}\cap\overline{X\setminus A})\subset \partial{A}\cup\partial(X\setminus A)=\partial A$
- $(A\cap B)\cap\partial(A\cap B)=A\cap B\cap(\overline{A\cap B}\cap\overline{(X\setminus A)\cup(X\setminus B)}=(A\cap B)\cap(\overline{X\setminus A}\cup\overline{X\setminus B})=(A\cap B\cap\overline{X\setminus A}) \cup (A\cap B\cap\overline{X\setminus B})=(A\cap B)\cap(\partial{A}\cup\partial{B})$
In summary,
Proposition 1: Given a topological space $(X,\tau)$, the boundary operator $\partial: \mathcal{P}(X)\rightarrow\mathcal{P}(X)$ defined as $\partial A =\overline{A}\cap\overline{X\setminus A}$ satisfies
- $\partial\emptyset=\emptyset$
- $\partial{A}=\partial(X\setminus A)$
- $\partial(\partial A)\subset \partial A$
- $(A\cap B)\cap\partial(A\cap B)=(A\cap B)\cap(\partial A\cup\partial B)$
Notice that a set $A$ is open ($A=A^o$) iff $A\cap \partial{A}=\emptyset$. Indeed $A\cap\partial{A}=A\setminus A^o$.
Conversely,
Proposition 2: Given a nonempty set $X$, suppose $\beta:\mathcal{P}(X)\rightarrow\mathcal{P}(X)$ satisfies properties (1) through (4) in Proposition 2, then $\mathscr{G}:=\{G\subset X: G\cap \beta(G)=\emptyset\}$ is topology on $X$, and in such topology, $\partial A = \beta(A)$ for all $A\subset X$.
This axiomatization based in the boundary operator is mentioned in for example Dugundji, J. Topology, William C Brown Pub, 1966, chapter 3 section 5. The details (proof of Proposition 2) are left there to the reader.
A detailed analysis of this axiomatization and a proof of Proposition 2 can be found in Wang, M.L., Relations among basic concepts in topology, M.S. Thesis, Oregon State U., 1968, pp. 24-28.
Here are other related concepts related to the boundary $\partial A$ of a set $A$.
- In many texts (especially in the French school of topology), $\partial A=\overline{A}\cap\overline{X\setminus A}$ is called frontier and is denoted by $\operatorname{Fr}(A)$.
- The set $\partial_I A:=A\cap\partial{A}=A\setminus(\overline{X\setminus A})=A\setminus A^o$ is called the internal boundary
- The set $\partial_EA:=\overline{A}\setminus A=(X\setminus A)\cap\partial (X\setminus A)=\partial_I(X\setminus A)$ is called the external boundary of $A$.
- Notice that
$$\partial_IA\cup\partial_EA=(A\cap\overline{X\setminus A})\cup((X\setminus A)\cap \overline{A})=\partial A$$
The axiomatization described in reference [2] in the OP is related to internal boundary operator.
That the first axiomatization implies the second is now trivial by virtue of Proposition 2, for if $U\subset V$, then
$$\partial U\subset \overline{U}=U\cup\partial U\subset \overline{V}=V\cup\partial{V}$$
It is known that the second axiomatization produces a topology $\mathscr{G}'$ on $X$ for which $\partial{A}=\overline{A}\cap{X\setminus A}$, where the closure is with respect to $\mathscr{G}'$. In particular $A\mapsto\partial A$ satisfies the axioms in the first axiomatization.
Thus, indeed, both axiomatizations are equivalent.
Best Answer
The axioms for $f$ are constructed from those of $g$ (which is the derived set operator) by replacing $\cup$ by $\cap$ and vice versa, interchanging $\emptyset$ and $X$ and reversing the inclusions where appropriate. So its axioms are dual to those of the derived set and that's why it's named the co-derived set operator I think.
They're dual in the same way as closure and interior operators are, if you look at their axioms.
I think that if $g$ is a derived set operator, $f(A)=g(A^\complement)^\complement$ will be a co-derived set operator and vice versa. The paper mentions that too, I saw.
Just like a set is closed iff it contains its derived set, we have the dual fact that a set is open iff its a subset of its coderived set. I don't think an alternative description is very enlightening. It's a completely artificial "Spielerei".