I have already known that in algebraic topology $\partial^2=0$ holds, or $\partial_{n-1}\partial_n=0$ to be more specifically. If one chooses all $n$-simplicies as the basis of the $n$-chain vector space, the matrix of boundary operator $\partial_n$ is exactly the $n$-th incidence matrix $B_n$. For oriented simplicial complexes, incidence matrices have -1,0,1 as their elements, and $B_nB_{n+1}=0$ holds for every order $n$. However, for the unoriented simplicial complexes, incidence matrices are 0-1 matrices, and there is no way $B_nB_{n+1}$ has to be zero matrix.
I figure it out that in an unoriented simplicial complex, the non-zero elements of $B_nB_{n+1}$ should all be even. Does it indicate that $B_nB_{n+1}=0$ only holds in $\mathbb{Z}/2$?
EDIT about details on oriented and unoriented simplicial complex:
given a simplicial complex, each $p$-simplex can be represented by a square bracket set, say $[v_0,v_1,\cdots,v_p]$. In an unoriented simplicial complex, the above set is unordered, that is, arbitrary permutation of its elements does not change what it represents. In this case, the incidence matrices are $0-1$ matrices, with the $1$s indicating that corresponding simplex is incident to corresponding 1 order higher simplex. But simplices of an oriented simplcial complex are ordered set, and odd permutations of the set elements should change the orientation of the simplices. In this case, the incidence matrices have $-1$s and $1$s, which respectively indicate negative and positive incidence relationship.
A toy example from this paper:
The orientation of $\sigma_1^{(1)}$ is coherent to that of the $2$-simplex, while $\sigma_1^{(2)}$ and $\sigma_1^{(3)}$ is incoherent. So its incidence matrix $B_2$ should be the upper right form.
Best Answer
The standard approach to define homology groups of a simplicial complex $K$ is based on oriented simplices. These are used to define the oriented chain complex of $K$ whose homology groups are called the (oriented) homology groups of $K$. This is a fairly complicated construction and one can legitimately ask whether it can be done simpler by using unoriented simplices.
As you say, the usual construction of a chain complex does not work for unoriented simplices. Noting that $n$-simplices of $K$ are just sets of $(n+1)$ vertices $\{v_0,\ldots,v_n\}$ of $K$, we can course define $$C'_n(K) = \text{ free abelian group generated by the set of all $n$-simplices}, $$ $$\partial : C'_n(K) \to C'_{n-1}(K), \partial(\{v_0,\ldots,v_n\}) = \sum_{i=0}^n \{v_0,\ldots,v_{i-1},v_{i+1}, \ldots ,v_n\} .$$ But then $\partial^2 \ne 0$. There is no way to redefine $\partial$ by adding signs to achieve $\partial^2 = 0$ simply because the sets $\{v_0,\ldots,v_n\}$ are unordered so that there is nothing like an $i$-th component. The definition $\partial(\{v_0,\ldots,v_n\}) = \sum_{i=0}^n (-1)^i\{v_0,\ldots,v_{i-1},v_{i+1}, \ldots ,v_n\}$ does not work because it is not invariant under permutations of the coordinates. Oriented simplices were introduced to circumvent this problem, but the price is a much more complicated construction of a chain complex.
However, as you observe, we have $\partial^2 = 0$ mod $2$. This shows that a reasonable definition of an unoriented chain complex is $$C'_n(K) = \text{ vector space with ground field $\mathbb Z_2$ generated by the set of all $n$-simplices}, $$ $$\partial : C'_n(K) \to C'_{n-1}(K), \partial(\{v_0,\ldots,v_n\}) = \sum_{i=0}^n \{v_0,\ldots,v_{i-1},v_{i+1}, \ldots ,v_n\} .$$
For a thorough treatment see