In many ways, I am atypical in the way that I approach a problem, but it works for me. Specifically, I try to understand an example in as much detail as I possibly can. If the example, is too complicated, then I make a simpler example. As much of the intricate detail that I can bring to bear on the example is brought.
For example, instead of trying to understand Lie groups and Lie algebras in general, start with the circle and the line that is tangent at the point (1,0). What is the exponential map? Oh, OK. Now how about $SU(2)$ and $su(2)$? Can you understand that the Lie group is the $3$-dimensional sphere? Can you understand the coordinates? Can you understand the equators? How do $i,j$ and $k$ really work?
What is the difference between the multiplication rule $i\times i =0$ and $i^2=-1$?
I spend time pondering. And often my notebooks will contain tangential problems or specific computations. I will keep doing the computation until I get it right! If necessary, I will write a program to complete the computation. When I understand the example completely, it is usually easy to abstract.
Then I follow up, usually writing in a notebook or several notebooks before I begin writing on the computer. I have an advantage in that I have long-distance collaborators, so it becomes necessary to explain the idea to the collaborator(s). That is the first writing stage: write for someone who knows your short-hand and your metaphors. the second stage is to write for someone who does not. Then I write with a set of colleagues in mind, but I assume the colleagues do not remember anything from the previous work. I also try to explicate the notation writing for example "the function $f$, the knot $k$, or the tubular neighborhood $N$.
A complex analytical colleague only uses $z$ for a complex number, $x$ for a real variable, and $n$ for an integer. These variable choices are culturally determined, and so one keeps with the culture of the discipline unless there is good reason to deviate. As a final example of this, the variable $A$ in the bracket polynomial is known to everyone in the field. The variables $q$, $t$, $X$ etc. are less known and involve different normalizations. So it is the burden of the author to relate these to the more well known choices.
If you are referring to algorithm complexity, every loop you have translates in a sum. Your algorithm thus have the following complexity:
$\sum_{i=0}^{N-1} \sum_{j=i+1}^{N} $ [...]
where [...] is the complexity of your function fn(i,j).
If you are referring the mathematical expression of the out variable, you seem to be requiring a matrix notation. Something along the lines of:
Let $A$ be a strictly triangular matrix where $A_{i,j} = fn(i,j)$.
If you need to explicitly explain a strictly triangular matrix, you can add:
such that $A_{i,j} = 0$ for $i \le j$
As shown here.
Best Answer
If you really want to use a "formal" and "succint" mathematical expression, you can use the Hadamard Product, where the Hadamard product of two matrices is got by mutiplying the corresponding entries.
So if the input is treated as a vector: $\text{inp}$, then the output vector $\text{out}$ is given by
$$ \text{out} = \text{inp} \circ \text{inp}$$
where $\circ$ is the Hadamard product.
Note that you can treat a vector as a $1\times n$ matrix.
Of course, this will probably force your readers to try and figure out what a Hadamard product is, and you might as well just write
$$ \text{out}[i] = \text{inp}[i]^2 \quad \forall i \in \{0, 1, \dots , n-1\} $$