The number of triangles with perimeter $n$ and integer side lengths is given by Alcuin's sequence $T(n)$. The generating function for $T(n)$ is $\dfrac{x^3}{(1-x^2)(1-x^3)(1-x^4)}$. Alcuin's sequence can be expressed as
$$T(n)=\begin{cases}\left[\frac{n^2}{48}\right]&n\text{ even}\\\left[\frac{(n+3)^2}{48}\right]&n\text{ odd}\end{cases}$$
where $[x]$ is the nearest integer function, and thus $T(36)=27$. See this article by Krier and Manvel for more details. See also Andrews, Jordan/Walch/Wisner, these two by Hirschhorn, and Bindner/Erickson.
Let c, a, b be the lengths of the sides AB, BC, and CA of triangle ABC. Use the Ravi substitution: a = x + y, b = y + z , and c = z + x with x, y, z > 0. Then a + b + c = 13 translates to: x + y + y + z + z + x <= 13 ==> x + y + z <= 6.5. Since x,y, and z are positive integers. There are cases to consider:
Case 1: x + y + z = 1. No solution.
Case 2: x + y + z = 2. No solution.
Case 3: x + y + z = 3. This gives: x = y = z = 1. But then a = b = c = 2. We require that the triangle be scalene so the sides have different lengths. So no solution in this case.
Case 4: x + y + z = 4. So either x = y = 1, z = 2 or x = 2, y = z = 1 or y = 2, x = z = 1. So ABC is isosceles. No solution in this case.
Case 5: x + y + z = 5. So either x = 1, y = 1, z = 3 or x = 2, y = 2, z = 1 and permutations of these values. But again ABC is then isosceles. No solution for this case.
Case 6: x + y + z = 6. So x = 1, y = 2, z = 3 and permutations or x = y = z = 2 or x = 1 = y, and z = 4 and permutations. The latter case gives ABC isosceles while the former case renders ABC scalene. So 1 solution for this case.
These are all possible cases. So in summary there is 1 answer: a = x + y = 1 + 2 = 3, b = y + z = 2 + 3 = 5, and c = z + x = 3 + 1 = 4. ABC is a scalene right triangle !
Best Answer
HINT: Let $a,b,c$ denote the lengths of the sides of the triangle. Then we should have $$a+b>c\quad\mbox{and}\quad a+b+c=n.$$ The above two equations implies $a+b-c=n-2c>0$, i.e. $c<n/2$ and similarly $a,b<n/2$. So you need to count the number of triples $(a,b,c)$ such that $0<a\leq b\leq c$ and $a,b,c<n/2$.
EDIT: Note that you only need to find the number of tuples $(a,b)$ such that $0<a\leq b$ and $a,b<n/2$, because $c$ is fixed once you specify $a$ and $b$. Let us consider two cases.
Case (i) $n$ is even: For a fixed $a$, you have the following choices for $b$: $a+1, a+2,\cdots ,n/2-1$, that is $n/2-1-a$ choices. Now $a$ can take values $1,2,\ldots ,n/2-1$. So the number of triangles in this case is $$\sum_{a=1}^{n/2-1}(n/2-1-a)=(n/2-2)+(n/2-3)+\cdots +1=\frac{(n/2-2)(n/2-1)}{2}.$$
Case (ii) $n$ is odd: For a fixed $a$, you have the following choices for $b$: $a+1, a+2,\cdots ,(n-1)/2$, that is $(n-1)/2-a$ choices. Now $a$ can take values $1,2,\ldots ,(n-1)/2$. So the number of triangles in this case is $$\sum_{a=1}^{(n-1)/2}((n-1)/2-a)=((n-1)/2-1)+\cdots +1=\frac{((n-1)/2-1)(n-1)/2}{2}.$$