The number of Dyck paths in a square is well-known to equal the catalan numbers:
http://mathworld.wolfram.com/DyckPath.html
But what if, instead of a square, we ask the same question with a rectangle? If one of its sides is a multiple of the other, then again there is a nice formula for the number of paths below the diagonal, but is there a nice formula in general? What is the number of paths from the lower-left corner of a rectangle with side lengths a and b to its upper-right corner staying below the diagonal (except for its endpoint)? I am also interested in asymptotics.
[Math] Dyck paths on rectangles
co.combinatorics
Related Solutions
UPD: the answer below is in fact completely wrong - it deals with counting walks $\gamma$ weighted by $\mu^{-\text{length}(\gamma)}$. It is clear that without restricting or penalizing for the lengths of the walks, the number of walks will grow exponentially with the volume (i. e., as $\eta^{N^2}$ for some $\eta>1$). In fact, in this paper it is shown that a walk chosen uniformly from this measure (or even from a measure $\hat{\mu}^{-\text{length}(\gamma)}$ with any $\hat{\mu}<\mu$) is space-filling in the scaling limit. It is also conjectured that its scaling limit is SLE$_8$, which suggests that it is in the same universality class as the perimeter of the Uniform Spanning tree. If we buy that, then the number of walks should behave similarly to the number of spanning trees (although with a different $\eta$), which is given by the determinant of the graph Laplacian. The asymptotics of the latter in rectilinear domains was studied, although with different boundary conditions (Kenyon has Dirichlet boundary conditions and we would be interested in Dirichlet+Neumann ones). As far as the dependence on $b$ is concerned, (i. e., if we are only interested in the ratios $P'_n(b_1)/P'_n(b_2)$ with $b_{1,2}$ in the bulk of the right side of the square), it should be captured by the $SLE_8$ partition function, and then the same conclusion as in discussion below might hold with a different exponent (namely, $a=\frac{6-\kappa}{2\kappa}$, which for $\kappa=8$ equals $-\frac18$). But again, these conjectures are all quite far-fetched.
Old Answer
Lawler, Schramm and Werner conjectured that the number $R_n$ of self-avoiding walks in the rectangle $[-N;N]\times[0;N]$ connecting the origin to the point $(0;N)$ behaves like $$ R_n\sim c\cdot N^{-2a}\mu^N, $$ where $a=5/8$ and $\mu$ is a lattice-dependent quantity known as the connective constant (it is known to be $\sqrt{2+\sqrt{2}}$ for hexagonal lattice and only numerically known for other lattices). Moreover, they conjectured that the measure should have a conformally covariant scaling limit, meaning that if $\Omega$ is another simply-connected domain and $x,y\in \partial \Omega$, with the boundary being nice (say, horizontal or vertical straight lines) near $x,y$, then the number of SAW from $a$ to $b$ in a $\delta$-mesh lattice approximation to $\Omega$ should behave like $$ R^\Omega_n \sim c |\varphi'(x)|^{a}|\varphi'(y)|^{a}\delta^{2a}\mu^{\delta^{-1}}, $$ where $\varphi$ is a conformal map from $\Omega$ to the rectangle $(-1,1)\times(0,1)$ that maps $x$ to the origin and $y$ to $i$. (We can also fix $|\varphi'(x)|=1$, and then $|\varphi'(y)|$ is proportional to the normal derivative of the Poisson kernel with a pole at $x$.) A straightforward extension of their conjecture would be that the number of SAW from the origin to $(N,b)$, with a given $b$, behaves like $$ P'_n\sim c(\theta)^a\cdot N^{-3a}\mu^N, $$ where $c(\theta)$ is proportional to the normal derivative at $(1;\theta)$ of the Poisson kernel in $(0;1)^2$ with a "pole" at the origin, and we are in the regime $b/N\to \theta\in (0,1)$. (If $b\equiv N$ or $b\equiv 0$, the power law exponent should change to $-4a$.)
No-one doubts the conformal covariance ansatz of Lawler, Schramm and Werner; but it is wide open to prove it rigorously. The best you can do rigorously is, probably, $$ \log P'_n\sim n\log\mu. $$ I am not sure this is explicitly done anywhere, but the general idea is that all 'reasonable' models of planar SAW should obey this property with the same $\mu$; for instance, here it is done for arbitrary paths, bridges, and closed paths.
In fact the result you attribute to Kreweras-Niederhausen is equivalent via a simple reformulation to counting plane partitions in a $p\times q \times 2$ box, i.e., plane partitions of $p\times q$ shape with entries in $\{0,1,2\}$, and hence follows from MacMahon's famous formula (which applies to rectangular plane partitions with entries in $\{0,1,\ldots,m\}$ for any $m$).
(What Kreweras-Niederhausen actually prove that is new in that paper is an analogous product formula for the order polynomial of the poset $V \times [n]$.)
Similarly, your question about Dyck paths amounts to counting plane partitions of staircase shape with entries in $\{0,1,2\}$. These plane partitions (for entries $\{0,1,\ldots,m\}$ for any $m$) were enumerated by Proctor: see, e.g., his "Odd symplectic groups" paper (https://doi.org/10.1007/BF01404455).
EDIT: For another explanation, see the discussion of fans of Dyck paths on pg. 53-54 of Ardila, "Algebraic and geometric methods in enumerative combinatorics," https://arxiv.org/abs/1409.2562.
EDIT: The specific formula in your case is:
$$ \prod_{1 \leq i \leq j \leq n-1} \frac{i+j+4}{i+j}.$$
Compare to the number of Dyck paths $(0,0)\to(n,n)$ being
$$ \prod_{1\leq i \leq j \leq n-1} \frac{i+j+2}{i+j}.$$
See my explanation in picture form:
Best Answer
Since that Mirko Visontai told me that the answer is ${a+b\choose a}/(a+b)$ if $\gcd(a,b)=1$. The proof is the following (with k=a and l=b):
The number of 0--1 vectors with $k$ 0's and $l$ 1's is ${k+l\choose k}$, so we have to prove that out of these vectors exactly $1/(k+l)$ fraction is an element of $L(k,l)$. The set of all vectors can be partitioned into equivalence classes. Two vectors $p$ and $q$ are equivalent if there is a cyclic shift that maps one into the other, i.e., if for some $j$, $p_i = q_{i+j}$ for all $i$. We will prove that exactly one element from each equivalence class will be in $L(k,l)$. This proves the statement as each class consists of $k+l$ elements because $gcd(k,k+l)=1$.
We can view each 0--1 sequence as a walk on $\mathbb R$ where each 0 is a $-l/(k+l)$ step and each 1 is a $+k/(k+l)$ step. Each $(k,l)$ walk starts and ends at zero and each walk reaches its maximum height exactly once, otherwise $ak + bl = 0$ for some $0 < a +b < k+l$ which would imply $\gcd(k,l) \neq 1$. If we take the cyclic shift that ``starts from the top'', we stay in the negative region throughout the walk, which corresponds to remaining under the diagonal in the lattice path case. Any other cyclic shift goes above zero, which corresponds to going above the diagonal at some point.