This types of problems are solved by observing projectile movements in $x$ and $y$ direction separately. In $x$ direction you have constant velocity movement
$$v_x = v_{x0} = v_0 \cos(\theta), \; (1)$$
$$x = v_{x0} t +x_0 = v_0 \cos(\theta) \; t +x_0, \; (2)$$
and in $y$ direction you have constant acceleration movement with negative acceleration $-g$
$$v_y = - g t + v_{y0} = - g t + v_0 \sin(\theta), \; (3)$$
$$y = - \frac{1}{2} g t^2 + v_{y0} t + y_0 = - \frac{1}{2} g t^2 + v_0 \sin(\theta) \; t + y_0. \; (4)$$
Your initial conditions are
$$x_0 = 0, \; y_0 \ne 0,$$
and final conditions (at moment $t=T$ projectile falls back on the ground) are
$$t = T, \; x = d, \; y = 0.$$
If you put initial and final conditions into equations (2) and (4) you end up with two equations and two unknowns $v_0, T$. By eliminating $T$ you get expression for $v_0$.
My calculations show that
$$v_0 = \frac{1}{\cos(\theta)}\sqrt{\frac{\frac{1}{2} g d^2}{d \tan(\theta)+y_0}}$$
which is I believe equal to your equation. Maybe your problem is that $d$ means displacement in direction $x$, while the total displacement is $\sqrt{d^2+y_0^2}$?
Given the mass m of the ball, the incident normal speed v, and the coefficient of restitution $\rho$, Then the integral of F over the duration of the collision $\Delta t$ is $$\int_0^{\Delta t} F dt = \frac{m(1 + \rho)v }{\Delta t}$$ assuning no rotational effects are incurred.
This follows from the fact that at any instant the acceleration of the ball away from the wall is F/m, so the integrated acceleration over the duration of the collision is the total change in normal velocity of the ball. Since the post-bounce velocity is simply the approach velocity times the coefficient of restitution, the total velocity change is as indicated.
If you want to simplify the force profile to assume a constant force FB during the collision (which is clearly not accurate) then $$\frac{(FB)(\Delta t)}{m} = (1 + \rho)v $$
The collision clearly does not produce a uniform force level over the duration of the collision, because the elastic forces on the ball will vary with the amount of deformation of the ball.
Best Answer
Exert a constant force $F$ pointing from the origin in $(x,y)$ to $(x_1,y_1)$ at angle $\theta$, for a period of time $\Delta t$, so that:
$$F=ma,$$ and: $$v_1=\frac{F}{m}\Delta t$$
Choose the angle $\theta$ so that $\vec{v_1}$ is tangential at $(x_1,y_1)$ to your trajectory $y=f(x)$ (I assume other forces will then take over, hence the curvature).
If it makes things easier, you can later skip from one coordinate system to the other, e.g.:
$$x'=x-x_1$$ $$y'=y-y_1$$
Edit: (in response to OP's edit to the question)
The initial velocity and angle of launch of the ball could be simply modelled as follows, see diagram below:
Some assumptions need to be made:
Due to assumption 2. the situation now becomes equivalent (simply change frame of reference) to the ball falling onto a stationary shoe with speed $v_b$ and rebounding.
$v_b$ is the radial component of $\vec{v}$ and is simply (in scalar form):
$$v_b=v\cos\theta,$$
and from simple trigonometry we can derive:
$$\tan\theta=\frac{R-h}{\sqrt{R^2-(R-h)^2}}.$$
If a restitution coefficient $r$ needs to be taken into account, then:
$$v_b=vr\cos\theta$$
Note also that with $y=f(x)$ the trajectory's function, then:
$$\tan\theta=\bigg(\frac{dy}{dx}\bigg)_{x_1}=y'(x_1).$$