Answering my own question:
The missing criterion is indeed related to the maximum flatness - the second derivative of the squared filter gain (d/(d omega))^2 G(omega)^2 with respect to frequency at omega=0 needs to be zero (the first derivative is always zero). G(omega)^2 = abs(H(exp(i*omega)))^2
This leads to the following design algorithm:
b=-2*cos(2*pi*Wnotch);
a=1+1/R;
c=1-1/R;
G=(-4*sqrt((b^2-4)*c*(a+c))+8*a+4*b*c)/(4*a^2+a*(-b^2+4*b+4)*c+4*c^2);
A = [1 1; -1 1] \ [(G*(b+2)-1); ((1/R*G*(2-b))-1)];
A = [1; A].';
B = G*[1 b 1];
Inverse Chebyshev filter with a given stopband ripple R (linear, not dB) and given notch position, without any pesky analog prototyping/bilinear transform/whatnot.
Best Answer