Walter is right - when you add noise(3) to the signal you are not actually adding random noise, you're just adding some positive or negative shift. Also, the noise vector you are generating is too short.
You can try out the code below (with the changes commented). When you run it several times, you will see that the BER average is close to the theoretical BER of 0.1029.
M = 2;
k = log2(M);
EbNo = 5;
Fs = 16;
nsamp = 17;
freqsep = 8;
n=100;
msg = randint(n,1,M);
txsig = fskmod(msg,M,freqsep,nsamp,Fs);
ab=abs(txsig);
ps=(sum(ab.^2))/n;
snr=5;
pn=10.^(-0.1.*snr).*ps;
noise= sqrt(pn)*randn(1,n*nsamp);
G1=randn(1,n);
G2=randn(1,n);
v= sqrt(power(G1,2)+ power(G2,2));
A=v(2);
theta=2*pi*rand;
msg_rx = txsig + noise';
msg_rrx = fskdemod(msg_rx,M,freqsep,nsamp,Fs);
[num,BER] = biterr(msg,msg_rrx)
BER_theory = berawgn(EbNo,'fsk',M,'noncoherent')
Best Answer