I believe you are seeing the ‘rolloff’ or transition-region characteristics of a Butterworth filter. Note that you did not specify the stopband, only the passband. Start with thebuttord function, define the stopband as well as the passband, and you will likely see a filter response closer to what you expect. (Also consider using a Chebyshev TypeII design.)
Best Answer