The main problem is that \leftmark doesn't contain \MakeUppercase{\bibname}, but a more complicated thing. There are also other expansion problems.
If your backmatter is in that form, I suggest
\newif\ifbiblio
\newcommand{\backmatter}{%
\fancyhead[C]{\ifbiblio \bibname\else\appendixname\ \thechapter\fi}%
\bibliotrue
}
\appto\appendix{\clearpage\bibliofalse}
The conditional will be true from \backmatter until after the page change for the start of the appendices.
The following code will remove the phrase "Appendix. " in front of "Bibliography" in the table of contents when using the LaTeX amsbook class or the LyX document class book (AMS).
\usepackage{xstring}
\renewcommand{\tocappendix}[3]{%
\indentlabel{\IfStrEq{#3}{Bibliography}{}{#1}\@ifnotempty{#2}{ #2.\quad}}#3}
For LyX, this code can be inserted into the LaTeX Preamble ("Document > Settings...").
Background:
For typesetting the table of contents, the ".toc" file is used (it can be found in the "tmpdir" / "tmpbuf" directory of LyX). It contains lines like
\contentsline {chapter}{\tocappendix {Appendix}{A}{My First Appendix}}{3}
\contentsline {chapter}{\tocappendix {Appendix}{}{Bibliography}}{3}{appendix*.1}
Furthermore, the class file amsbook.cls defines
\newcommand{\tocappendix}[3]{%
\indentlabel{#1\@ifnotempty{#2}{ #2}.\quad}#3}
By redefining the command \tocappendix we can omit the printing of the first argument (e.g., "Appendix") if the third argument is "Bibliography". For string comparison, the xstring package is used. Furthermore, the bracket } in front of .\quad was moved behind \quad.
Best Answer
One should distinguish between the Table of Contents and the bookmarks: the former is part of the printed document, the latter is a property of the PDF previewer. From what you say the problem seems to be with the bookmarks.
Add
\usepackage{bookmark}after loadinghyperref(it's one of the few packages that should be called so) and, just before the bibliography in your document, say