Lyx can handle a variety of math mode items, but where it cannot handle them you have to insert raw LaTeX code ('ERT'). Other than extending the Lyx binary, there is not much that can be done here. (Even then, there will still be limits to what Lyx can handle.)
If you setup your document to use XeTeX, it works just as it should: You write \text then in the little box you enter your text in Hebrew as is. The only caveat is that LyX will display the text in the formula reversed, but in the output it is fine.