Actually, the problem is that you domatrix operation but instead, you need to doelementwise operation, which means using ./ and .* instead of / and *. Try this:
Your error is that inside myfun, you call fsolve with myfun as an argument. So fsolve will then call myfun, which will call fsolve, etc. This rather incestuously recursive behavior will cause fsolve to fail in frustration. It won't return anything useful either as you have written the code.
Learn to use fsolve. Do so by reading the tutorials for fsolve. READ THE HELP. There are examples of use in there.
Best Answer