"...approaches for dealing with Fortran code with a lot of common blocks and transferring it over to Matlab."
My first plan would be to turn it into a mex function rather than converting it.
"with very large systems of equations, where there are a lot of variables describing the physical problem, will separate functions be faster than nested functions?"
Only profiling would tell for certain but the answer is almost certainly "no". That there has to be an invocation of the argument list on the call that isn't there in the nested function implies the obverse unless there's some other "trick" going on that the JIT compiler can take care of transparently and behind the scenes.
How much difference there would be in an absolute sense would have much to do with how the nesting is; but typically if it's the functional and jacobean, etc., etc., etc., ... they're called at the deepest level so the call/return overhead may in fact be a sizable factor overall.
Again, if it runs as the Fortran code, I'd just build an interface to that code to call from Matlab and be done.
Best Answer