LaTeX
Traditional fonts for LaTeX can have only 256 glyphs, which is clearly insufficient to cover multiple alphabets. This is a problem similar to legacy code pages for operating systems. However, LaTeX can use multiple fonts.
The problem has been (not completely) solved with introducing font encodings. Leaving out old and outdated seven bit encodings, the LaTeX community has developed a set of encodings.
What is a font encoding?
A font has 256 slots where putting glyphs. A font encoding is a correspondence between slots and glyphs. Having such a predetermined correspondence, LaTeX can define macros that point to the correct slot, without the user needing to know the glyphs' positions. For instance, when the encoding is T1, \'a generates instructions to print the character in slot octal 301 (decimal 193). However, when the encoding is T2A, that doesn't contain the “á” glyph, the same \'a will instruct TeX to superimpose an acute accent to an “a”.
What are the main font encodings?
T1 covers many of the European languages using the Latin alphabet
T2A covers the European languages using the Cyrillic alphabet (Russian, Ukrainian, Serbian, Bulgarian, Bielorussian)
T2B and T2C cover Asian languages using the Cyrillic alphabet
T3 covers the IPA phonetic symbols
T4 covers African languages using the (modified) Latin alphabet
T5 covers Vietnamese
LGR covers Greek
There are other encodings, notably LY1 that's a modified T1.
Do users need to bother with encodings?
Yes and no. The default encoding is, for backwards compatibility, the old seven bit OT1. One has to plan what languages the document will need. So, for a Serbian with Cyrillic script document, the initial declaration
\usepackage[T2A]{fontenc}
should be issued. Say that an English summary is needed, the declaration should be
\usepackage[T1,T2A]{fontenc}
so the main encoding for the document will be T2A, the last one specified.
A following declaration
\usepackage[serbianc]{babel}
will set up LaTeX to use hyphenation patterns and keywords (such as the ones for “chapter” or “figure”). Note that support by babel for Serbian in the Cyrillic script is very recent, so an updated TeX distribution is needed.
How do I choose a font?
The font families used for the document should support the required encodings. For the T1 encoding the choice is vast, it isn't for the T2? encodings; see What fonts are compatible with T2A (Cyrillic) encoding? for more information.
If a font does not include a set for the T2A encoding, it's impossible to use it for writing in the Cyrillic script. So, requests about using Cyrillic Times must be dismissed unless a commercial font is available. However, as the answers to the linked question show, there are free font families supporting Cyrillic. So, for example
\usepackage[T2A]{fontenc}
\usepackage[utf8x]{inputenc}
\usepackage[serbianc]{babel}
\usepackage{paratype}
will set up a document using the ParaType fonts. The inputenc call has nothing to do with fonts and their encodings, but rather with the form of the LaTeX input files; using UTF-8 for saving the files is the preferred form, nowadays.
XeLaTeX and LuaLaTeX
If one opts for a newer typesetting engine, such as XeTeX or LuaTeX, with a LaTeX format preloaded, the font choice is different. For these engines the concept of font encoding is irrelevant, because the recommended way to employ them is via OpenType or TrueType fonts and the fontspec package. For instance one can set up a document for using the ParaType fonts by
\usepackage{fontspec}
\setmainfont{PT Serif}
\setsansfont{PT Sans}
\setmonofont{PT Mono}
provided the (TrueType) ParaType fonts are installed as system fonts. The fonts are included in TeX Live, however, and can also be called from their file name, see the documentation of fontspec.
For the particular case of Serbian in the Cyrillic script, my choice would be XeLaTeX, since support for the language by babel is somewhat limited. The situation is developing, however.
Assuming XeLaTeX, here's a minimal document:
\documentclass[a4paper]{article}
\usepackage{fontspec} % use OpenType or TrueType fonts
\usepackage{polyglossia} % support for different languages
% use the ParaType fonts (or change for other fonts)
\setmainfont{PT Serif}
\setsansfont{PT Sans}
\setmonofont{PT Mono}
\setmainlanguage[Script=Cyrillic]{serbian}
If you're using mathematics in your document, don't choose the ParaType fonts, as there's not a companion math font. Here are some possibilities.
Computer Modern like fonts
\documentclass[a4paper]{article}
\usepackage{fontspec} % use OpenType or TrueType fonts
\usepackage{unicode-math} % use OpenType math fonts
\usepackage{polyglossia} % support for different languages
% use the CMUnicode fonts for text
\setmainfont{CMU Serif}
\setsansfont{CMU Sans Serif}
\setmonofont{CMU Typewriter Text}
\setmathfont{Latin Modern Math}
\setmainlanguage[Script=Cyrillic]{serbian}
Times like fonts
\documentclass[a4paper]{article}
\usepackage{fontspec} % use OpenType or TrueType fonts
\usepackage{unicode-math} % use OpenType math fonts
\usepackage{polyglossia} % support for different languages
% use the Times font for text
\setmainfont{Times New Roman}
% \setsansfont{Arial} % ???
\setmonofont{CMU Typewriter Text}
\setmathfont{TeX Gyre Termes Math}
\setmainlanguage[Script=Cyrillic]{serbian}
Palatino like fonts
\documentclass[a4paper]{article}
\usepackage{fontspec} % use OpenType or TrueType fonts
\usepackage{unicode-math} % use OpenType math fonts
\usepackage{polyglossia} % support for different languages
% use the Palatino font for text
\setmainfont{̦Palatino}
% \setsansfont{Arial} % ???
\setmonofont{CMU Typewriter Text}
\setmathfont{TeX Gyre Pagella Math}
\setmainlanguage[Script=Cyrillic]{serbian}
Further reading
The LaTeX Companion has extensive coverage of fonts, encodings and so on for LaTeX. Also fntguide.pdf has detailed information
Best Answer
"Modern" is a somewhat loaded term. You probably want to distinguish between:
You'll get lots of very good info about functional "edge of the envelope" possibilities just by typing some of the keywords mentioned in point 1 above (LuaTeX, etc.) into the searchbox at the top of this page, following links on and off this site, then asking more directed questions here as and when you'd like to learn more. You'll find that the quality of responses increases the sharper the questions you ask.
Another approach might be to search out those users with specialties you're interested in (Joseph Wright for LaTeX3, Aditya for ConTeXt, Harald and Ulrike for seemingly impossible solutions to raw TeX problems, frabjous, Lev Bishop and Andrew Stacey for academic writing, Stefan and Will for, gosh, everything ...) then "follow" the line of the questions they've answered by chasing them individually through their Users pages. This, I think, would be more fruitful than holding out for a consolidated "reference (book, free pdf, website) that would summarize the working of 'modern' latex for scientific works".
As for what's stylistically modern package-wise, you'll probably find ClassicThesis about as close to the edge as likely to be accepted in a science-oriented masters thesis (actually, probably closer to postmodern than modern). Otherwise, check out Tufte-Book for a raft of stylistically interesting ideas.