Please note that the options 10pt, 11pt, and 12pt of the \documentclass command set the default font size for the entire document. Some document classes, notably the memoir class, recognize additional settings for the default size of the text fonts. However, no document classes I'm familiar with recognize the 5pt option.)
To reduce the font size locally, e.g., just for a tabular environment that's too large to fit into the text block, LaTeX provides the commands \small, \footnotesize, \scriptsize, and \tiny. These instructions are relative to the document's main font size: the \small instruction reduces the font size by about 10 percent, while the \tiny instruction does so by about 50 percent. Hence, if you've specified 12pt as the default font size and if you're using one of the "standard" LaTeX document classes such as article, report, and book (or one of the document classes that are based on the standard classes), text set in a \tiny group will we rendered at 6pt.
To limit the scope of the fontsize change to the tabular environment in question, you should delimit the part of the document with a pair of curly braces, { and }. Hence, your tabular environment might now look like this:
{\footnotesize % change the font size
\begin{tabular}{ *{4}{R{3.5cm}} }
...
\end{tabular}
} % end of scope of \footnotesize macro
Finally, if you encase the tabular environment inside a table environment, you needn't even use the braces to keep the fontsize-changing command local. Just issue the command \small, \footnotesize, etc. immediately after the begin{table} instruction (and before the \begin{tabular}{...} instruction, and the command's scope will end automatically at the \end{table} statement.
From info latex:
\fontsize{size}{skip}
Set font size. The first parameter is the font size to switch to;
the second is the \baselineskip to use. The unit of both
parameters defaults to pt. A rule of thumb is that the
baselineskip should be 1.2 times the font size.
As @frabjous noted, you'll need to add \selectfont afterwards to make it kick in.
Best Answer
Your question shows a misunderstanding, you can have fonts any size you like
the options that you mention are not font sizes or even lengths, they are option names that set a whole range of things, the default font size, the page size, vertical spaces used around display environments and lists etc.
The names are just vaguely reminiscent of lengths so you can remember what they do. the
10ptand12ptoptions do, amongst other things, set the default font size to 10pt and 12pt respectively. The11ptoption doesn't set anything to 11pt, it sets the default font size to 10.95pt for historical reasons, but the option name is just a name.For the default Computer Modern font family the font sizes are restricted to a range of sizes (but not just 10,11,12pt, the actual list of allowed sizes is 5pt 6pt 7pt 8pt 9pt 10pt 12pt 10.95pt 14.4pt 17.28pt 20.74pt 24.88pt) but that restriction is just for historical compatibility and if you add the
fix-cmpackage you can have Computer Modern at any size, just as you can other font families such as the latin modern I used above.