Let me address one piece of your question, namely "I don't really know what a density function tells you". If $p$ is the probability density function of a random variable $X$, then the intuitive meaning of $p(c)$ would be described by a physicist (or an 18th century mathematician) as follows: For an infinitesimal interval containing $c$, consider the probability that $X$ lies in that interval, and consider the length of the interval. The ratio of these is $p(c)$. The decision to use a ratio here is based on the intuitive idea that, if you double the length of an interval near $c$, then that doubles the probability that $X$ will be in the interval --- a double-sized target is twice as easy to hit. And similarly if you replace "double" with "triple" or other factors. So it is reasonable to expect that (for well-behaved, continuously distributed $X$) the ratio that defines $p(c)$ will not depend (much) on the exact size of the infinitesimal interval. The probability of hitting the interval does depend sensitively on its length, but we compensate for that by dividing by the interval's length. So the probability density is the probability (of $X$ being in an interval near $c$) per unit length.
In the 19th century, mathematicians wanted more rigorous concepts, not involving infinitesimals, so this intuitive definition would be replaced with a definition involving a limit, namely that $p(c)$ is the limit, as $\epsilon\to0$, of the ratio $\text{Prob}(c-\epsilon<X<c+\epsilon)/2\epsilon$. [Technicalities: I should probably also have allowed intervals around $c$ that aren't centered exactly at $c$, but for well-behaved continuous random variables it won't matter.] [In the 20th century, infinitesimals were rehabilitated by nonstandard analysis. So we could revert to the intuitive definition that I gave first, except that we should replace "the ratio" with "the standard part of the ratio.]
I suggest that, in order to see what a probability density function does, you ignore the technicalities, ignore (at least for a while) the epsilon definition, and meditate on the 18th century definition.
EDIT, after the comment asking why we integrate $p$ to get the probability of hitting an interval: Still in the style of physics or 18th-century math, I'd get the probability of hitting an interval $[a,b]$ by imagining this interval cut up into infinitely many infinitesimal pieces, and then adding the probabilities of hitting any of those pieces. Such a sum of infinitely many infinitesimal pieces is exactly what an integral is. To see the connection with areas, imagine computing the area bordered by the graph of a positive function $f$, the $x$-axis, and the vertical lines $x=a$ and $x=b$. This area, given by $\int_a^bf(x)\,dx$, can be considered (in the 18th century) as follows. Chop up the interval $[a,b]$ into infinitely many infinitesimal intervals and consider the rectangles that have these infinitesimal intervals as bases and have heights given by $f$. (That is, an interval around $x$ has height $f(x)$.) Then the total area over $[a,b]$ is the sum of the infinitely many infinitesimal areas of these thin rectangles. The moral of this story is that integration, as in $\int_a^bf(x)\,dx$, is exactly adding up an infinite bunch of infinitesimal pieces, each of which is a product of a genuine number (like the height $f(x)$ of these rectangles) and an infinitesimal (the width of the rectangles). (In fact, the integral sign $\int$ was originally a fancy $S$, standing for "sum"; people thought of it as the sum of the products $f(x)$ times $dx$.) Coming back to probability, you have the analogous situation --- summing up infinitely many products of ordinary numbers $p(x)$ times infinitesimal widths of intervals. So the mathematical situation is the same as with areas; and so the method of computing these things is also the same, integration.
Given a family of probability distributions $\mathcal{P}$ on a measurable space $(\Omega,\mathcal{F})$, the triple $(\Omega,\mathcal{F},\mathcal{P})$ is a statistical model/space. That is, for each $P\in\mathcal{P}$, $(\Omega,\mathcal{F},P)$ is a probability space. Often $\mathcal{P}$ is an indexed family $\mathcal{P}=\{P_{\theta}:\theta\in \Theta\}$, which means that the expectation of $X$ is $\theta$-dependent and the notation $\mathsf{E}_{\theta}X$ helps to avoid confusion.
Best Answer
It's used as a grand kind of comma. In this case, I think $x$ and $y$ refer to possible values of random variables, and $\theta$ an unknown non-random parameter, and the conditional density of $x$ given $y$ might be denoted $f(x|y)$, but the author wants to emphasize the fact that the whole expression depends on $\theta$ as well, and so might have written $f(x|y,\theta).$ But because in his world $y$ and $\theta$ are different kinds of things, writes $f(x|y;\theta)$ instead.