We can get a similar result to the proposition you mention, if we assume the diagram is connected and non-empty.
Proposition. Let $I$ be a connected and non-empty category and let $\mathcal{C}$ be some category that has limits of type $I$. Fix some object $C$ in $\mathcal{C}$. Then $\mathcal{C}/C$ has all limits of type $I$ and they are calculated in the same way as in $\mathcal{C}$, in the sense that the forgetful functor $U: \mathcal{C}/C \to \mathcal{C}$ preserves limits of type $I$.
Proof. Let $F: I \to \mathcal{C}/C$ be some diagram. Denote by $U: \mathcal{C}/C \to \mathcal{C}$ the forgetful functor. Then as you already noted, we have a limiting cone $\lim UF$ in $\mathcal{C}$ with projections $p_i: \lim UF \to UF(i)$ for each object $i$ in $I$.
Now let $i$ be any object in $I$, then $F(i)$ is an object in $\mathcal{C}/C$, so it is some arrow $f_i: UF(i) \to C$ in $\mathcal{C}$. Define $\ell: \lim UF \to C$ as $\ell = f_i p_i$. This does not depend on the choice of $i$, which follows from the assumption that $I$ is connected. (This is the point where I hoped to draw a diagram, but I cannot make it work properly. So if someone else can, please do! In the meantime, try drawing it yourself on a piece of paper.) To see this, let $j$ be some object in $I$. There is a sequence of arrows between $UF(i)$ and $UF(j)$. For every step $k$ in this sequence we have a projection $p_k: \lim UF \to UF(k)$ and an arrow $f_k: UF(K) \to C$, such that everything commutes and $i$ and $j$ really give the same arrow $\ell$.
Now we do find a good candidate for the limit in $\mathcal{C}/C$, namely $\ell: \lim UF \to C$ together with the same set of projections $p_i$. This does indeed form a limit. Let $d: D \to C$ together with projections $q_i$ be some cone of $F$ in $\mathcal{C}/C$. Then $D$ together with $q_i$ forms a cone in $\mathcal{C}$. So there is an induced morphism of cones $u: D \to \lim UF$. Now we only need to check that $u$ is indeed an arrow in $\mathcal{C}/C$ as well. Let $f_i: UF(i) \to C$ be some object in the diagram of $F$, then because $q_i$ is an arrow in $\mathcal{C}/C$:
$$
d = f_i q_i,
$$
and since $u$ is a morphism of cones we have $q_i = p_i u$, so
$$
f_i q_i = f_i p_i u,
$$
finally by the definition that $\ell = f_i p_i$:
$$
f_i p_i u = \ell u.
$$
So summing up we have indeed
$$
d = f_i q_i = f_i p_i u = \ell u,
$$
as required. QED.
If the diagram is not connected, or if it is empty, we have no hope of the above proposition being true in general. Even if we assume $\mathcal{C}$ to have all limits. Consider the following two examples.
Example 1. No matter what category $\mathcal{C}$ and object $C$ we start with, the category $\mathcal{C}/C$ always has a terminal object and it is given by $Id_C: C \to C$. So if $\mathcal{C}$ already had a terminal object $1$, and we take $C$ to be non-terminal, then the forgetful functor does not preserve the terminal object.
Example 2. Let us consider $\mathbf{Set}$, the category of sets. Let us consider the set $\mathbb{N}$ of natural numbers, together with the subsets $E$ and $O$ of even and odd numbers respectively. We can naturally find $E$ and $O$ in $\mathbf{Set} / \mathbb{N}$ as well, by just considering the inclusions $E \hookrightarrow \mathbb{N}$ and $O \hookrightarrow \mathbb{N}$. The product of $E \times O$ in $\mathbf{Set}$ is just their cartesian product (with the obvious projections). The product in $\mathbf{Set} / \mathbb{N}$ does exist, but this is the empty set (with the empty function to $\mathbb{N}$)! This last part will be clear in a bit, when we prove that products in $\mathbf{Set} / \mathbb{N}$ are given by pullbacks in $\mathbf{Set}$ (so in this case, by the intersection $E \cap O$).
If we are just interested in $\mathcal{C}/C$ being complete, we have the following result.
Proposition. If $\mathcal{C}$ is complete, then so is $\mathcal{C}/C$.
This result does (implicitly) appear in most books about topos theory. When proving that for any topos $\mathcal{E}$ the slice topos $\mathcal{E}/X$, by some object $X$ from $\mathcal{E}$, is again a topos, one has to show that $\mathcal{E}/X$ is complete (although, technically this is about being finitely complete, but it easily generalises). This part of the proof only uses completeness of $\mathcal{E}$. For example, a proof can be found in Sheaves in Geometry and Logic by MacLane and Moerdijk, at the start of theorem IV.7.1. I will present a (sketch of a) proof here as well, so we can link it to the proposition at the start of this answer.
Proof. As mentioned in example 1 above, the category $\mathcal{C}/C$ always has a terminal object. By the proposition at the start of this answer, $\mathcal{C}/C$ has equalisers (and they are in fact 'the same' as in $\mathcal{C}$). So all we need to check is products. So let $(A_i \to C)_{i \in I}$ be a non-empty set of objects in $\mathcal{C}/C$. Form their wide pullback $P$ in $\mathcal{C}$. There is only one arrow $P \to C$ to be considered, and this will be the desired product in $\mathcal{C}/C$ (check this!). We have now shown that $\mathcal{C}/C$ has all small products and equalisers, so it is complete. QED.
We have essentially obtained a way to calculate limits in $\mathcal{C}/C$. For any diagram $F: D \to \mathcal{C}/C$ we obtain a diagram $F'$ in $\mathcal{C}$ by just 'forgetting' that we lived in $\mathcal{C}/C$. So I do not mean to just apply the forgetful functor here, because we want to keep all the arrows to $C$ in our diagram $F'$ (another way to describe this would be to apply the forgetful functor, and then add all the arrows to $C$ back in). Now we calculate the limit $\lim F'$ of $F'$ in $\mathcal{C}$. Since $C$ was in the diagram $F'$, we have a projection $\lim F' \to C$ and this will be the limit in $\mathcal{C}/C$.
The connection with the propisition at the start of this answer is that if $F$ is non-empty connected, we do not need to keep $C$ in the diagram to make things work.
Best Answer
First of all, that wikipedia definition is not great. You should however go back to it once your wrap your head around all this stuff.
When one first encounters the definition of a universal construction, it's a bit weird. The true way to understand this concept is with examples: just keep looking at examples until it makes sense.
Let's start with a simple example. Let $X, Y$ be sets. Then (one possible way) I can define the disjoint union is $$ X \amalg Y = \{(x, 0), (y, 1) \mid x \in X, y \in Y \} $$ and we can define injection morphisms $$i_X: X \to X \amalg Y \qquad i_X(x) = (x, 0)\\ i_Y: Y \to X \amalg Y \qquad i_Y(y) = (y, 1) $$ Now suppose I have a function $f: X \to Z$ and $g: Y \to Z$. Then I can construct a unique (this is really key here) map $h: X \amalg Y \to Z$ where $$ h(x, 0) = f(x) \text{ and } h(y, 1) = g(y) $$ Why is it unique? Because the definition depends directly on $f$ and $g$. So what this really gives me is this diagram
Given the arrows $f:X \to Z, g: Y \to Z$, we can get a unique arrow $h: X \amalg Y \to Z$. The forced existence of $h$ is indicated by the dashed arrow.
This is an example of a universal construction. Actually, this is an example of a colimit, which I will explain. This happens in so many places in math that there's a name for it, and thats what this universal arrow stuff is about.
So suppose $\mathcal{C}$ and $\mathcal{D}$ are categories with a functor $F: \mathcal{C} \to \mathcal{D}$. A universal morphism from an object $D$ to the functor $F$ is a pair $(C, u: D \to F(C))$ such that, for any $f: D \to F(C')$, the following diagram holds.
So if you have a morphism $f: D \to F(C')$, you automatically get a morphism $h: C \to C'$.
A colimit is an example of this construction. To understand this, you first need to understand the diagonal functor $$ \Delta: \mathcal{C} \to \mathcal{C}^J. $$ Here, $\mathcal{C}^J$ is the functor category of functors $F: J \to C$, with morphisms as natural transformations. This functor $\Delta$ takes each object $C$ to the functor $F_C: J \to C$, where for each $j \in J$ $$ F(j) = C. $$ So it sends it to a constant valued functor.
Now when we speak of a "colimit" in a category $\mathcal{C}$ it's with respect to some functor $F: J \to C$. This is an element of the functor category $\mathcal{C}^J$. So, we define a colimit to be a universal morphism from $F$ to $\Delta$. That is, $$ (\text{Colim }F, u: F \to \Delta(\text{Colim} F). $$ Note that $u$ is natural transformation; as I said earlier $\Delta$ sends objects to functors. So, you get the diagram
However, this isn't very intuitive. A better way to look at this is to realize that if you have a natural transformation $u: F \to \Delta(\text{Colim } F)$, then you have a family of morphisms. How so? For each object $i \in J$, our natural transformation should give us a morphism $$ u_i: F(i) \to \Delta(\text{Colim } F)(i). $$ But $\Delta(\text{Colim } F)$ is a constant valued functor. So this really becomes a family of morphisms $$ u_i: F(i) \to \text{Colim }F. $$ This is usually how people define colimits, but since you asked how they're related to universal constructions, this is how. Anyways, one way to imagine the above diagram is to picture the one below. This isn't exactly precise, but it's a good way to imagine the colimit.
Compare this diagram with the one in the beginning with the coproduct. If you understand that, then you can understand the concept of limits because limits are the same exact story, just with the arrows reversed.