That is basically my question, it arose when I saw an article (here is the scientific paper, which should be free to read) saying two Caltech scientists might have found the 9th planet of the solar system.
[Physics] How to see planets thousands of light years away but don’t know if there are more planets in the solar system
astronomyexoplanetsplanetssolar system
Related Solutions
This was previously a comment to space_cadet's answer but became long (down-vote wasn't me though).
I don't understand space_cadet's talk about unstable orbits. Recall that two-body system with Coulomb interaction has an additional $SO(3)$ symmetry and has a conserved Laplace-Runge-Lenz vector which preserves the eccentricity. Because interactions between planets themselves are pretty negligible one needs to look for explanation elsewhere. Namely, in the initial conditions of the Solar system.
One can imagine slowly rotating big ball of dust. This would collapse to the Sun in the center a disk (because of preservation of angular momentum) with circular orbits and proto-planets would form, collecting the dust on their orbits. Initially those planets were quite close and there were interesting scattering processes happening. The last part of the puzzle is mystery though. If there were still large amount of dust present in the Solar system it would damp the orbits to the point of becoming more circular than they are today. The most popular explanation seems to be that the damping of the eccentricity was mediated by smaller bodies (like asteroids). Read more in "Final Stages of Planet Formation" - Peter Goldreich, Yoram Lithwick, Re'em Sari.
The answer kind of depends on how old you are. At a very introductory level, say, maybe middle school or younger, it's "okay" to refer to Jupiter as a failed star to get the idea across that a gas giant planet is sort of similar to a star in composition. But around middle school and above (where "middle school" refers to around 6-8 grade, or age ~12-14), I think you can get into enough detail in science class where this is fairly inaccurate.
If you ignore that the solar system is dominated by the Sun and just focus on mass, Jupiter is roughly 80x lighter than the lightest star that undergoes fusion. So it would need to have accumulated 80 times what it already has in order to be a "real star." No Solar System formation model indicates this was remotely possible, which is why I personally don't like to think of it as a "failed star."
Below 80 MJ (where MJ is short for "Jupiter masses"), objects are considered to be brown dwarf stars -- the "real" "failed stars." Brown dwarfs do not have enough mass to fuse hydrogen into helium and produce energy that way, but they do still produce their own heat and glow in the infrared because of that. Their heat is generated by gravitational contraction.
And Jupiter also produces heat through both gravitational contraction and differentiation (heavy elements sinking, light elements rising).
Astronomers are not very good at drawing boundaries these days, mostly because when these terms were created, we didn't know of a continuum of objects. There were gas giant planets, like Jupiter and Saturn, and there were brown dwarf stars, and there were full-fledged stars. The line between brown dwarf and gas giant - to my knowledge - has not been drawn. Personally, and I think I remember reading somewhere, the general consensus is that around 10-20 MJ is the boundary between a gas giant planet and brown dwarf, but I think it's fairly arbitrary, much like what's a planet vs. minor planet, Kuiper belt object (KBO) or asteroid.
So during Solar System formation, was there a chance Jupiter could have been a star and it failed ("failed star!") because the mean Sun gobbled up all the mass? Not really, at least not in our solar system. But for getting the very basic concept across of going from a gas giant planet to a star, calling Jupiter a "failed star" can be a useful analogy.
Best Answer
The problem with finding a new planet in our solar system is not that it is too faint, but knowing where to look in a big, big sky. This putative planet 9 is likely to be in the range 20-28th magnitude (unless it is a primordial, planet-mass black hole, in which case it will be invisible except for any accretion luminosity). This is faint (especially at the faint end), but certainly not out of reach of today's big telescopes. I understand that various parts of the sky are currently being scoured, looking for a faint object with a (very) large parallax.
The problem is that whilst it is comparatively easy to search large areas of the sky quite quickly if you are interested in bright objects; to do deep searches you are normally limited (by time) to small areas. And you have to repeat your observations to find an object moving with respect to the background stars.
If planet 9 had been a gas giant, it would have been self-luminous, due to gravitational contraction, and would have been picked up by infrared surveys like 2MASS and WISE. But the suggestion is that it is rocky or icy, is only observable in reflected light from the Sun and is hence a very faint object at visible wavelengths.
With exoplanets around other stars that can be hundreds or thousands of light years away, you know where to look - basically close to the star. The solid angle that you have to search is comparatively small. That being said, there are other problems to overcome, mostly the extreme contrast in brightness between planet and star, which means that the only directly imaged exoplanets (or low-mass companions) to other stars are much more massive (by at least an order of magnitude) than the possible new planet 9. Indeed if these objects existed in our solar system we would have easily found them already in infrared all-sky surveys such as 2MASS and WISE.
The smaller planets that have been found around other stars are not found by directly imaging them. They are found indirectly by transiting their parent star or through the doppler shift caused by their gravitational pull on their parent star. For an object in our solar system that is far away from the Sun then the first of these techniques simply isn't possible - planet 9 will never transit in front of the Sun from our point of view. The second technique is also infeasible because (a) the amplitude of the motion induced in the Sun would be too small to detect and (b) the periodic signal one would be looking for would have a period of about 20,000 years! All of the indirectly detected exoplanets have periods of about 15 years or less (basically similar to the length of time we have been monitoring them).
It is also worth emphasising, that if we observed our solar system, even from a nearby star, it is unlikely we would pick up planet 9, but we would find Jupiter, Saturn and possibly one of the inner planets if it happened to transit. In other words, our census of exoplanets around other stars is by no means complete. See If Alpha Centauri A's solar system exactly mirrored our own, what would we be able to detect? for more details.