I do not know of any active set theorists who think large cardinals are inconsistent. At least, within the realm of cardinals we have seriously studied.
[Reinhardt suggested an ultimate axiom of the form "there is a non-trivial elementary embedding $j:V\to V$". Though some serious set theorists found it of possible interest immediately following its formulation, Kunen quickly afterward showed that this is inconsistent, using choice. It is not known whether choice is needed, but current research suggest that, even without choice, natural strengthenings of this axiom may be inconsistent in $\mathsf{ZF}$ alone. This is hardly an argument against large cardinals in general. Instead, it provides us with natural limitations on their reach. For another example, see here.]
There are several active set theorists who do not commit themselves one way or the other to the consistency of large cardinals, but use them if necessary, and do not object on mathematical grounds to arguments that involve them. For some of them, set theory is about all possible (natural) extensions of $\mathsf{ZFC}$, and there are certainly many interesting such extensions (such as $V=L$) that rule out large cardinals. Thus, it is not that they consider the cardinals inconsistent.
(I confess, this may be ignorance on my part.)
And I know of only two mathematicians who years ago were serious set theorists and who have expressed doubts about the consistency of (certain) large cardinals. Neither is currently active within the field, and so their position should be taken with a grain of salt, since it missed the significant results from the late 80s that could very well have forced them to reconsider.
Why do we expect $\mathsf{ZFC}$ to be consistent, to begin with? We expect more than mere consistency, of course, but doubting large cardinals usually means distrust in set theory as a whole. I am not a philosopher, so I will not discuss philosophical positions or justifications. A good reference for the heuristics behind the basic ZFC axioms is the wonderful paper
Penelope Maddy. Believing the axioms. I, J. Symbolic Logic, 53 (2), (1988), 481-511. MR0947855 (89i:03007).
Large cardinals are discussed in its follow-up,
Penelope Maddy. Believing the axioms. II, J. Symbolic Logic, 53 (3), (1988), no. 736-764. MR0960996 (89m:03007).
Maddy's two books on Platonism and Naturalism discuss extensively why the view of a set theoretic universe with large cardinals rather than not is the reasonable choice, given our current understanding, see
Penelope Maddy. Realism in mathematics. The Clarendon Press, Oxford University Press, New York, 1990. MR1075998 (92h:00007).
and
Penelope Maddy. Naturalism in mathematics. The Clarendon Press, Oxford University Press, New York, 1997. MR1699270 (2000e:00009).
The books present several subtle technical points that can only be completely understood once one is aware of the deep connections between large cardinals and (generic) absoluteness. Maddy's more recent views on the subject can be seen here:
Penelope Maddy. Defending the axioms: on the philosophical foundations of set theory. Oxford University Press, Oxford, 2011. MR2779203.
How do set theorists measure the internal plausibility of large cardinal assumptions, beyond their usefulness in proving results? The point of the inner model program (and of its most recent offspring, descriptive inner model theory) is to develop fine structural ("$L$ like") models for large cardinals. These models are canonical in several precise ways, and have a rich internal structure that many set theorists take as evidence of the consistency of the large cardinals under consideration. Thanks to its advances, we have a much clearer view of the set theoretic universe nowadays (for example, we now have the different covering lemmas, and several generic invariance results) than when the program began, motivated by what we now call Gödel's program.
The program has currently reached well past Woodin cardinals, but is not yet at the level of supercompact cardinals. This can be interpreted as saying that, using the strongest tools currently at our disposal, we are fairly certain of the consistency of, say "there is a Woodin limit of Woodin cardinals". Time will tell whether the program will reach supercompactness. If it does not, this will provide us with strong evidence of their inconsistency, though I am not sure anybody actually expects this to be the outcome.
John Steel and Tony Martin have over the years refined something they call "the speech", where they explain their position towards large cardinals. It is well worth reading, and trying to summarize it in a few lines would be an injustice. It can be found in these two postings to the Foundations of Mathematics (FOM) list:
1, 2 (the notation here is $P_T =$ set of $\Pi^0_1$ consequences of $T$), and in the papers from the "Does mathematics need new axioms?" panel, see
Solomon Feferman, Harvey M. Friedman, Penelope Maddy, and John R. Steel. Does mathematics need new axioms?, Bull. Symbolic Logic, 6 (4),(2000), 401–446. MR1814122 (2002a:03007).
Steel's own views are also presented in some detail in Maddy's books. For very recent developments, see his talk:
John Steel. Gödel's program, given at the CSLI meeting at Stanford, June 1, 2013.
At the risk of not being balanced, let me point out some highlights: We have a coherent picture of the universe of sets, with large cardinals. We can, within this picture, interpret theories where there are no such cardinals. However, we do not have such a coherent picture in the opposite direction. The consequences of large cardinals, at the arithmetic level (and more, as we climb up through the hierarchy) are compatible. The arithmetic consequences of any natural extension of $\mathsf{ZFC}$ fall somewhere within this hierarchy (as far as the theories we can currently analyze), even if the theory does not mention large cardinals. In fact, determinacy statements, incompatible with choice, also fall within this hierarchy and are mutually interpretable with large cardinals (again, as far as those theories we can currently analyze). This deep connections with determinacy are behind what we now call descriptive inner model theory, see
Grigor Sargsyan. Descriptive inner model theory, Bull. Symbolic Logic, 19 (1), (2013), 1-55.
Large cardinals provide us with generic absoluteness, and generic absoluteness, a natural requirement if we are interested in understanding the projective theory of the reals, requires the consistency of large cardinals. See this answer for a bit more on this issue; let me emphasize that this is not some technical or artificial requirement, but rather a natural extension of basic results in classical descriptive set theory.
Large cardinals seem inherently necessary to mathematical practice, not just set theory. Harvey Friedman has written extensively on this issue.
In short: We have a very clear measure of progress understanding large cardinals and their consequences. By this measure, we can now understand many set theoretical issues that do not involve large cardinals but for which they are necessary in deeper ways (not just consistency-wise). This measure actually requires the large cardinals, we do not have anything like that without them. This measure is meaningful even in settings that are not set theoretical, and seems unavoidable even within mathematical practice (though it is perhaps too soon to tell how significant this will be at the end for "practicing mathematicians"). We do not have any serious mathematical model where large cardinals would be inconsistent, however, we have a serious program of research that would ultimately teach us that, were this the case. The program has provided us, instead, with many positive results (in particular, we have nice inner models for measurability, for strong cardinals, for Woodin cardinals, and we have nice inner models of models of determinacy, that capture the large cardinals that provide us with the consistency of the determinacy statements).
To conclude, we understand (motivate/explain) large cardinals within the larger context of reflection principles, the simplest of which follow already from $\mathsf{ZFC}$. (So, we have a natural generating principle for them.) On the other hand, I know of no objections to large cardinals beyond "they are too large" or "they do not feel right", neither of which seems mathematical to me. The first also seems particularly artificial.
The only 'program' towards their inconsistency (that I am aware of) instead produced many interesting consequences for the partition calculus at the level of $0^\sharp$ (and is perhaps responsible for the early theory of $0^\sharp$ itself). As far as I understand, a similar attempt to disprove measurable cardinals resulted instead in the development of the covering lemma, which has since been one of the key tools to measure our understanding of particular large cardinals as part of the inner model program, see
William J. Mitchell. The covering lemma. In Handbook of set theory. Vols. 1, 2, 3, Matthew Foreman, and Akihiro Kanamori, eds., pp. 1497–1594, Springer, Dordrecht, 2010. MR2768697. (Wayback Machine)
Perhaps I should add that our intuitions about large cardinals do not come for free, but are the result of the programs mentioned above. I am in particular suspicious of a priori mistrust of large cardinals, since it tends to hide misunderstanding, or ignorance, of the actual mathematics involved in these programs.
Best Answer
Here are some references and following them a short answer.
A good reference for the current situation to start with is John Preskill's recent paper "Quantum Computing in the NISQ era and beyond" NISQ stands for "Noisy Intermediate Scale Quantum" and it is a very useful notion coined by Preskill who also coined the notion of "quantum supremacy" - the ability of quantum computers to perform certain computational tasks hundreds orders of magnitude better than classical computers. The crucial theoretical and experimental issue is to understand quantum devices in the NISQ regime.
As for my papers: The two mathematical theorems on which the argument against quantum computers is largely based, are from my paper Gaussian Noise Sensitivity and BosonSampling with Guy Kindler; In my ICM 2018 paper Three Puzzles on Mathematics, Computation, and Games the feasibility of quantum computers is the third puzzle (Sections 1.3, 1.4 and 4) and it is also related to the Benjamini-Kalai-Schramm notions of noise-stability and noise-sensitivity which is discussed in the second puzzle (Sections 1.2, 3). Another good source is my 2016 paper from the Notices of the AMS "The quantum computer puzzle" and its arxive extended version.
On the practical side, these works give clear predictions (see e.g. my ICM paper) for what we can expect from NISQ devices and, in particular, from quantum computers with 10-80 qubits that many try to build. Those predictions are sharply different from what experimentalists hope to achieve.
On the theoretical side we have a good argument for Scott Aaronson's point number 9. (This argument does not rely on error-correlation.) We also have a clear difference between classical and quantum computing, namely why the argument against quantum computing does not apply to classical computing.
My earlier work from 2006-2012 (See e.g. this paper) indeed studied correlation of errors. These works suggest that in the NISQ regime errors for a pair of entangled qubits will have substantial positive correlation. This is another interesting prediction for current and future NISQ devices. This is part of the theoretical picture because the value of the "fault tolerance threshold" in Scott's point 9 depends on such correlation.