The trivfloat package seems to defer (much of) the definition of its new floats using \AtBeginDocument, so you also have to use \AtBeginDocument{\numberwithin{graph}{section}} to change the floats' numbering.
\documentclass{amsart}
\usepackage{trivfloat}
\trivfloat{graph}
\AtBeginDocument{\numberwithin{graph}{section}}
\begin{document}
\section{First}
\begin{graph}[h]
(Graph content)
\caption{A graph}
\end{graph}
\end{document}
With package remreset the reset dependencies can be removed. For example,
if counter theorem is reset if counter subsection is incremented, then
\usepackage{remreset}
\makeatletter
\@removefromreset{theorem}{subsection}
\makeatother
the resetting is removed.
The other way \@addtoreset does not need a package. For example, if counter theorem should be reset if counter section is incremented:
\makeatletter
\@addtoreset{theorem}{section}
\makeatother
The appearance of the counter is controlled by \the<counter>. For example, in the latter case it could be redefined to:
\renewcommand*{\thetheorem}{\thesection.\arabic{theorem}}
Macro \numberwithin of package amsmath performs both steps, adding reset dependency and renaming \the<counter>:
\usepackage{amsmath}
\numberwithin{theorem}{section}
I hope, you get the idea. A minimal working example is missing. Thus it is not clear to me, what you are doing. Especially the numbering inside subsections is quite unusual.
Perhaps it is wise to check the preamble if this is not intended.
Also it looks even more odd, if the numbering occurs inside subsections without the section number, but with subsection number. The numbers are not unique, thus it is not clear which theorems are meant if only the reference is given.
I would keep the numbering scheme easy and understandable:
Plain numbers without resetting at all: Theorem 1, Theorem 2, …
1. First chapter
Theorem 1
Theorem 2
2. Second chapter
Theorem 3
Theorem 4
Numbering in a large entity like chapter or section, if the class does not provide chapters:
1. First chapter
Theorem 1.1
Theorem 1.2
2. Second chapter
Theorem 2.1
Theorem 2.2
Then the reader has a chance to understand the numbering without going in deciphering mode.
Best Answer
Define the counter, say,
acntrto be in the reset list ofsubsectionand make a query within\theacntrwhether the current subsection counter value is0or otherwise.If
0use\thesection.\arabic{acntr}, otherwise use\thesubsection.\arabic{acntr}.Since
\stepcounter{section}or\refstepcounter{section}will also reset thesubsectioncounter, the correct value is given at least with LaTeX kernels later than April 2015, otherwise the (outdated)fixltx2epackage must be loaded.