This combinatorial question has a musical motivation, which I provide below using as little musical jargon as I can. But first, I'll present a purely mathematical formulation for those not interested in the motivation:
Define a signature as a 7-tuple over the set $\{1,2,3\}$ such that the sum of the elements in the tuple is $12$. Two signatures are said to be equivalent if they are either identical or there is a circular shift relation between them (i.e. one can be circularly shifted between 1 and 6 times to obtain the other).
How many unique signatures are there?
Most modern, Western music is based on the equal temperament system, which divides the octave logarithmically equally into 12 notes.
Let's refer to 2 adjacent notes from these 12 as being "1 step" apart, two notes with one skipped note in between them as being "2 steps" apart, and so on.
The (arguably) most natural scale is the major scale, which uses 7 notes from these 12, and has the following signature:
$$\text{major signature}=(2,2,1,2,2,2,1)$$
This means that we can construct a major scale as follows: given any note to start from, the second note is 2 steps away from first note, the third note is 1 step away from the 2nd note, and so on according to the above signature.
Now this signature really has not just one, but seven scales embedded in it. This is because we can circularly shift the signature, effectively meaning that we are picking a different degree of the major scale to serve as our home note (these seven 'permutations' are called the modes of the signature). Thus, for example, $(1,2,2,2,1,2,2)$ is not a new signature, but just the major signature circularly shifted left twice (and is called the Phrygian mode).
My question is: how many unique 7-note signatures are there under the restriction that any signature must not contain an interval greater than 3 steps (this is to respect the fact that any 7-note scale in common use uses only 1, 2 and 3 step intervals, to the best of my knowledge).
Some signatures in common use are:
$$\text{harmonic minor signature} = (2,1,2,2,1,3,1)$$
$$\text{melodic minor signature} = (2,1,2,2,2,2,1)$$
$$\text{harmonic major signature} = (2,2,1,2,1,3,1)$$
The number of possible 7-note scales within the 12-note system is simply given by multiplying the number of unique signatures by 7.
Best Answer
Since $7$ is prime and there is no $7$-note signature that sums to $12$ with all $7$ steps identical, we don't have to worry about periodicity; we can just divide by $7$ in the end. Thus, we just have to count the number of ways of distributing $12-7=5$ balls into $7$ bins with capacity $3-1=2$. There are $\binom75=21$ ways to have $5$ steps of $2$, $\binom7{1,3,3}=140$ ways to have $3$ steps of $2$ and $1$ step of $3$, and $\binom7{2,1,4}=105$ ways to have $1$ step of $2$ and $2$ steps of $3$, for a total of $21+140+105=266$ scales in $266/7=38$ cyclically inequivalent types.
In the present case, inclusion-exclusion would be a bit of an overkill, but since you said you'd like a method that generalises to any number of notes with any number of maximum steps, let's generalise: For $k$ notes with a maximum of $m$ steps that sum to $12$, we want to distribute $12-k$ balls into $k$ bins with capacity $m-1$. As explained at Balls In Bins With Limited Capacity, inclusion-exclusion yields a count of
$$ \sum_{t=0}^{12-k}(-1)^t\binom{12-k}t\binom{12-k+k-tm-1}{12-k-1}=\sum_{t=0}^{12-k}(-1)^t\binom{12-k}t\binom{11-tm}{11-k}\;, $$
where, contrary to convention, binomial coefficients with negative upper index are taken to be zero. For the present case of $k=7$, $m=3$, this again yields
$$ \sum_{t=0}^7(-1)^t\binom7t\binom{11-3t}6=\binom{11}6-\binom71\binom86=266 $$
signatures. If $k$ isn't prime, or if it divides $12$, then you have to do a bit more to deal with periodicity; otherwise, you can just divide the above result by $k$.