Hi Damian, I think first we have to make a significant distinction in the interest of clarity...
EQing to remove a resonant frequency refers specifically to searching for an undesireable overtone that is already evident in the raw recording. For example, a rhythm guitar performance may have a constant ringing undampened open string enharmonic to the rest of the notes being played. As a result, it is distracting from an otherwise fine recorded performance and needs to be de-emphasised.
In other words, because you can ALREADY HEAR the ringing note, by boosting an eq with a narrow "Q" and sweeping through the frequencies, the aim is to get that tone (that you can already hear, although it may be somewhat masked by the surrounding audio) to "jump out", at which point you can then turn it down, thereby minimising the negative effect of the resonant frequency.
The same principle and process can apply to virtually any sound... For example, a snare drum "boing" can be identified by the same process... But you already probably know that...As you've already observed, when boosting and sweeping with a narrow Q, EVERYTHING sounds weird....
The key to it is already being able to ALREADY HEAR the sound you want to get rid of BEFORE you start sweeping - thus my example above of the ringing open guitar string. I often find singing or humming the frequency that is annoying me to be helpful. Some mixers like to use a keyboard to help identify the specific note they are hearing.
Keep in mind that resonant frequencies often have overtones above and below the fundamental, so for example, if the problem is at 200hz, then the octaves above, at 400 & 800hz may be contributing to the problem.
However, as you have kind of implied in your question, not all reductive eq is done with the goal of finding resonant frequencies per se, and this is where it can get confusing watching mix tutorials, because some mixers don't always specify the reasons why they are using eq...
In many cases, the use of reductive eq is just to more broadly de-emphasise freqency AREAS that mask other sounds, or are not needed in the context of the mix. To a greater or lesser extent, this may be an taste/artistic/stylistic decision.
In this case, the same principle applies - that you have to already have an idea of what you don't like about the sound BEFORE you start sweeping around - but usually the sweeping can be done with wider "Q" settings. The sweeping method effectively becomes the audio equivalent of using a magnifying glass to find something that you know you have dropped.
As with anything in mixing, the first step is to LISTEN before doing anything. Only after first identifying the problem, then you can start using the tools at hand (in this case EQ) to solve the problem that you've already identified.
...and as with anything else in music, it takes time and practice to train our ears to hear and identify these things, so that is a BIG factor in the process,
I hope that answers your question - it's actually quite a tricky thing to explain clearly!