You're completely right. Ultimately, the "price" of a note is determined by the person who wants it second most of all, and when a note appears at auction so infrequently, it's very hard to predict what will happen.
What continually amazes me, though, is how "affordable" the toughest WPM is relative to USPM. I collect Egypt, which is a hot area. The three absolute hardest notes, P1, P2 and P16 (issued) are going to cost low five-figures in any grade. There are only a couple dozen P1 and P2 out there to begin with, and maybe a couple of issued P16s. Unless I'm mistaken, US notes of that rarity are mid-to-high six figures. I've put together a totally solid Egypt collection by spending less than $1,000 for 99% of my notes. Much more accessible.
There is so much affordable stuff out there. Makes you wonder if WPM is not the better deal right now. Who knows, maybe some day if things really heat up, we may look back and wish we could still buy at these prices.
I am putting together sets of all the major Commonwealth countries, and I can say most of them are relatively affordable, except the Bahamas.