Having an interesting discussion about that last night.
I saw a picture of an impressive Torquay following at Bromley (or Borehamwood I forget which one) and I just made the comment that I bet a lot of them are exiles, and it wouldn't surprise me if there were more Torquay fans in and around London/the Southeast than County fans.
I reasoned that lovely though Torquay/Torbay is, it's the sort of place that lacks opportunities for jobs, education, entertainment amenities. Whereas Stockport being bigger and with Manchester slap there isn't as much.
Same true of Yeovil, Hereford, Barrow etc...
Not saying we don't have a sizeable diaspora, especially around London/SE, but not as big percentage/numbers wise as others.