Oh I can have a ton of merchants, I just didn't train up to my limit is all. You can stack merchants in any army stack (so: a unit of Peasants) which is standing in the right spot. No fort needed. A general might be better though, as your army can get bribed by the AI (in that case the merchants seem to just disappear)
A market gives 1 merchant each (except the very top level: 2)
A castle can give you 1 merchant as well
A Large City can build a merchant bank (+1 merchant), while a huge city can upgrade that for a further +1 merchant.
So a maxed out Huge city can get you 4 merchants, while a built up Large City can give you 2.
With my stacked merchants, yeah you can spend 500 on a merchant and he'll give you between 300-1000 a turn, so they're a method that can greatly increase your cashflow (say something like 500 a turn per settlement early on, it's handy but not major).
When building a massive empire though, it's still unneeded, your tax income is enough to ensure you'll never run out for building. If I removed my merchants, I could still support all my building and double or triple my military.
(If I built merchants to the cap I could have like 6x the military which would be something like 60-80 stacks?)
I took a look. There's a pattern, more unique items traded seems to do it, with something for roads perhaps...
It may be also affected by your trade bonus, but since I have a ton on every settlement (Master Merchants' Guilds) if so it's probably maxed out for these settlements. Note some of the very high growths though...