More so than even that, to really defend/take a castle you only need muster 70 odd people. Sure FC/Goon want all the castles, but if all the sieges happen at the same time, or they can't bring their numbers to a fight, well, they may take some serious hits/lose a few. Technically it is a fair fight, not withstanding the whole siege aspect. The initial land rush is going to be something though. Not to mention you could make the lives of people there hell. Mind you, in the beginning, they're going to rake in serious profit from the land taxes.