Might be loosely size-related thing, bigger villages have more stuff. You should also take a look inside the towers (of all sizes), they also can have totems.