I've found a totem in a couple of small villages (3 buildings, 1 shaman and 4 fulings) and I've had 2 large villages with 3 berserkers, 4-6 shaman, a horde of ordinary fulings, 2-3 watch towers, 8-12 huts, 2 bonfires, 4 stash chests, and 3 fields without a single displayed or stashed totem.