Well there are 5 types of buildings:
A) Small rock structure with stairs -- no totem
B) A rock fort with roof on top (always totem)
C) Full rock tower (never)
D) Very small camp -- never seen totem here
E) Big camps -- 100% chance of a totem

I didnt count boss stones wich are usualy protected by big ones.
Its also possible to get one in raids on your base.