Eventually enemies come along and wipe out many of the towers after your visit them. You can come along later and loot them. There are two things of importance in those towers.