I would also really like to have OP's question answered. I keep structures intact and often actually repair them and/or convert them to mini camps with basics for aesthetics. Also don't like the brown flat patch they will leave behind.

Though if it would improve performance to tear them down I definitely would for most of them. Would not mind cleaning up all those map markers for stone towers etc.