Early on you'll probably want to destroy these so you don't have to deal with constant Greydwarf attacks. Later on, you're going to want to keep a few of these around and farm them for stuff. Specifically, resin. Once you start building big and have a lot of torches to feed, you're gonna need a lot of the stuff.