Yes, they are an aggro on sight item like other "player base" objects. If an aggressive mob sees it they'll attempt to destroy it.
Bear in mind if no one is remotely near nothing can happen to it, but they should always be somewhere safe so you don't come through a portal to a troll in your face.