I've noticed that it is possible to cut down trees and break rocks within the radius of the ward. However, if I were to start mining the marble walls of a Dverger outpost within that radius I pull aggro. Off the top of my head I can't tell you for sure how mining the skull will go, but my hunch is it will draw aggro being a marble structure. Can you test and confirm here for us all?