While using ghost command in my base i have never had an event start, if you want to fight as normal out of your base just use ghost again to disable it once your away from your base as events only trigger at the players position if your near anything you have built .
You also have the option to use the command stopevent which will end any event that starts ,note it only stops the current event though and not anymore in the future .
Also if it is trolls bothering you most don't build near troll caves there is a online tool called Valheim world generator which if you put in your world seed will show you where all troll caves are, apart from trolls and drakes as said in a previous comment stone walls and a moat will stop other mobs easy or build an earth wall using your hoe