You need to stay within the red circle on the map for a certain amount of time. The easiest way to deal with the trolls, when you're still too weak to fight them, is to just run around in circles, having them follow you, until the invasion is over. That way, they'll focus on you instead of destroying your buildings.