They will totally aggro on you as long as that Dvergr ward is intact.

one fun thing you can do, is if you already have some refined eitr, bring it over to the ward and plop them down next to the ward and the damage bits coming off the eitr will eventually destroy the ward without you being recognised as the culprit.

Once the ward is down, the dvergr will only aggro if you attack them directly. Oh, you can also use the eitr to kill them too without issue.

But yeah, as long as the ward is active it designates that whole zone as dvergr property until it is destroyed and any building or personnel damage you make directly will draw aggro on you.