A lone portal will not be attacked unless it is close enough for aoe or in swing radius.

Just put them away from everything else.
Also check volume settings and see what one that may be and if you can turn it down happily.

Personally I like background stuff loud but it might be an effect volume or something.