Boar panic reasons:

Fire nearby. Any fire, cooking, forge, kiln, torchs, lamps etc.

Predators nearby. Even standing outside a wall. Leeches in swamp swimming near, wolves near, draugr.. shamans, etc..

Flooding.. if your base is low down or next to coast, the rising of the sea can flood the taming area and panic the boars.

You. Go into stealth, if they can't see you, it's fine. If they can.. they panic.

Foods: Raspberries preferred. Red Mushrooms, Blueberries, Flax? Barley? haven't checked last 2. They do not eat Yellow Mushrooms. If boar go hungry for too long, they can also panic.