You don't HAVE to fight them. Just ignoring them (once you have good enough stuff) works just as well.

When you are bothered by them while working, don't be. They'll end up walking in your (pick)axe themselves sooner or later.

In short... just don't bother with them.