I usually murder them all. They will primarily aggro you. If you are around them, they wont attack your base.