Just ban them if it's a big deal to you. Your server, your rules. Simple.