Had this problem a couple times on a dedicated sever playing with friends, when a friend was afk for an extended period of time. Friend had to be kicked or log out to resolve.