Are you sure there weren't just another 6 or 10 lying in wait? They don't all come out of their hidy holes straight away all at the same time.

Just saying this is an odd post to me as I've cleared many villages and never touched the buildings or put up workbenches and they are devoid of life. Usually they get taken over by lox if anything lol