Yes, I think they spawn at night. I've heard that if you don't place the trophy you got, then they won't spawn, but too late now

I would secure the base (I like to have a moat) and sleep at night so you don't have to deal with them once a day.