This happened to me. Just not with skeletons.

I knew I could pause, but i wanted all my carrots and turnips to grow so I decided to go for a bit of an ol' poop and leave the game running.

In 100+ hours I have had a few invasions and they are all just foul smells or greydwarves.

I have a moat and a small bridge and nothing ever gets inside - so I was like - it's safe.

I came back in the middle of TROLLS invading my home. Man they knocked down my fences, trees, some of my garden and side of my base.

FML