The game is very doable solo. I think bosses have less hp when soloing but don't quote me on that.
The biggest difficulty I've seen is that the game doesn't have an organic progression, so it can be deceiving sometimes and some things are hard to know without guides or dying.
Regarding your problem, moats usually are more than enough to stop raids on their tracks. Unless your moat is not deep or steep enough. I trapped a troll and a few greys in a hole that wasn't even troll-deep.

Oh and by the way. Trolls and greys will follow you into the sea and attack you even when swimming. They can sink your raft pretty easely if they come in a full patrol and you don't have arrows (you can't hit things downwards except with the 2-handed hammers, which could potentially deal damage to your raft also, or bows).