With a stone wall and a deep moat, you can laugh off anything the game throws at you with the exception of drake attacks. Skeleton surprises, Fuling raids, visits from Surtlings, wolf packs, and troll incursions are all little better than jokes for the well-prepared. It's only when something powerful enough to do property damage (but, to be frank, not much property damage) flies over the wall that you need to be concerned.