You sort of can, but the farming and building is a means of gearing up for tougher boss battles. Look at the screenshots to see what other people have built to see if it's what you had in mind. It's mostly for show though, no npcs in your village.

If you dig a ditch all around your area most things can not get to you from outside. Workbenches and other select buildables can keep monsters from spawning inside your base. If you gather resources and kill bosses on one world and then build on another, you can avoid the difficult raids.

You'd need to use console commands or mods if you want to avoid fighting.