The only penalty this game has is if you keep losing your equipment by dying and not able to get back to your corpse, and also occasional base raids can cause damage and destroy buildings.
They should be rebuildable no problem... and ofcourse you can build defenses but the bigger attacks require your attention. So far with my friend we havent lost anything from our bases ever... but we have always responded to those attacks so we made sure enemies didnt level the place down. (i dont think you can lose the materials, you can pick them back from the ground... just have to rebuild)

Also there are some "automated" defenses you can create yourself which keep the base safe from smaller threats. Its always beneficial to build main base in the starting area... traveling wont be a problem later on and the first area is by far the safest.