You don't have to build much, or build fancy, but you gotta build something. Rested bonuses, gear, supplies all require you to build. Without them, you will always be weak. With them, you are stronger than anything.