Once you realise there are different stances, you'll be fine and everything will start to click (literally).
Stances include: Holding nothing, steering a boat, holding a tool, or holding a weapon (and/or shield/torch combination).

For instance, when your hammer is out, you are exclusively in build mode. You cannot interact with doors or chests, nor can you place food on the cooking station or even sneak in this mode without sheathing the tool first, (or swapping it out for a weapon).
Note: I've found that the "Hide/Show Weapon" key used for sheathing/unsheathing your last tool or weapon combo is just as important as "Use" and "Inventory" keys in this game more than others as it's not just for show. (It also comes in very handy after you've been swimming in deep water where anything you were wielding is automatically sheathed).

The system is similar to the Dragon Quest Builders games which started life as a Playstation 4 title. You needed to be wearing a special pair of gloves to pick up or drop items, or a hammer to build and destroy etc..