armor reduces poison damage a LOT. Bonemass, as you could probably guess from having spent more than 2 seconds in the swamp, does poison damage.
It is probably not wise to try to do him in weak armor, but you can certainly do that if you are willing to bow-cheese him (you can bow-cheese anything in the game, but some enemy heal slowly and can't be capped with wood arrows out of a starting bow) or use some other goofy trick like firepits or whatever.

I recommend you go farm iron. You will need a lot of iron anyway: its needed for many home items and upgrades, its needed for support beams if you decide to make a tall home, and its needed for not only iron armor but the next tiers of armor as well.

boating long ways is a risk. Be sure to put some portals along the coast as you go.

anyway, bottom line is, you can do it, but its a lot easier to just wear the *upgraded* iron armor.