Two ways of getting rid of things:

1. Toss them into the Ocean. Seems simple, but unless it floats, you won't see it again. Do it a bit away from your base.

2. Later on, there is a structure you can build that you can place some things in, pull a lever (and then run!). Thor would happily turn it into charcoal for you. And everyone needs that!