You know I never really noticed it, but you're right. With banners especially, materials end up all over the place.

It would actually be nice if by default, when deconstructing something the materials just ended up in your inventory (if there is room) instead of going through the mechanic of dropping on the floor and then being automatically picked up like other items.