Go to a seed that you haven't killed any bosses on for your building and hanging out. Only boars and necks ever "raid" take your boss items with you!