It seems relogging fixed this issue, going to leave this up so if anyone else has this issue of not removing an item just relog and it should fix it apparently