Remember the game only provides partial controller support.
Use a mouse to click and drag an item out of your inventory and release.
This will drop it.