I have had both portals and beds go wonky on me. Destroying them then rebuilding them works the vast majority of times. First thing to do when something seems wonky is make sure you do not have any corrupted files. Right click on Valheim in your library, click on Properties, then Local Files, verify integrity of game files, then Validate the Valheim game files.