Just checked the one in my first world... it didn't despawn. So i can confirm that they do not despawn in vanilla game.
I guess using mods broke something. It happened to me with raids. I had mods that modified raids... i removed it and started a new vanilla world. Had my first raid at day 76. Now i'm at day 180 with a total of 3 raids :)