Ok, I think the culprit was the smoke.

I put my fires outside near my wall and put a roof above it. It seems the fire that have been turning off had roofs that were rather low (no snap placement).

Thank you all for the replies.