I should have been more specific earlier, my mistake: Someone has to be on the server for time to pass. If time is passing in the world, crops will grow.

If everyone leaves the server the game... pauses. So no growth.