more precisely,
single player starts the server when you begin playing and will kill the server when you stop playing: time will stop totally between sessions.
single player on a multi player server, you would 'start' the server and it would run whether you were in the game playing or not. I do not know how that behaves, though, with only one player it seems like all that would happen is time passing, maybe crops and such grow but unclear if monsters attack the homes / etc when no one around (I think not).

even if playing single on a multi player server, no one can log in without your password and possibly an IP address or something.

Or, in short, do not start the server at all in single player is probably the best bet unless you want to grow something offline or something.