I've got a large Lox farm going and haven't had any issues with mobs coming in and stirring them up. I have a nice big stone wall around the property. I've usually got about 20 roaming about at any one time, I regularly cull and feed to breed. Definitely put a little space between them and your main building though, just in case. :)