Technically, you can go there at any time. When is a good time? Never, really. Even with your best armor and weapons they can still deal deadly damage against you. It's wise to get good with the bow and kill them from afar. Watch out for Fulings too, they're even worse, especially if they're able to gang up on you.