They (only / mainly?) spawn at night. Set up some ballistae to target them and you'll get a trickle of black metal and gold when you're home.