We are the good guys. The Dverger are the bad guys because they aren't us, the good guys, therefore it's okay to kill them and take their stuff. Simple as.