If you mean to ban all XBox players, you can simply remove the crossplay option. If you meant specific players, I don't really know.

You can also look into using the "permitted" list, which will ONLY allow the players on that list to join. Or require a password.