Could be several things. Some possible troubleshooting steps:
1 - The host should be watching resource utilization while running the server. Watch CPU, RAM, and Network most of all
2 - Have the client side players check their internet speeds.
3 - Have the client side players ping the IP while playing. 'ping /c server.ip.goes.here'

Make sure these steps are done while playing. Hope this helps. :)