Ok, my server has a server port of 23073. If you do not use that you can't join. But I've realized that the lobby gives random port numbers for my server. And this is causing me to get the slightest traffic in my server. Does anyone know the problem?

Im guessing the random port is in between 50000 and 51023, if that is the case than its not the lobby changing the port, its your modem (D-Link Modems do this). Nothing can be done about it, your modems NAT is whats doing it.