Two users including myself are the only one using VPN on my network, how safe is it to disable the user portal at all?
Once both of you have downloaded the config file and/or software you can disable it.
If you need to update certificate for example, and so regenerate a conf file you should enable it back, d/l file, disable after.