Guest User!

You are not Sophos Staff.

This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Open VPN - Too many open files

Hi,

We have UTM 9 (9.501-5) deployed in AWS for VPN purposes and it was working fine for about 1 year, but since a few weeks now we're having major problems.  Every day the VPN becomes unresponsive and in the openvpn.log files we see thousands of lines like this:

2017:09:21-01:00:43 vpn openvpn[6974]: TCP: accept(7) failed: Too many open files (errno=24)

Rebooting the machine clears the problem, but anywhere between 4 and 24 hours later it happens again.

A contact at Sophos suggested we update to the latest version which we did, but the problem remains.

Has anyone else had a similar problem or any idea of a solution?

Best regards

Paul



This thread was automatically locked due to age.
  • Hi, Paul, and welcome to the UTM Community!

    I don't know how to do it, but maybe restarting the OpenVPN service at the command line would be better/faster than a reboot.  Please let us know if there is such a workaround.

    I prefer IPsec using the "AES-128 PFS" Policy with X509 certificates, so I haven't developed too many tricks with OpenVPN for Site-to-Site.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Hello Paul,

    looks there is a bug in the software that opens files and does not close them properly. In linux you can increse the number of open files with ulimit -n. I did not try this with sophos but probably it works the same way. This would not resolve the issue but might give you more time between the reboot.

    9.501-5 is not the newes version. Just try the newest version or go back to the latest version where you did not have the issue.

    From my point of view openvpn on sophos is not rock solid an I alwas have some bad feelings when I update the devices. We already hat some issues with 2 Factor authentication when dialing in and the way openvpn handles the DNS cache is not always what you expect (clear when a new connection is setup).

    For a site to site connection I'd also use IPSEC. I did this with various diffent pieces of software and hardware on either side an it always worked quite stable.

    Best regards,
    Bernd

  • Hi Bernd,

    thanks for your reply. I think the same, that the OpenVPN process is not cleaning up files/ports after itself in a timely manner.

    We tried the open files fix already, adding lines to /etc/security/limits.conf to allow all users 65535 open files, but it made no difference.  Logging into the shell, ulimit -n shows 65535, but I'm not sure if the OpenVPN process gets these values, even though its running as root.  I've noticed that the OpenVPN process is chroot'ed, so I'm not sure if there are any other limits imposed on it.

    The version we're running was the latest that's available on AWS about 2 weeks ago, there are a few updates available since then but the release notes don't hint at some fix that will help us.  Previously we'd been running 9.4.something for months without any problem. This problem just started one day and had got steadily worse.

    We're trying to get some help via our reseller now, hopefully they can help/push sophos to do something.

    Any idea how many concurrent users this should support?  The license we have is unlimited, but there will obviously be some limits on user connections.

    Best regards

    Paul

  • Is this Remote Access instead of Site-to-Site, Paul?  You mention the number of users.  The default "VPN Pool (SSL)" works for 63 users or less.  You will need a /23 for 64 to 127 users, a /22 for 128 to 255 users, etc.  Was that what was causing your issue?

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Hi Bob,

    We're using it for Remote Access for PC's all around the world to link them to a central content server.  There might be several connections from the same 'site' but each PC is a separate connection.  There was some reasoning behind this, but it was from before I joined the project.

    This VPN Pool is setup with a /19, so >8000 possible IP's.  Until this issue occurred we regularly had >1000 active connections without any problem - Sophos and network both performing very well.

    The only recent 'change' would be that we added a lot of new users.  It's my thinking that the OpenVPN process is limited to 4096 open files/ports.  This is the default limit reported by ulimit -n.  With the last lot of new users, I think somehow we've broken that limit.  For example, if for each user it opens a temp and 2 ports, then 1365 concurrent users would be the max.  

    We tried changing this in the limits.conf file, but either this doesn't apply to the OpenVPN process or maybe it's something else.

    We just wait now for our reseller now, hopefully they can talk with Sophos directly and find a resolution.

    Best regards

    Paul

  • Paul, a /19 limits you to 2047 users.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA