This discussion has been locked.

You can no longer post new replies to this discussion. If you have a question you can start a new discussion

WAF becoming unresponsive due to possible vulnerability.

Hello,

I'm running a Sophos UTM 110 device with WAF enabled to protect some web sites. For the last couple of weeks, WAF would frequently become unresponsive.

Since I have premium support, I logged a request (#4064760). This post if for anyone seeing the same behaviour.

In the WAF live logs, the following comes up every time it becomes unresponsive.

2013:10:20-14:12:47 vpn reverseproxy: [Sun Oct 20 14:12:47.971059 2013] [core:notice] [pid 21131:tid 4148041408] AH00051: child pid 21614 exit signal Segmentation fault (11), possible coredump in /tmp
2013:10:20-14:36:01 vpn reverseproxy: [Sun Oct 20 14:36:01.641503 2013] [mpm_worker:notice] [pid 21131:tid 4148041408] AH00295: caught SIGTERM, shutting down

I believe at the route of this is an attack from an Amazon registered IP address (54.243.31.192):

2013:10:20-14:12:41 vpn reverseproxy: [Sun Oct 20 14:12:41.342311 2013] [avscan:error] [pid 21614:tid 3958836080] [client 54.243.31.192:47519] [21614] not all the file sent to the client: 95532 != 90112\n
2013:10:20-14:12:41 vpn reverseproxy: srcip="54.243.31.192" localip="66.209.67.107" size="90112" user="-" host="54.243.31.192" method="GET" statuscode="200" reason="-" extra="-" time="721687" url="/" server="wernerstrydom.com" referer="-" cookie="-" set-cookie="***; expires=Sun, 20-Oct-2013 21:47:25 GMT; path=/, HASH_***; expires=Sun, 20-Oct-2013 21:47:25 GMT; path=/"

The cookies has been obfuscated for security reasons. My website, wernerstrydom.com, is a Wordpress site, and the aim is for WAF to protect it against an attack, since Wordpress isn't the most secure software.

The only way I can address this is by restarting WAF. Once restarted, entries similiar to the following keep appearing every 20 minutes.

[avscan:error] [pid 21614:tid 3958836080] [client 54.243.31.192:47519] [21614] not all the file sent to the client: 95532 != 90112\n

Then at some point, the WAF crashes and all access to websites are denied.

My gut tells me that the client, 54.243.31.192, is sending incomplete files to the server, which causes the WAF to crash and in essence exposes a vulnerability. My expectation is that the WAF will detect this behaviour and after a day or so, black list the IP address and have the firewall block all requests from the IP addresses. However, it doesn't seem like WAF detects this as an attack.

Are my assumptions correct?

In the past, I simply dropped all packets that originates from Amazon EC2. However, that lists keeps on changing. Its tedious to keep the network definitions up to date and as the lists change, makes the device vulnerable to an attack. Am I the only one who thinks Amazon EC2 IP addresses should be blocked by default, except for ones I explicitly trust?

Sincerely,
Werner

This thread was automatically locked due to age.

Parents

0 bloudraak_01 over 11 years ago

BarryG,

I'm running an UTM 110, with relatively little traffic by other standards (about 1.5 GB of internet traffic a day). Connections wise, the server has usage has spike since last week.

I spend a couple of hours creating network definitions for EC2 public IP addresses and then configured the firewall to block all traffic from EC2 and most can be attributed to traffic from EC2. My sites are for humans, so there is no reason for constant EC2 traffic.

Screenshot 2013-10-23 18.55.29.png

But guess what; the following is appearing my the WAF logs:

2013:10:23-18:34:55 vpn reverseproxy: srcip="54.243.31.224" localip="66.209.67.107" size="112050" user="-" host="54.243.31.224" method="GET" statuscode="200" reason="-" extra="-" time="748351" url="/" server="wernerstrydom.com" referer="-" cookie="-" set-cookie="***X; expires=Thu, 24-Oct-2013 02:09:03 GMT; path=/, HASH_wfvt_***X; expires=Thu, 24-Oct-2013 02:09:03 GMT; path=/"
2013:10:23-18:34:55 vpn reverseproxy: srcip="54.241.32.96" localip="66.209.67.107" size="112050" user="-" host="54.241.32.96" method="GET" statuscode="200" reason="-" extra="-" time="702585" url="/" server="wernerstrydom.com" referer="-" cookie="-" set-cookie="***X; expires=Thu, 24-Oct-2013 02:09:03 GMT; path=/, HASH_wfvt_***X; expires=Thu, 24-Oct-2013 02:09:03 GMT; path=/"

54.243.31.224 and 54.241.32.96 are (friendly) servers from Amazon, which essentially is keeping my servers busy. One web server was already rendered useless.
I have a network definition for 54.242.0.0/15

Screenshot 2013-10-23 18.42.15.png

Added it to a a network group:

Screenshot 2013-10-23 18.43.02.png

And then proceeded to block all incoming traffic from that network group.

Screenshot 2013-10-23 18.47.32.png

When I ran my own reverse proxy server with Snort, the UTM firewall use to block traffic. However, now that I paid for WAF and configured it to do the same thing, the firewall is not taking affect.

To say the least, I'm rather frustrated.

Sincerely,
Werner
Cancel
Vote Up 0 Vote Down

Cancel

Reply

0 bloudraak_01 over 11 years ago

BarryG,

I'm running an UTM 110, with relatively little traffic by other standards (about 1.5 GB of internet traffic a day). Connections wise, the server has usage has spike since last week.

I spend a couple of hours creating network definitions for EC2 public IP addresses and then configured the firewall to block all traffic from EC2 and most can be attributed to traffic from EC2. My sites are for humans, so there is no reason for constant EC2 traffic.

Screenshot 2013-10-23 18.55.29.png

But guess what; the following is appearing my the WAF logs:

2013:10:23-18:34:55 vpn reverseproxy: srcip="54.243.31.224" localip="66.209.67.107" size="112050" user="-" host="54.243.31.224" method="GET" statuscode="200" reason="-" extra="-" time="748351" url="/" server="wernerstrydom.com" referer="-" cookie="-" set-cookie="***X; expires=Thu, 24-Oct-2013 02:09:03 GMT; path=/, HASH_wfvt_***X; expires=Thu, 24-Oct-2013 02:09:03 GMT; path=/"
2013:10:23-18:34:55 vpn reverseproxy: srcip="54.241.32.96" localip="66.209.67.107" size="112050" user="-" host="54.241.32.96" method="GET" statuscode="200" reason="-" extra="-" time="702585" url="/" server="wernerstrydom.com" referer="-" cookie="-" set-cookie="***X; expires=Thu, 24-Oct-2013 02:09:03 GMT; path=/, HASH_wfvt_***X; expires=Thu, 24-Oct-2013 02:09:03 GMT; path=/"

54.243.31.224 and 54.241.32.96 are (friendly) servers from Amazon, which essentially is keeping my servers busy. One web server was already rendered useless.
I have a network definition for 54.242.0.0/15

Screenshot 2013-10-23 18.42.15.png

Added it to a a network group:

Screenshot 2013-10-23 18.43.02.png

And then proceeded to block all incoming traffic from that network group.

Screenshot 2013-10-23 18.47.32.png

When I ran my own reverse proxy server with Snort, the UTM firewall use to block traffic. However, now that I paid for WAF and configured it to do the same thing, the firewall is not taking affect.

To say the least, I'm rather frustrated.

Sincerely,
Werner
Cancel
Vote Up 0 Vote Down

Cancel

Children

No Data

WAF becoming unresponsive due to possible vulnerability.

Submitted a Tech Support Case lately from the Support Portal?