Thread Info
  • State Verified Answer
  • +9 person also asked this people also asked this
  • Locked Locked
  • Replies 105 replies
  • Answers 1 answer
  • Subscribers 23 subscribers
  • Views 16728 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

WAF issues after updating to 9.709-3

FrancWest

Hi,

anyone else noticed that after updating to 9.709-3 Exchange Web Services is not working anymore? We get HTTP Error 500 when connecting to EWS published trhrough WAF. Also, the virtual server changes to orange when this error occurs. Accessing EWS through the browser shows the service page after authentication, but when interacting with EWS by using the Exchange Remote Connectivity Analyzer or EWS Editor generates the HTTP 500 error and the WAF rule turns orange.

When directly connecting to EWS and bypassing UTM works fine and we can interact with EWS.

Before the update everything worked fine.

Franc.



This thread was automatically locked due to age.

Top Replies

  • in reply to Jay Jay +4 verified

    It's related to the handling of the '100 Continue' message in the HTTP protocol. This message is sent by a Server after receiving the headers for a request to indicate that it is ready to receive the body of the request from the client - it allows a server to check the headers and potentially reject the request before the client unecessarily sends all the data. Normally, the client would send an additional "Expect:" header to indicate that it is going to wait for the server to send this message before it sends the request body.

    The recent update to Apache changed how this process is handled by the proxy, in a way that didn't work well with Exchange.

    The old behaviour was that the proxy would itself respond to the client with a '100 Continue' and would not wait for the server to do so. The new behaviour of Apache forwards the "Expect:" header to the server and waits for the server to respond with 100 Continue before passing the "100 Continue" to the client. This change defends against potential issues where a client could send a very large request, which the WAF proxy would have to buffer until the server is ready for it.

    The change should be fine for servers that respond to 'Expect' headers and use 100 Continue strictly according to the HTTP specification, but it seems that Exchange does not.

    Changing the configuration option as specified reverts this behaviour so that the proxy itself responds with a "100 Continue" message instead of waiting for the Server to do so. The risk of this behaviour is that the proxy has to buffer the entire body of the request before the server is ready to receive it. This shouldn't be a problem in most situations because request bodies are usually quite small.

    Jump to answer
  • 0 in reply to BAlfson

    Hoi,

    which suggestions are you referring to? Setting to monitor or even not applying a firewall profile at all didn’t help. Enabling sockets also doesn’t help. Everything worked fine for years, until installing 9.709-3.

  • 0 in reply to BAlfson

    Here's are the complete log entries for the issue:

    2022:03:02-13:07:00 firewall-1 httpd[29345]: [proxy_http:error] [pid 29345:tid 4084087664] [client <ip>:64185] AH01086: read less bytes of request body than expected (got 0, expected 634)
    2022:03:02-13:07:00 firewall-1 httpd[29345]: [proxy_http:error] [pid 29345:tid 4084087664] [client <ip>:64185] AH10154: pass request body failed to <ip>:443 (<ip>) from <ip> () with status 500
    2022:03:02-13:07:00 firewall-1 httpd: id="0299" srcip="<ip>" localip="<ip>" size="538" user="-" host="<ip>" method="POST" statuscode="500" reason="-" extra="-" exceptions="SkipURLHardening, SkipFormHardeningMissingToken" time="17848" url="/EWS/Exchange.asmx"

  • 0 in reply to FrancWest

    Hello Franc,

    I have left a note for the engineer about this and also reported to their manager to follow the process to arrange for the session properly, if you don't hear from the engineer by the end of your working day tomorrow, let me know! 

    Regards,

     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support Videos | Product Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.
  • 0 in reply to FrancWest

    Have you had any update on this as I am having the same issue.  This is having an impact on our Hybrid setup with Office365.  One major headache, we can't migrate users.  This is affecting  Autodiscover and EWS.  I don't have the firewall profile set, so hardening should not be taking place.  All was working before the upgrade.

    This is the error you get back using the Microsoft connectivity Analyzer 

    The Microsoft Connectivity Analyzer failed to obtain an Autodiscover XML response.
     
    Additional Details
    An HTTP 500 response was returned from Unknown.
    HTTP Response Headers: Connection: close
    Content-Length: 530
    Content-Type: text/html; charset=iso-8859-1
    Date: Sat, 05 Mar 2022 20:38:47 GMT
    Server: Apache
    The connection never makes it to the internal Exchange server.
  • 0 in reply to M Fuller

    No, I didn’t hear anything yet, but the support call goes through our supplier, but they didn’t inform me either that they received a response from Sophos support. Waiting for more than a week now to schedule a remote session.

    The error you get is the same as we are having. We took the UTM out and are using our Kemp load balancer now to do the proxying,

  • 0 in reply to emmosophos

    Is there any update on what Sophos Engineering have found on this?  When can we expect the fix?

  • 0 in reply to M Fuller

    Sophos Support still hasn't contacted us.

  • 0 in reply to emmosophos

    emmosophos can you pass this onto your engineering team.  when connecting to my Exchange 2019 server using  EWSEditor, (available from github) via UTM I am seeing the following error "DTD is prohibited in this XML document". If I connect without UTM it works.

  • 0

    Hi, we have the same issue after updating from 9.708-6 to 9.709-3 (ASG virtual appliance). However, it does not affect our EWS services (don't have hybrid Azure / Exchange infrastructure), but one of our applications:
    Customer calls the URL of the application we host, which forwards to the appropriate internal server via WAF. If certain requirements are not met, the application connects to an external third-party system and expects an XML as response. Since the update to 9.709-3, the application receives an HTML with status 500 instead of the required XML, resulting in an application error. Regardless of whether a firewall profile is active or not.
    (Also in virtual webserver the status for the real web server is "in error" (yellow), as mentioned by FrancWest. Other web applications / virtual webservers pointing to the same real web server have status "active" (green). By disabling and reenabling the virtual webserver, the status changes to green until the next connection to the URL.)

    httpd[55393]: [proxy_http:error] [pid 55393:tid 3776043888] [client xx.xx.xx.xx:44050] AH01086: read less bytes of request body than expected (got 0, expected 418)
httpd[55393]: [proxy_http:error] [pid 55393:tid 3776043888] [client xx.xx.xx.xx:44050] AH10154: pass request body failed to xxx.xxx.xxx.xxx:443 (xxx.xxx.xxx.xxx) from xx.xxx.xx.xxx () with status 500

    After restoring the sophos appliance from backup (virtual machine) before the update (back to 9.708-6), it works again.
    Possibly the similarity to the problems described here with EWS, since with ews XML is also returned?



  • 0 in reply to FrancWest

    Hello Franc,

    I have asked the engineer to send you an email with the link to the remote session and call you at the same time, in case the call doesn't complete. 

    Regards,

     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support Videos | Product Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.