This discussion has been locked.

You can no longer post new replies to this discussion. If you have a question you can start a new discussion

XG 210 Multiple failure "Internal Server Error"

For the past week our XG 210 has been failing. To start with users report not being able to connect via VPN and when checking the web interface all I see is "Internal Server Error" message or the following:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /webconsole/webpages/login.jsp.

Reason: Error reading from remote server

Users who have been connected before the error do not seem to be affected, but when a user tries to connect after the error they are unable to. I have updated to the latest firmware version and restarted the tomcat and apache services through the SSH interface but this has not resolved the issue. When the device fails I am unable to restart the tomcat/apache services as I cannot gain access via SSH, this has to be done after the device is re-booted manually.

We are getting the error every few hours and this is affecting many staff who are still working remotely.

Any help/advise would be grateful.

This thread was automatically locked due to age.

Top Replies

FormerMember over 4 years ago in reply to Paul Moor +1 suggested

Hi Paul Moor , The command will show the core dump files, if you see any recent core dump files, I'd suggest you open a support case for in-depth investigation. However, try to turn off the firewall…

0 FormerMember over 4 years ago
Hi Paul Moor,

Thank you for reaching out, and welcome to the Sophos Community!

What is the current firmware version on your firewall? Would it be possible for you to check the core dumps and firewall acceleration by running the following commands?

ls -al /var/cores

system firewall-acceleration show >>> Run this command from console

Thanks,
Cancel
Vote Up 0 Vote Down

Cancel
0 Paul Moor over 4 years ago in reply to FormerMember

Hi H_Patel,

Firmware is the latest release: SFOS 18.0.5 MR-5-Build586

Firewall Acceleration is enabled. I am not sure what I should be looking at with the core dumps though, can you advise?
Cancel
Vote Up 0 Vote Down

Cancel
0 FormerMember over 4 years ago in reply to Paul Moor
Hi Paul Moor,

The command will show the core dump files, if you see any recent core dump files, I'd suggest you open a support case for in-depth investigation.

However, try to turn off the firewall acceleration temporary for testing and observe the issue.

You can turn it off with the following command:

system firewall-acceleration disable

Thanks,
Cancel
Vote Up +1 Vote Down

Cancel
0 Paul Moor over 4 years ago in reply to FormerMember

I have turned of firewall acceleration and I will monitor this. There is also a dump file dated today at 8:31 which is approximately the last time the device was re-booted. I have already opened a case file and am in contact with Sophos support as well.
Cancel
Vote Up 0 Vote Down

Cancel
0 Paul Moor over 4 years ago in reply to Paul Moor

The device failed again when firewall acceleration was turned off so this did not help. I have re-enabled it now. Could this be a hardware issue? Nothing changed on the network when the problem first arose so I am wondering if there is some kind of hardware failure within the device that is causing the problem.
Cancel
Vote Up 0 Vote Down

Cancel
0 Paul Moor over 4 years ago in reply to Paul Moor

I was unable to re-enable the firewall acceleration, I got no error message in the console but I was unable to log into the web interface after doing this. Users were still able to connect via VPN and internal users could still get external access but after a short while the VPN connections started to fail which resulted in another re-boot needed. This is now the 4th time the device has been re-booted in the space of 3 hours.

I am not keen on trying the reenable the firewall acceleration in case it fails again.
Cancel
Vote Up 0 Vote Down

Cancel

0 Paul Moor over 4 years ago in reply to Paul Moor

Latest failure produced this error when trying to access the web interface:

HTTP ERROR 404

Problem accessing /webconsole/webpages/pagenotfound.html. Reason:

    Not Found

Caused by:

javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NoClassDefFoundError: java/awt/Color
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.servlet.ServletException: java.lang.NoClassDefFoundError: java/awt/Color
	at org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:909)
	at org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:838)
	at org.apache.jsp.webpages.login_jsp._jspService(login_jsp.java:353)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:214)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at cyberoam.utilities.csrf.CSRFCheckFilter.doFilter(CSRFCheckFilter.java:114)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at cyberoam.sessionmanagement.SessionCheckHelper.filter(SessionCheckHelper.java:100)
	at cyberoam.sessionmanagement.SessionCheckFilter.doFilter(SessionCheckFilter.java:55)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	... 10 more
Caused by: java.lang.NoClassDefFoundError: java/awt/Color
	at cyberoam.corporate.utilities.CaptchaHelper.<clinit>(CaptchaHelper.java:44)
	at org.apache.jsp.webpages.login_jsp._jspService(login_jsp.java:220)
	... 37 more
Caused by: java.lang.ClassNotFoundException: java.awt.Color
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:565)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	... 39 more

Caused by:

javax.servlet.ServletException: java.lang.NoClassDefFoundError: java/awt/Color
	at org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:909)
	at org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:838)
	at org.apache.jsp.webpages.login_jsp._jspService(login_jsp.java:353)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:214)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at cyberoam.utilities.csrf.CSRFCheckFilter.doFilter(CSRFCheckFilter.java:114)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at cyberoam.sessionmanagement.SessionCheckHelper.filter(SessionCheckHelper.java:100)
	at cyberoam.sessionmanagement.SessionCheckFilter.doFilter(SessionCheckFilter.java:55)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: java/awt/Color
	at cyberoam.corporate.utilities.CaptchaHelper.<clinit>(CaptchaHelper.java:44)
	at org.apache.jsp.webpages.login_jsp._jspService(login_jsp.java:220)
	... 37 more
Caused by: java.lang.ClassNotFoundException: java.awt.Color
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:565)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	... 39 more

Caused by:

java.lang.NoClassDefFoundError: java/awt/Color
	at cyberoam.corporate.utilities.CaptchaHelper.<clinit>(CaptchaHelper.java:44)
	at org.apache.jsp.webpages.login_jsp._jspService(login_jsp.java:220)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:214)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at cyberoam.utilities.csrf.CSRFCheckFilter.doFilter(CSRFCheckFilter.java:114)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at cyberoam.sessionmanagement.SessionCheckHelper.filter(SessionCheckHelper.java:100)
	at cyberoam.sessionmanagement.SessionCheckFilter.doFilter(SessionCheckFilter.java:55)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: java.awt.Color
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:565)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	... 39 more

Caused by:

java.lang.ClassNotFoundException: java.awt.Color
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:565)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at cyberoam.corporate.utilities.CaptchaHelper.<clinit>(CaptchaHelper.java:44)
	at org.apache.jsp.webpages.login_jsp._jspService(login_jsp.java:220)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:214)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at cyberoam.utilities.csrf.CSRFCheckFilter.doFilter(CSRFCheckFilter.java:114)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at cyberoam.sessionmanagement.SessionCheckHelper.filter(SessionCheckHelper.java:100)
	at cyberoam.sessionmanagement.SessionCheckFilter.doFilter(SessionCheckFilter.java:55)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)

0 FormerMember over 4 years ago in reply to Paul Moor
Hi Paul Moor,

Thanks for the update and the logs. I'd suggest you run the hard drive test on your firewall and post the result. I suspect the issue is related to the hard drive.

Sophos XG Firewall: Disk Test

Thanks,
Cancel
Vote Up 0 Vote Down

Cancel
0 Paul Moor over 4 years ago in reply to FormerMember

Hi H_Patel,

I have already performed a hard drive test and this returned no errors. We will be performing a re-image out of hours to see if this resolves the problem.
Cancel
Vote Up 0 Vote Down

Cancel