This discussion has been locked.

You can no longer post new replies to this discussion. If you have a question you can start a new discussion

DNS Resolution Timeout

AstaroFan2009 over 12 years ago

Hello Fellow Astaro Fans!

We have an Astaro ASG320 appliance (v8.309) configured with dual ISP connections.

- Cox Cable Modem is on Eth2
- Verizon FIOS is on Eth4

The connections are both used throughout the day in a randomized load balancing model.  We have about 100 users on the internal network and when they are web browsing, they sometimes go out via Cox and sometimes go out via Verizon.

For about 1.5 years, this has worked great.  As of 2 weeks ago, we've been getting a lot of people who cannot browse the web at all.   Most web requests that seem to go out the FIOS pipe result in an Astaro error page that reads:

DNS resolution timeout

This does not make any sense because none of our user workstations utilize the DNS services of the Astaro.  Instead, they all point to our internal Windows Active Directory DNS which then forwards to Google (8.8.8.8) and Level 3 (4.2.2.1) for FQDN's that it cannot resolve. Again, this has been fine for 1.5 years so I know it is not our setup.

The only way I can seem to fix this issue is by disabling the FIOS connection on Eth4.  When I do that, the DNS resolution timeout goes away.  I have done full integrity testing on the FIOS connection by plugging a PC directly into it (with no firewall/router in between) and it is fine.  DNS requests to both Google and Level 3 come back fine (FIOS is not blocking DNS requests to these 3rd party servers) and the web browsing is fast (35 Mb/s up and down).

My suspiscion is something odd has started happening with Astaro's load balancing and I think the DNS is a red herring sort of error message.

Any thoughts?  We are currently on just one ISP connectio (Cox) which is OK, but I'd like to get back to full redundancy ASAP.  Sophos support suggested I send all DNS traffic requests via Cox, but that defeats my redundancy model (e.g. what happens if Cox goes down?  People won't be able to fulfill DNS requests to the Internet).

I'm looking for a solution or confirmation that this is a bug somewhere in the Astaro system that will be resolved soon.

Many thanks!

This thread was automatically locked due to age.

0 BarryG over 12 years ago

Hi, you could run tcpdump on the external interfaces to see which path the DNS requests are going out, and see if there are replies.

Also, take a look at the firewall and IPS logs, and the HTTP/Web Protection log if you're using it.

Are you using IPv6?

Barry
Cancel
Vote Up 0 Vote Down

Cancel
0 jeff.welling over 12 years ago

What are the DNS Forwarders that are in use in the Astaro? That DNS Resolution timeout came from the Astaro itself, you were trying to get to the website fubar.com so the Astaro's web proxy was trying to lookup fubar.com, and that's what failed.

And if that doesn't work----

What if you tried to go to a website by it's IP address? Does this work?
I'm wondering if it's just DNS, or if it's everything and DNS is just the first part of the sequence, and it's failing.

Do you have any SNAT rules that are trying to apply an IP address from the Cox connection to packets that are leaving via Fios?

There is a masquerade rule in place for this traffic right? As a sanity check?

Do you have any Policy Routes configured?
Cancel
Vote Up 0 Vote Down

Cancel
0 BAlfson over 12 years ago

Hi, fan, and welcome to the User BB!

You might be interested in DNS Best Practice.

Try what I call Rule #1 (enhanced):
Whenever something seems strange, always check the Intrusion Prevention,
Application Control and Firewall logs.

Cheers - Bob
Cancel
Vote Up 0 Vote Down

Cancel
0 AstaroFan2009 over 12 years ago

Sorry for the delay and thanks for the feedback.  We got on the phone with Astaro Support and they setup a multi-path rule so all DNS request go out the Cox pipe.  We are now monitoring it to see if the issue re-occurs.  My understanding is that multi-path rule is smart enough to route DNS through FIOS if Cox goes down (despite the rule forcing it through the Cox pipe).

In terms of logs, you can see the error in the Web Filtering Log.  Here's an example:

Line 7035: 2013:05:23-07:40:48 ASGFW httpproxy[6434]: id="0002" severity="info" sys="SecureWeb" sub="http" name="web request blocked" action="block" method="GET" srcip="192.168.90.107" dstip="" user="" statuscode="500" cached="0" profile="REF_DefaultHTTPProfile (Default Proxy)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="0" request="0xc238b060" url="www.aloharag.com/.../logo.gif" exceptions="" error="DNS resolution timeout" category="174" reputation="neutral" categoryname="Fashion/Beauty"

The 2 DNS forwarders are 8.8.8.8 and 4.2.2.1 in the Astaro itself

I did note that on some independent testing of our Verizon FIOS connection, DNS resolution works fine, but continuous ping to 8.8.8.8 results in 7% packet loss.  Not sure if they are doing something weird with that.

The overall config has been confirmed with Astaro Support.  Let's see if this DNS Multi-Path rule helps.  Stay tuned!
Cancel
Vote Up 0 Vote Down

Cancel
0 BarryG over 12 years ago

Hi,
Did support know why it suddenly stopped working well?

but continuous ping to 8.8.8.8 results in 7% packet loss

You may be triggering the ICMP flood protection; check the IPS logs and/or tune the flood protection settings.

Barry
Cancel
Vote Up 0 Vote Down

Cancel
0 AstaroFan2009 over 12 years ago

They suspected the FIOS connection is doing something odd with the DNS - hence the redirect of all DNS traffic through the Cox pipe.

The continuous ping testing I did to Google which results in 7% packet loss anytime I test it was durig my being directly connected to FIOS (no Astaro in between). Are you thinking that the cause of our random web surfing issues that give error "DNS Resolution Timeout" is caused by ICMP flood protection? Why would that be?
Cancel
Vote Up 0 Vote Down

Cancel
0 BarryG over 12 years ago

I thought the flood protection might be causing your ping packetloss, not the DNS problem.

Barry
Cancel
Vote Up 0 Vote Down

Cancel
0 AstaroFan2009 over 12 years ago

I see. No, the ping loss is an isolated FIOS behavior - very weird in and of itself since my Cox connection has no % packet loss on continuous pings.

Thanks for you input either way. On a positive note, ever since we put in the multi-path rule that pushes all DNS resolution out the Cox pipe, I have not seen any "DNS Resolution Timeout" errors in my Web Filtering log - it's been several days now. Still watching for it nonetheless.
Cancel
Vote Up 0 Vote Down

Cancel

0 BarryG over 12 years ago

FWIW, I'm on FiOS in California, and I've had an MTR traceroute running all day across 15 hops and about 50 miles and 3 ISPs (FiOS to AlterNet to ComCast to Peer1) and only lost 1 packet out of 2000.


|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                            192.168.11.1 -    0 | 2031 | 2031 |    0 |    1 |   78 |    0 |
|   L100.LSANCA-VFTTP-142.verizon-gni.net -    1 | 2031 | 2030 |    0 |    6 |  453 |   15 |
|  G0-3-2-0.LSANCA-LCR-22.verizon-gni.net -    0 | 2031 | 2031 |    0 |   12 |  172 |   16 |
|so-5-1-0-0.LAX01-BB-RTR2.verizon-gni.net -    0 | 2031 | 2031 |    0 |   12 |  157 |   46 |
|          0.so-1-3-0.XL4.LAX15.ALTER.NET -    1 | 2031 | 2030 |    0 |   10 |  109 |   16 |
|           0.xe-7-0-0.XL4.LAX1.ALTER.NET -    1 | 2031 | 2022 |    0 |   13 |   94 |   16 |
|          0.xe-11-2-0.GW2.LAX1.ALTER.NET -    1 | 2031 | 2029 |    0 |   11 |  141 |   31 |
|   customer.alter.net.customer.alter.net -   12 | 2030 | 1801 |    0 |   30 |  187 |    0 |
|te-2-1-0-1-cr01.losangeles.ca.ibone.comcast.net -    1 | 2030 | 2029 |    0 |   31 |  187 |   31 |
|pos-0-4-0-0-pe01.600wseventh.ca.ibone.comcast.net -    0 | 2030 | 2030 |    0 |   30 |  110 |   31 |
|                           173.167.58.18 -    1 | 2030 | 2028 |    0 |   29 |  125 |   32 |
|                   No response from host -  100 | 2030 |    0 |    0 |    0 |    0 |    0 |
|                la-600w-fe3-2b.peer1.net -    0 | 2030 | 2030 |    0 |   32 |  219 |   31 |
|                la-600w-fe3-2a.peer1.net -    0 | 2030 | 2030 |    0 |   32 |  172 |   31 |
|                        destination -    1 | 2030 | 2029 |    0 |   34 |  140 |   31 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR - 0.8. Copyleft @2000-2002 Vasile Laurentiu Stanimir  ( stanimir@cr.nivis.com )

Barry

0 AstaroFan2009 over 12 years ago

I hear ya! It is the weirdest thing. I had FIOS come in, test the ONT box, replace the ONT box, nothing changes it. I hook my laptop directly to the FIOS box, ping 8.8.8.8 for about 10 minutes and consistently get approx. 7% packet loss. Also get similar ICMP packet loss on continuous ping to some other hosts on the Internet (BUT STRANGELY, NOT ALL). For example, I'll ping 4.2.2.1 and get no loss. Verizon says they can't control anything beyond their perimeter and have no idea why that is happening. I gave up on the ping issue because it does not appear to affect the web surfing - still get fast Internet, 35 Mbs up and down. I think maybe Verizon is putting a lower priority on ICMP packets to certain routing paths on the Internet - sound plausible. I know we are way off topic now so I won't go further on this. Thanks for your input.
Cancel
Vote Up 0 Vote Down

Cancel