This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

dynamic XG wan aggressive mode ipsec L2L vpn stops passing traffic with static Cisco ASA

A handful of Cisco routers and a handful of XG's deployed at branch offices all on static isp connections with main mode ipsec L2L vpn using their wan ip as the peer id linked to a couple central Cisco ASA5520's on static ip's all work fine.  Phase 1 + Phase 2 lifetime on these xg's are set to 86400.
    A new branch office with an XG on a dynamic isp connection using xg's built in dynamic dns service to tie into the asa ACL and xg vpn peer id with aggressive mode ipsec stops passing traffic over the vpn at predictable intervals.  Dynamic dns registration is working fine based on our 24/7 icmp monitoring of the xg wan interface address using the myfirewall.co hostname.  With phase 1 & phase 2 key lifetimes set to 86400 (24 hours) on the xg the pattern is traffic successfully passes across vpn for 18 hours, traffic stops passing for 3-4 hours, then the process repeats. With the phase 1 & phase 2 key lifetimes set to 28800 (8 hours) on the xg which is where I've left it right now the pattern is traffic successfully passes across vpn for 6 hours, traffic stops passing for ~2 hours, then the process repeats  In both cases it's stable for 75% of the key lifetime and the time to recover and start passing traffic again decreases with the decrease in key lifetime.  vpn connection status still shows green typically when i check into the gui and errors logged in the xg are "EST-P1: Malformed paylod in packet. Verify preshared secrets at both the ends" & "EST-P1: System did not accept any proposal received. Need to reconfigure the connection on either of the ends".  If I reboot the XG the problem resolves until the expected time interval.  Deactivating & reconnecting the XG vpn from the gui will get traffic passing normally again for most subnets, clearing isakmp & ipsec sa's at the ASA side gets traffic for any remaining subnets passing normally again.
    Varying key times on phase 1 vs phase 2 and disabling rekeying & disabling dpd on the xg have been experimented with, the result of disabling the rekeying on the xg was that the connection would no longer reestablish on its own after the initial disconnect.  Cisco mentions "The VPN tunnel gets disconnected after every 18 hours even though the lifetime is set for 24 hours" as a common problem  @ https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/81824-common-ipsec-trouble.html#vpndisc but it's not clear what the solution should be in XG to ASA terms.  Getting ipsec L2L working at all for this specific setup involved a lot of trial & error and having a dynamic ip in the XG equation will be a first so I'm open to suggestions if my approach is broken and hopefully others can benefit from the info.



ASA 5520 8.4(7)30:

ASA# sh ip
System IP Addresses:
Interface                Name                   IP address      Subnet mask     Method
GigabitEthernet0/0       outside                1.1.1.1    255.255.255.252 CONFIG

ASA# sh dns
Name: myoffice-mycity-myisp.myfirewall.co
  Address: 2.2.2.2                                TTL 00:01:59

ASA# sh cry isakmp sa detail
2   IKE Peer: 2.2.2.2
    Type    : L2L             Role    : responder
    Rekey   : no              State   : AM_ACTIVE
    Encrypt : aes-256         Hash    : SHA
    Auth    : preshared       Lifetime: 28800
    Lifetime Remaining: 12337

ASA# sh cry ipsec sa peer 2.2.2.2
peer address: 2.2.2.2
    Crypto map tag: spoke-dynamic, seq num: 99, local addr: 1.1.1.1

      access-list outside_cryptomap_99 extended permit ip X.16.0.0 255.255.0.0 X.21.5.0 255.255.255.0
      local ident (addr/mask/prot/port): (X.16.0.0/255.255.0.0/0/0)
      remote ident (addr/mask/prot/port): (X.21.5.0/255.255.255.0/0/0)
      current_peer: 2.2.2.2

      #pkts encaps: 6122, #pkts encrypt: 6122, #pkts digest: 6122
      #pkts decaps: 6854, #pkts decrypt: 6854, #pkts verify: 6854
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 6122, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: 1.1.1.1/0, remote crypto endpt.: 2.2.2.2/0
      path mtu 1500, ipsec overhead 74, media mtu 1500
      current outbound spi: 999C5B53
      current inbound spi : B43F91F1

    inbound esp sas:
      spi: 0xB43F91F1 (3024065009)
         transform: esp-aes-256 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 5, }
         slot: 0, conn_id: 53260288, crypto-map: spoke-dynamic
         sa timing: remaining key lifetime (sec): 12270
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0x999C5B53 (2577161043)
         transform: esp-aes-256 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 5, }
         slot: 0, conn_id: 53260288, crypto-map: spoke-dynamic
         sa timing: remaining key lifetime (sec): 12270
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

 

config:

object network myoffice-wan-ddns
 description mycompany myoffice mycity myisp dynamic ip - Sophos XG DDNS
 fqdn myoffice-mycity-myisp.myfirewall.co
 
object-group network trusted
 network-object object myoffice-wan-ddns

access-list ike-filter extended permit udp object-group trusted any eq isakmp
access-list ike-filter extended permit udp object-group trusted any eq 4500
access-list ike-filter extended deny udp any host 1.1.1.1 eq isakmp
access-list ike-filter extended deny udp any host 1.1.1.1 eq 4500
access-list ike-filter extended permit ip any any

crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac

crypto dynamic-map spoke-dynamic 99 match address outside_cryptomap_99
crypto dynamic-map spoke-dynamic 99 set pfs group5
crypto dynamic-map spoke-dynamic 99 set ikev1 transform-set ESP-AES-256-SHA
crypto dynamic-map spoke-dynamic 99 set reverse-route

crypto map outside_map 65535 ipsec-isakmp dynamic spoke-dynamic
crypto map outside_map interface outside

crypto ikev1 policy 10
 authentication pre-share
 encryption aes-256
 hash sha
 group 5
 lifetime 86400

tunnel-group myoffice-mycity-myisp type ipsec-l2l
tunnel-group myoffice-mycity-myisp ipsec-attributes
 ikev1 pre-shared-key *****


 

 

 

XG105w SFOS 16.05.7 MR-7:

 

 



This thread was automatically locked due to age.
  • Deployed,

    check the ipsec log from advanced shell:

     tail -f /log/ipsec.log

    and see if you find something useful, otherwise open a ticket with support.

    Thanks

  • this portion repeats in xg ipsec.log at all times every 30 seconds whether vpn is passing traffic or not:
    Sep 12 08:39:50 listening for IKE messages
    Sep 12 08:39:50 forgetting secrets
    Sep 12 08:39:51 loading secrets from "/cfs/ipsec.secrets"
    Sep 12 08:39:51   error in PKCS#1 private key
    Sep 12 08:39:51 "/cfs/ipsec.secrets" line 4: error loading RSA private key file
    Sep 12 08:39:51 forgetting secrets
    Sep 12 08:39:51 loading secrets from "/cfs/ipsec.secrets"
    Sep 12 08:39:51   error in PKCS#1 private key
    Sep 12 08:39:51 "/cfs/ipsec.secrets" line 4: error loading RSA private key file
    Sep 12 08:39:51 Changing to directory '/conf/certificate/cacerts'
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   file coded in unknown format, discarded
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51   unsupported public key algorithm
    Sep 12 08:39:51   error in X.509 certificate
    Sep 12 08:39:51 Changing to directory '/conf/certificate/aacerts'
    Sep 12 08:39:51 Changing to directory '/conf/certificate/ocspcerts'
    Sep 12 08:39:51 Changing to directory '/conf/certificate/crls'
    Sep 12 08:39:51   file coded in unknown format, discarded
    Sep 12 08:39:51   digest algorithm not supported
    Sep 12 08:39:51 crl issuer cacert not found for (file:///conf/certificate/crls/ClientAuthentication_CA.crl\321dw\002)


     

    logged in xg ipsec.log at a point when traffic started passing again on its own after a couple hours with phase 1 + phase 2 key lifetimes set to 28800 in xg profile:

    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: initiating Aggressive Mode #1235, connection "branchisp_datacenter_coloisp-2"
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: received Vendor ID payload [Cisco-Unity]
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: received Vendor ID payload [XAUTH]
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: received Vendor ID payload [Dead Peer Detection]
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: received Vendor ID payload [RFC 3947] method set to=110
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: ignoring Vendor ID payload [FRAGMENTATION c0000000]
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: ignoring Vendor ID payload [Cisco VPN 3000 Series]
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: protocol/port in Phase 1 ID Payload must be 0/0 or 17/500 but are 17/0
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: Aggressive mode peer ID is ID_IPV4_ADDR: '1.1.1.1'
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: NAT-Traversal: Result using RFC 3947 (NAT-Traversal): no NAT detected
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: Aggressive mode peer ID is ID_IPV4_ADDR: '1.1.1.1'
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: transition from state STATE_AGGR_I1 to state STATE_AGGR_I2
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: STATE_AGGR_I2: sent AI2, ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY cipher=aes_256 prf=oakley_sha group=modp1536}
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1235: Dead Peer Detection (RFC 3706): enabled
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-3" #1236: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP+AGGRESSIVE+failureDROP to replace #1229 {using isakmp#1235}
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1237: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP+AGGRESSIVE+failureDROP {using isakmp#1235}
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-3" #1236: Dead Peer Detection (RFC 3706): enabled
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-3" #1236: transition from state STATE_QUICK_I1 to state STATE_QUICK_I2
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-3" #1236: STATE_QUICK_I2: sent QI2, IPsec SA established {ESP=>0x5c403799 <0xd76121b7 xfrm=AES_256-HMAC_SHA1 NATD=none DPD=enabled}
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1237: Dead Peer Detection (RFC 3706): enabled
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1237: transition from state STATE_QUICK_I1 to state STATE_QUICK_I2
    Sep 12 02:01:23 "branchisp_datacenter_coloisp-2" #1237: STATE_QUICK_I2: sent QI2, IPsec SA established {ESP=>0xef16d4ee <0x9b28c56e xfrm=AES_256-HMAC_SHA1 NATD=none DPD=enabled}

     

     

    logged in xg ipsec.log at a point when traffic stopped passing after 6 hours with phase 1 + phase 2 key lifetimes set to 28800 in xg profile:

    Sep 12 08:01:54 "branchisp_datacenter_coloisp-2" #1235: received Delete SA payload: deleting ISAKMP State #1235
    Sep 12 08:01:54 packet from 1.1.1.1:500: received and ignored informational message
    Sep 12 08:01:55 "branchisp_datacenter_coloisp-3" #1236: DPD: Serious: could not find newest phase 1 state
    Sep 12 08:01:55 "branchisp_datacenter_coloisp-2" #1237: DPD: Serious: could not find newest phase 1 state
    Sep 12 08:01:55 "branchisp_datacenter_coloisp-1" #1238: DPD: Serious: could not find newest phase 1 state
    Sep 12 08:01:55 "branchisp_datacenter_coloisp-5" #1240: DPD: Serious: could not find newest phase 1 state
    Sep 12 08:01:55 "branchisp_datacenter_coloisp-4" #1239: DPD: Serious: could not find newest phase 1 state
    Sep 12 08:02:05 "branchisp_datacenter_coloisp-5" #1241: next payload type of ISAKMP Hash Payload has an unknown value: 216
    Sep 12 08:02:05 "branchisp_datacenter_coloisp-5" #1241: malformed payload in packet
    | payload malformed after IV
    |   91 24 3b 76  14 dd 7e 84  1b 84 46 2e  20 0e f2 f2
    |   e1 1d e5 91
    Sep 12 08:02:05 "branchisp_datacenter_coloisp-5" #1241: sending notification PAYLOAD_MALFORMED to 1.1.1.1:500
    Sep 12 08:02:07 "branchisp_datacenter_coloisp-5" #1241: next payload type of ISAKMP Hash Payload has an unknown value: 221
    Sep 12 08:02:07 "branchisp_datacenter_coloisp-5" #1241: malformed payload in packet
    | payload malformed after IV
    |   91 24 3b 76  14 dd 7e 84  1b 84 46 2e  20 0e f2 f2
    |   e1 1d e5 91
    Sep 12 08:02:07 "branchisp_datacenter_coloisp-5" #1241: sending notification PAYLOAD_MALFORMED to 1.1.1.1:500
    Sep 12 08:02:09 "branchisp_datacenter_coloisp-5" #1241: next payload type of ISAKMP Hash Payload has an unknown value: 161
    Sep 12 08:02:09 "branchisp_datacenter_coloisp-5" #1241: malformed payload in packet
    | payload malformed after IV
    |   91 24 3b 76  14 dd 7e 84  1b 84 46 2e  20 0e f2 f2
    |   e1 1d e5 91
    Sep 12 08:02:09 "branchisp_datacenter_coloisp-5" #1241: sending notification PAYLOAD_MALFORMED to 1.1.1.1:500
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: next payload type of ISAKMP Hash Payload has an unknown value: 43
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: malformed payload in packet
    | payload malformed after IV
    |   91 24 3b 76  14 dd 7e 84  1b 84 46 2e  20 0e f2 f2
    |   e1 1d e5 91
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: sending notification PAYLOAD_MALFORMED to 1.1.1.1:500
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: next payload type of ISAKMP Hash Payload has an unknown value: 29
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: malformed payload in packet
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: next payload type of ISAKMP Hash Payload has an unknown value: 110
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: malformed payload in packet
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: next payload type of ISAKMP Hash Payload has an unknown value: 117
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: malformed payload in packet
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: next payload type of ISAKMP Hash Payload has an unknown value: 96
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: malformed payload in packet
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: next payload type of ISAKMP Hash Payload has an unknown value: 57
    Sep 12 08:02:11 "branchisp_datacenter_coloisp-5" #1241: malformed payload in packet

  • Any ideas on this?  I have the same ipsec secrets file error, invalid x509 report in the log and an IPSec VPN that just won't connect.

     

    I had to rebuild and restore a backup after a disk failure and this started after the restore.

  • providing more info about your config would be recommended unless it's identical to my own.  haven't figured out a fix for the problem I posted yet other than rebooting the XG nightly from an Expect ssh cron job on a remote server but regarding the "/cfs/ipsec.secrets" line 4: error loading RSA private key file" and "error in X.509 certificate" entries specifically, I checked the logs on a different XG we have on a static ip + L2L main mode ipsec tunnel that has no problems and it's also logging those continuously so it might make sense to disregard them and focus on the configs at each end along with any other log entries.