This discussion has been locked.

You can no longer post new replies to this discussion. If you have a question you can start a new discussion

IPSEC vpn with X.509 certs working only intermittently - any suggestion ?

*newbie disclaimer*: I'm new at this, please don't bite [:)]

I'm trying to establish a VPN between our HQ and one of our offices.
2 astaros 6.303; One static endpoint, one dynamic; everything configured according to the document "ASG-V6-x-Guide_ipsec-vpn_X509_net-to-net_en.pdf" found in the knowledge base. I made sure several times that the same policies are used, with the same lifetimes, etc etc.
Dead peer detection is set to ON
Nat-Traversal is set to ON, and used (on one end of the tunnel)

Everything works properly at first, but after a semi-random period of time, the tunnel will start malfunctioning for a while. It will then re-establish itself, sometime after 10 minutes, sometimes after 1 hour, then go down again, and so on.

Here an exemple of what I could find in the logs.

2006:10:17-05:08:44 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #152: STATE_MAIN_I3: sent MI3, expecting MR3
2006:10:17-05:09:11 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #152: next payload type of ISAKMP Hash Payload has an unknown value: 56
2006:10:17-05:09:11 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #152: malformed payload in packet
2006:10:17-05:09:11 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #152: sending notification PAYLOAD_MALFORMED to xxx.xxx.xxx.111:4500
2006:10:17-05:09:33 (none) pluto[22581]: pending Quick Mode with xxx.xxx.xxx.111 "S_Vienna_-_Brussels_INT_0" took too long -- replacing phase 1
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: initiating Main Mode to replace #152
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: received Vendor ID payload [Openswan (this version) 2.4.3  X.509-1.5.4 LDAP_V3 PLUTO_SENDS_VENDORID PLUTO_USES_KEYRR]
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: received Vendor ID payload [Dead Peer Detection]
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: received Vendor ID payload [RFC 3947] method set to=109
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: enabling possible NAT-traversal with method 3
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: transition from state STATE_MAIN_I1 to state STATE_MAIN_I2
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: STATE_MAIN_I2: sent MI2, expecting MR2
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: NAT-Traversal: Result using 3: i am NATed
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: I am sending my cert
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: transition from state STATE_MAIN_I2 to state STATE_MAIN_I3
2006:10:17-05:09:33 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: STATE_MAIN_I3: sent MI3, expecting MR3
2006:10:17-05:10:00 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: length of ISAKMP Hash Payload is larger than can fit
2006:10:17-05:10:00 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: malformed payload in packet
2006:10:17-05:10:00 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: sending notification PAYLOAD_MALFORMED to xxx.xxx.xxx.111:4500
2006:10:17-05:10:43 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0" #153: max number of retransmissions (2) reached STATE_MAIN_I3.  Possible authentication failure: no acceptable response to our first encrypted message

static endpoint :

2006:10:17-05:09:19 (none) pluto[7017]: packet from xxx.xxx.xxx.166:500: received Vendor ID payload [Openswan (this version) 2.4.3  X.509-1.5.4 LDAP_V3 PLUTO_SENDS_VENDORID PLUTO_USES_KEYRR]
2006:10:17-05:09:19 (none) pluto[7017]: packet from xxx.xxx.xxx.166:500: received Vendor ID payload [Dead Peer Detection]
2006:10:17-05:09:19 (none) pluto[7017]: packet from xxx.xxx.xxx.166:500: received Vendor ID payload [RFC 3947] method set to=109
2006:10:17-05:09:19 (none) pluto[7017]: packet from xxx.xxx.xxx.166:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-03] meth=108, but already using method 109
2006:10:17-05:09:19 (none) pluto[7017]: packet from xxx.xxx.xxx.166:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02] meth=107, but already using method 109
2006:10:17-05:09:19 (none) pluto[7017]: packet from xxx.xxx.xxx.166:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-00]
2006:10:17-05:09:19 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: responding to Main Mode from unknown peer xxx.xxx.xxx.166
2006:10:17-05:09:19 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: transition from state STATE_MAIN_R0 to state STATE_MAIN_R1
2006:10:17-05:09:19 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: STATE_MAIN_R1: sent MR1, expecting MI2
2006:10:17-05:09:20 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: NAT-Traversal: Result using 3: peer is NATed
2006:10:17-05:09:20 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: transition from state STATE_MAIN_R1 to state STATE_MAIN_R2
2006:10:17-05:09:20 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: STATE_MAIN_R2: sent MR2, expecting MI3
2006:10:17-05:09:20 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: Main mode peer ID is ID_DER_ASN1_DN: 'C=at, ST=Vienna, L=Vienna, O=ITUC, OU=IT, CN=ITUC Vienna Office, E=xx.xx@xx.org'
2006:10:17-05:09:20 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: no crl from issuer "C=be, ST=Brussels, L=Brussels, O=XX, OU=IT, CN=ITUC Certification Autority, E=xx.xx@xx.org" found (strict=no)
2006:10:17-05:09:20 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: I am sending my cert
2006:10:17-05:09:20 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: transition from state STATE_MAIN_R2 to state STATE_MAIN_R3
2006:10:17-05:09:20 (none) pluto[7017]: | NAT-T: new mapping xxx.xxx.xxx.166:500/4500)
2006:10:17-05:09:20 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: STATE_MAIN_R3: sent MR3, ISAKMP SA established {auth=OAKLEY_RSA_SIG cipher=aes_256 prf=oakley_md5 group=modp1536}
2006:10:17-05:09:20 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: Dead Peer Detection (RFC 3706): enabled
2006:10:17-05:09:32 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: retransmitting in response to duplicate packet; already STATE_MAIN_R3
2006:10:17-05:09:51 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: Informational Exchange message must be encrypted
2006:10:17-05:09:54 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56] xxx.xxx.xxx.166 #167: retransmitting in response to duplicate packet; already STATE_MAIN_R3

Am I doing anything wrong ? does anyone have any kind of tips for me ?

This thread was automatically locked due to age.

0 Krischeu over 19 years ago

thousands of errors could be ....
are you using x509 certificate with one signing ca and the other one as a verification ca?
Cancel
Vote Up 0 Vote Down

Cancel
0 Andreas Steffen over 19 years ago

Your problem is not the X.509 based authentication which is ok because
the peer is sending the MR3 response back:

2006:10:17-05:09:20 (none) pluto[7017]: "S_VPN_Vienna_INT_0"[56]
  xxx.xxx.xxx.166 #167: STATE_MAIN_R3: sent MR3, ISAKMP SA

Rather the MR3 packet gets mangled (probably due to NAT-T UDP encapsulation because the initiator cannot correctly decrypt the MR3
message:

2006:10:17-05:09:11 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0"
  #152: next payload type of ISAKMP Hash Payload has an unknown value: 56
2006:10:17-05:09:11 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0"
  #152: malformed payload in packet

and the second time around:

2006:10:17-05:10:00 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0"
#153: length of ISAKMP Hash Payload is larger than can fit
2006:10:17-05:10:00 (none) pluto[22581]: "S_Vienna_-_Brussels_INT_0"
#153: malformed payload in packet

These kind of payload mangling problems are usually quite tough to debug.
One approach would be to ramp up the debugging on both sides to

  plutodebug=all     in ipsec.conf

or the equivalent debug setting via the Astaro web interface. We could
then compare the MR3 raw payload on both the sender and receiver
ends and check if the packet gets mangled during transmission.

Regards

Andreas
Cancel
Vote Up 0 Vote Down

Cancel