VPN Issues

Hi Glen and welcome to the UTM Community!

In fact, I'm one of those that got burned clicking on an external link here about 10 years ago.  We can't know if that external site is properly protected.

Using Debug has never been necessary to solve a problem as long as I've been here, but it does make it difficult to see the big picture. ;-)

Please say which client you're using - L2TP/IPsec or IPsec.  Also, whether the client is outside your network and whether the UTM is behind a NAT.

1. Confirm that Debug is not enabled.
2. Disable the relevant Remote Access Profile.
3. Start the IPsec Live Log and wait for it to begin to populate.
4. Enable the relevant Remote Access Profile and attempt to connect.
5. Show us about 60 lines from enabling through the error.

 Cheers - Bob

L2TP/IPSEC. Router is direct DHCP connection to the fiber NTD.

1. correct debug is not enabled

2->5 (didn't get that many lines sorry but hopefully this will help)

Live Log: IPsec VPN     
Filter:     
        Autoscroll     
Reload
2018:08:15-16:24:55 router pluto[26422]: "L_for vpntest": deleting connection
2018:08:15-16:24:55 router pluto[26422]: shutting down interface lo/lo ::1
2018:08:15-16:24:55 router pluto[26422]: shutting down interface lo/lo 127.0.0.1
2018:08:15-16:24:55 router pluto[26422]: shutting down interface lo/lo 127.0.0.1
2018:08:15-16:24:55 router pluto[26422]: shutting down interface eth0/eth0 192.168.1.254
2018:08:15-16:24:55 router pluto[26422]: shutting down interface eth0/eth0 192.168.1.254
2018:08:15-16:24:55 router pluto[26422]: shutting down interface eth1/eth1 ***EXTERNAL_IP_ADDRESS***
2018:08:15-16:24:55 router pluto[26422]: shutting down interface eth1/eth1 ***EXTERNAL_IP_ADDRESS***
2018:08:15-16:24:55 router ipsec_starter[26415]: pluto stopped after 20 ms
2018:08:15-16:24:55 router ipsec_starter[26415]: ipsec starter stopped
2018:08:15-16:25:21 router openl2tpd[27149]: Start, trace_flags=00000000
2018:08:15-16:25:21 router openl2tpd[27149]: OpenL2TP V1.8, (c) Copyright 2004-2010 Katalix Systems Ltd.
2018:08:15-16:25:21 router openl2tpd[27149]: Loading plugin /usr/lib/openl2tp/ppp_unix.so, version V1.5
2018:08:15-16:25:21 router openl2tpd[27149]: Using config file: /etc/openl2tpd.conf
2018:08:15-16:25:23 router ipsec_starter[27189]: Starting strongSwan 4.4.1git20100610 IPsec [starter]...
2018:08:15-16:25:23 router pluto[27202]: Starting IKEv1 pluto daemon (strongSwan 4.4.1git20100610) THREADS VENDORID CISCO_QUIRKS
2018:08:15-16:25:23 router ipsec_starter[27195]: pluto (27202) started after 20 ms
2018:08:15-16:25:23 router pluto[27202]: loaded plugins: curl ldap aes des blowfish serpent twofish sha1 sha2 md5 random x509 pubkey pkcs1 pgp dnskey pem sqlite hmac gmp xauth attr attr-sql resolve
2018:08:15-16:25:23 router pluto[27202]: including NAT-Traversal patch (Version 0.6c)
2018:08:15-16:25:23 router pluto[27202]: Using Linux 2.6 IPsec interface code
2018:08:15-16:25:24 router pluto[27202]: loading ca certificates from '/etc/ipsec.d/cacerts'
2018:08:15-16:25:24 router pluto[27202]: loaded ca certificate from '/etc/ipsec.d/cacerts/VPN Signing CA.pem'
2018:08:15-16:25:24 router pluto[27202]: loading aa certificates from '/etc/ipsec.d/aacerts'
2018:08:15-16:25:24 router pluto[27202]: loading ocsp certificates from '/etc/ipsec.d/ocspcerts'
2018:08:15-16:25:24 router pluto[27202]: Changing to directory '/etc/ipsec.d/crls'
2018:08:15-16:25:24 router pluto[27202]: loading attribute certificates from '/etc/ipsec.d/acerts'
2018:08:15-16:25:24 router pluto[27202]: adding interface eth1/eth1 ***EXTERNAL_IP_ADDRESS***:500
2018:08:15-16:25:24 router pluto[27202]: adding interface eth1/eth1 ***EXTERNAL_IP_ADDRESS***:4500
2018:08:15-16:25:24 router pluto[27202]: adding interface eth0/eth0 192.168.1.254:500
2018:08:15-16:25:24 router pluto[27202]: adding interface eth0/eth0 192.168.1.254:4500
2018:08:15-16:25:24 router pluto[27202]: adding interface lo/lo 127.0.0.1:500
2018:08:15-16:25:24 router pluto[27202]: adding interface lo/lo 127.0.0.1:4500
2018:08:15-16:25:24 router pluto[27202]: adding interface lo/lo ::1:500
2018:08:15-16:25:24 router pluto[27202]: loading secrets from "/etc/ipsec.secrets"
2018:08:15-16:25:24 router pluto[27202]: loaded PSK secret for ***EXTERNAL_IP_ADDRESS*** %any
2018:08:15-16:25:24 router pluto[27202]: listening for IKE messages
2018:08:15-16:25:24 router pluto[27202]: added connection description "L_for vpntest"
2018:08:15-16:25:24 router pluto[27202]: added connection description "L_for vpntest"
2018:08:15-16:25:49 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: received Vendor ID payload [RFC 3947]
2018:08:15-16:25:49 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: ignoring Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02]
2018:08:15-16:25:49 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: ignoring Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02_n]
2018:08:15-16:25:49 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: ignoring Vendor ID payload [draft-ietf-ipsec-nat-t-ike-00]
2018:08:15-16:25:49 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: ignoring Vendor ID payload [FRAGMENTATION 80000000]
2018:08:15-16:25:49 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: received Vendor ID payload [Dead Peer Detection]
2018:08:15-16:25:49 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: responding to Main Mode from unknown peer ***CLIENT_IP_ADDRESS??***
2018:08:15-16:25:50 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: NAT-Traversal: Result using RFC 3947: peer is NATed
2018:08:15-16:25:50 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:25:50 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:25:50 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:25:52 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:25:52 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:25:52 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:25:56 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:25:56 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:25:56 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:25:59 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:25:59 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:25:59 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:26:01 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:26:01 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:26:01 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:26:04 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:26:04 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:26:04 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:26:08 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:26:08 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:26:08 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:26:10 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:26:10 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:26:10 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:26:13 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:26:13 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:26:13 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:26:17 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:26:17 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:26:17 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:26:21 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: ERROR: asynchronous network error report on eth1 for message to ***CLIENT_IP_ADDRESS??*** port 500, complainant ***CLIENT_IP_ADDRESS??***: Connection refused [errno 111, origin ICMP type 3 code 3 (not authenticated)]

OK noticed my last I'd used the wrong connection on the phone so next attempt....

2018:08:15-16:26:10 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:26:13 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:26:13 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:26:13 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:26:17 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: byte 2 of ISAKMP Identification Payload must be zero, but is not
2018:08:15-16:26:17 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)
2018:08:15-16:26:17 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: sending encrypted notification PAYLOAD_MALFORMED to ***CLIENT_IP_ADDRESS??***:500
2018:08:15-16:26:21 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: ERROR: asynchronous network error report on eth1 for message to ***CLIENT_IP_ADDRESS??*** port 500, complainant ***CLIENT_IP_ADDRESS??***: Connection refused [errno 111, origin ICMP type 3 code 3 (not authenticated)]
2018:08:15-16:27:00 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: max number of retransmissions (2) reached STATE_MAIN_R2
2018:08:15-16:27:00 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??***: deleting connection "L_for vpntest"[1] instance with peer ***CLIENT_IP_ADDRESS??*** {isakmp=#0/ipsec=#0}
2018:08:15-17:27:28 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: received Vendor ID payload [RFC 3947]
2018:08:15-17:27:28 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: ignoring Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02]
2018:08:15-17:27:28 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: ignoring Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02_n]
2018:08:15-17:27:28 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: ignoring Vendor ID payload [draft-ietf-ipsec-nat-t-ike-00]
2018:08:15-17:27:28 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: ignoring Vendor ID payload [FRAGMENTATION 80000000]
2018:08:15-17:27:28 router pluto[27202]: packet from ***CLIENT_IP_ADDRESS??***:500: received Vendor ID payload [Dead Peer Detection]
2018:08:15-17:27:28 router pluto[27202]: "L_for vpntest"[2] ***CLIENT_IP_ADDRESS??*** #2: responding to Main Mode from unknown peer ***CLIENT_IP_ADDRESS??***
2018:08:15-17:27:28 router pluto[27202]: "L_for vpntest"[2] ***CLIENT_IP_ADDRESS??*** #2: NAT-Traversal: Result using RFC 3947: peer is NATed
2018:08:15-17:27:28 router pluto[27202]: | NAT-T: new mapping ***CLIENT_IP_ADDRESS??***:500/4500)
2018:08:15-17:27:28 router pluto[27202]: "L_for vpntest"[2] ***CLIENT_IP_ADDRESS??***:4500 #2: Peer ID is ID_IPV4_ADDR: '10.174.20.147'
2018:08:15-17:27:28 router pluto[27202]: "L_for vpntest"[3] ***CLIENT_IP_ADDRESS??***:4500 #2: deleting connection "L_for vpntest"[2] instance with peer ***CLIENT_IP_ADDRESS??*** {isakmp=#0/ipsec=#0}
2018:08:15-17:27:28 router pluto[27202]: "L_for vpntest"[3] ***CLIENT_IP_ADDRESS??***:4500 #2: Dead Peer Detection (RFC 3706) enabled
2018:08:15-17:27:28 router pluto[27202]: "L_for vpntest"[3] ***CLIENT_IP_ADDRESS??***:4500 #2: sent MR3, ISAKMP SA established
2018:08:15-17:27:28 router pluto[27202]: "L_for vpntest"[3] ***CLIENT_IP_ADDRESS??***:4500 #2: ignoring informational payload, type IPSEC_INITIAL_CONTACT
2018:08:15-17:27:29 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??***:4500 #3: responding to Quick Mode
2018:08:15-17:27:29 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??***:4500 #3: IPsec SA established {ESP=>0x08d7302c <0xf9eda0e2 NATOA=0.0.0.0 DPD}
2018:08:15-17:28:26 router pluto[27202]: "L_for vpntest"[3] ***CLIENT_IP_ADDRESS??***:4500 #2: received Delete SA(0x08d7302c) payload: deleting IPSEC State #3
2018:08:15-17:28:26 router pluto[27202]: "L_for vpntest"[3] ***CLIENT_IP_ADDRESS??***:4500 #2: deleting connection "L_for vpntest"[1] instance with peer ***CLIENT_IP_ADDRESS??*** {isakmp=#0/ipsec=#0}
2018:08:15-17:28:26 router pluto[27202]: "L_for vpntest"[3] ***CLIENT_IP_ADDRESS??***:4500 #2: received Delete SA payload: deleting ISAKMP State #2
2018:08:15-17:28:26 router pluto[27202]: "L_for vpntest"[3] ***CLIENT_IP_ADDRESS??***:4500: deleting connection "L_for vpntest"[3] instance with peer ***CLIENT_IP_ADDRESS??*** {isakmp=#0/ipsec=#0}

2018:08:15-16:26:17 router pluto[27202]: "L_for vpntest"[1] ***CLIENT_IP_ADDRESS??*** #1: malformed payload in packet. Probable authentication failure (mismatch of preshared secrets?)

Are you certain that the UTM has a public IP on its External interface and that the ISP's router is not NATting to it?

2018:08:15-17:27:28 router pluto[27202]: "L_for vpntest"[2] ***CLIENT_IP_ADDRESS??***:4500 #2: Peer ID is ID_IPV4_ADDR: '10.174.20.147'

L2TP/IPsec won't work from inside the UTM - is the test being run from outside any of the LANs connected to the UTM?

Cheers - Bob

Before I changed the hardware (spin->SSD hdd upgrade etc) I had L2TP/IPSEC forwarding fine to the QNAP box and it was working well. I just want to remove that functionality from the QNAP box for security reasons amongst other things. So yeah fairly sure the ISP is not natting the iP. We used to run an SBS08 server here with nill issues routing SMTP, RPC/HTTPS, etc etc and part of the attraction of this ISP was the proper static IP.

Yes testing from outside since I figured it wouldn't work from internal nor was it meant to. I've been testing with my Galaxy S7E which would be one of the few devices that would regularly connect to obtain files.

I saw the secrets mismatch and went and changed all the user and IPSEC PSK passwords on both the router and the phone and still got this which has me scratching.

Before I changed the hardware (spin->SSD hdd upgrade etc) I had L2TP/IPSEC forwarding fine to the QNAP box and it was working well. I just want to remove that functionality from the QNAP box for security reasons amongst other things. So yeah fairly sure the ISP is not natting the iP. We used to run an SBS08 server here with nill issues routing SMTP, RPC/HTTPS, etc etc and part of the attraction of this ISP was the proper static IP.

Yes testing from outside since I figured it wouldn't work from internal nor was it meant to. I've been testing with my Galaxy S7E which would be one of the few devices that would regularly connect to obtain files.

I saw the secrets mismatch and went and changed all the user and IPSEC PSK passwords on both the router and the phone and still got this which has me scratching.