This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

S2S-VPN between Sophos UTM 9 and ISA Server 2006: unexpected payload type (ISAKMP_NEXT_HASH)

Hello IT professionals,

Situation: L2TP/IPSec VPN Site2Site-connection between ISA server 2006 (side A) and Sophos UTM 9 (side B)
Problem: VPN-tunnel works from my side (A): RDP-session, access to SMB-share on side B are working fine, but collegues on side B have difficulties with RDP-connection to side A: it hangs during message "Remotes session is secured". But when I connect via RDP to server on side B I can establish a RDP-session (inside this session) from server side B to server side A.

Sophos IPSec log says (nealy ervery hour):

2017:09:15-13:41:24 sophos pluto[8887]: "S_REF_IpsSitS2sdigicon3_0" #18: IKE message has the Commit Flag set but Pluto doesn't implement this feature; ignoring flag
2017:09:15-13:41:24 sophos pluto[8887]: "S_REF_IpsSitS2sdigicon3_0" #18: message ignored because it contains an unexpected payload type (ISAKMP_NEXT_HASH)
2017:09:15-13:41:24 sophos pluto[8887]: "S_REF_IpsSitS2sdigicon3_0" #18: sending encrypted notification INVALID_PAYLOAD_TYPE to bla.bla.bla.bla:500 (public IP address of remote site)

Side A:

ISA Server 2006 connected to a bridged Port of a FritzBox-router (ISA-Server has public IP address)

IPSec-configuration:

VPN-Network: Internal network of side B (192.168.50.0 - 192.168.50.255)
Connection:
remote tunnel endpoint: public IP address of side B
IP address of local VPN gateway: public IP address of side A
IPsec-settings:
Phase I encryption algorithm: 3des
Phase I authentication algorithm: sha1
Phase I SA lifetime: 28800
Phase I DH group: 2 (1024 bit)
Phase II encryption algorithm: 3des
Phase II authentication algorithm: sha1
Phase II SA lifetime: 3600
Phase II PFS DH Group 2 (1024 Bit)
Authentication: preshared key


Side B:

Sophos UTM 9 connected to a bridged DSL-modem, internet-connection via PPPoE-Sophos-interface (public IP address)

IPSec-Configuration:

Remote Gateway:
Type: Initiate Connection
Gateway: public IP-Address of side A
Authentication type: preshared key
VPN ID: IP address
VPN ID: (empty)
Remote networks: internal network side A (192.168.2.0/24)
Advanced: Support path MTU discovery (enabled)
Policy:
IKE encryption algorithm: 3des
IKE authentication algorithm: sha1
IKE SA lifetime: 28800
IKE DH group: Group 2: MODP 1024
IPSec encrytion algorithm: 3des
IPSec authentication algorithm: sha1
IPsec SA lifetime: 3600
IPsec PFS group: Group 2 MODP 1024
Local networks: local network of side B (192.168.50.0/24)

(Internal traffic side B (192.168.1.0/24) is mapped by NAT-rule to VPN-side A as 192.168.50.0/24 to avoid network-conflict between side A and side C)

What could be the reason for the message: unexpected payload type (ISAKMP_NEXT_HASH)?

Any help is welcome to optimize the VPN-connection. Thank you.


Peter



This thread was automatically locked due to age.
  • Hi, again, Peter,

    You write: "(Internal traffic side B (192.168.1.0/24) is mapped by NAT-rule to VPN-side A as 192.168.50.0/24 to avoid network-conflict between side A and side C)"

    Perhaps a simple, stick diagram with IPs at sites A, B and C would help us to visualize your issue.

    Have you seen  Hub and Spoke Site-to-Site VPNs? Is DPD enabled on all VPN endpoints? 

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA