IPSEC VPNs keep logging FIVE terminate/established log entries every hour or so

I have 1 HQ firewall (XG135) and 4 branches (XG105). They all have IPSEC tunnels with the default IKEv2 setup. They are all on 17.5.3.372

 

Spaced about 1 hour 13 minutes apart, I get 5 established and 5 terminated IPSEC vpn tunnel log entries from a branch firewall all with the exact same timestamp. Then a few minutes later, approximately the same hour/minutes apart, I get another 5 established and 5 terminated IPSEC vpn tunnel log entries. A few of the firewalls have TWO IPSEC tunnels, and I'll get an alert on one of the tunnels, and at some point over the next hour, I'll get another alert for the other tunnel, but both don't terminate at once.

 

No internet outages, and no perceived downtime with the tunnels. I have 100 users all using VoIP and no-one has said a word. This has been happening since deployment 2 days ago.

I'm getting bombarded with alerts for tunnel disconnection/re-connection.

What configuration item should I be looking at? Or is this a bug?

  • Hi  

    Thanks for reaching out!

    To start:

    • What firmware versions are all the firewalls on?
    • What IPsec policies are used?
    • Any relevant log outputs from your charon.log/strongswan.log during the time of disconnection/reconnection?

    Regards,

  • In reply to FloSupport:

    1. 17.5.3 MR3

    2. The Built in IKEv2 Policy

    3. Here's some snippets:

    strongswan.txt
    2019-04-04 08:11:43 12[ENC] <Company_HQ_Data-1|66> generating INFORMATIONAL request 155 [ ]
    2019-04-04 08:11:43 12[NET] <Company_HQ_Data-1|66> sending packet: from BRANCH_IP[500] to HEADQUARTERS_IP[500] (96 bytes)
    2019-04-04 08:11:43 31[NET] <Company_HQ_Data-1|66> received packet: from HEADQUARTERS_IP[500] to BRANCH_IP[500] (96 bytes)
    2019-04-04 08:11:43 31[ENC] <Company_HQ_Data-1|66> parsed INFORMATIONAL response 155 [ ]
    2019-04-04 08:11:59 22[IKE] <Company_HQ_Data-1|66> reauthenticating IKE_SA Company_HQ_Data-1[66]
    2019-04-04 08:11:59 22[IKE] <Company_HQ_Data-1|66> initiating IKE_SA Company_HQ_Data-1[67] to HEADQUARTERS_IP
    2019-04-04 08:11:59 22[ENC] <Company_HQ_Data-1|66> generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
    2019-04-04 08:11:59 22[NET] <Company_HQ_Data-1|66> sending packet: from BRANCH_IP[500] to HEADQUARTERS_IP[500] (1482 bytes)
    2019-04-04 08:11:59 24[NET] <Company_HQ_Data-1|67> received packet: from HEADQUARTERS_IP[500] to BRANCH_IP[500] (242 bytes)
    2019-04-04 08:11:59 24[ENC] <Company_HQ_Data-1|67> parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(MULT_AUTH) ]
    2019-04-04 08:12:00 24[IKE] <Company_HQ_Data-1|67> authentication of '172.16.5.2' (myself) with RSA_EMSA_PKCS1_SHA2_384 successful
    2019-04-04 08:12:00 24[IKE] <Company_HQ_Data-1|67> establishing CHILD_SA Company_HQ_Data-2
    2019-04-04 08:12:00 24[ENC] <Company_HQ_Data-1|67> generating IKE_AUTH request 1 [ IDi IDr AUTH SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
    2019-04-04 08:12:00 24[NET] <Company_HQ_Data-1|67> sending packet: from BRANCH_IP[500] to HEADQUARTERS_IP[500] (928 bytes)
    2019-04-04 08:12:00 09[NET] <Company_HQ_Data-1|67> received packet: from HEADQUARTERS_IP[500] to BRANCH_IP[500] (736 bytes)
    2019-04-04 08:12:00 09[ENC] <Company_HQ_Data-1|67> parsed IKE_AUTH response 1 [ IDr AUTH SA TSi TSr N(AUTH_LFT) ]
    2019-04-04 08:12:00 09[CFG] <Company_HQ_Data-1|67>   using trusted certificate "172.16.5.1"
    2019-04-04 08:12:00 09[IKE] <Company_HQ_Data-1|67> authentication of '172.16.5.1' with RSA_EMSA_PKCS1_SHA2_384 successful
    2019-04-04 08:12:00 09[IKE] <Company_HQ_Data-1|67> IKE_SA Company_HQ_Data-1[67] established between BRANCH_IP[172.16.5.2]...HEADQUARTERS_IP[172.16.5.1]
    2019-04-04 08:12:00 09[IKE] <Company_HQ_Data-1|67> scheduling reauthentication in 4801s
    2019-04-04 08:12:00 09[IKE] <Company_HQ_Data-1|67> maximum IKE_SA lifetime 5161s
    2019-04-04 08:12:00 09[IKE] <Company_HQ_Data-1|67> CHILD_SA Company_HQ_Data-2{448} established with SPIs c2f8d631_i c0b715c2_o and TS 192.168.5.0/24 === 192.168.201.0/24
    2019-04-04 08:12:00 09[APP] <Company_HQ_Data-1|67> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:00 09[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting) ref_count: 1 to 2 ++ up ++ (192.168.5.0/24#192.168.201.0/24)
    2019-04-04 08:12:00 09[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 6 to 7 ++ up ++ (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:00 09[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) UID: 67 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:00 09[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' up-client
    2019-04-04 08:12:00 06[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:00 06[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:00 06[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.201.0/24) already set up
    2019-04-04 08:12:00 09[IKE] <Company_HQ_Data-1|67> received AUTH_LIFETIME of 4685s, scheduling reauthentication in 4325s
    2019-04-04 08:12:00 09[IKE] <Company_HQ_Data-1|67> establishing CHILD_SA Company_HQ_Data-6
    2019-04-04 08:12:00 09[ENC] <Company_HQ_Data-1|67> generating CREATE_CHILD_SA request 2 [ SA No KE TSi TSr ]
    2019-04-04 08:12:00 09[NET] <Company_HQ_Data-1|67> sending packet: from BRANCH_IP[500] to HEADQUARTERS_IP[500] (1168 bytes)
    2019-04-04 08:12:00 29[NET] <Company_HQ_Data-1|67> received packet: from HEADQUARTERS_IP[500] to BRANCH_IP[500] (272 bytes)
    2019-04-04 08:12:00 29[ENC] <Company_HQ_Data-1|67> parsed CREATE_CHILD_SA response 2 [ SA No KE TSi TSr ]
    2019-04-04 08:12:00 29[IKE] <Company_HQ_Data-1|67> CHILD_SA Company_HQ_Data-6{449} established with SPIs c10c06b7_i c94ca234_o and TS 192.168.5.0/24 === 192.168.3.0/24
    2019-04-04 08:12:00 29[APP] <Company_HQ_Data-1|67> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:00 29[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting) ref_count: 1 to 2 ++ up ++ (192.168.5.0/24#192.168.3.0/24)
    2019-04-04 08:12:00 29[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 7 to 8 ++ up ++ (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:00 29[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) UID: 67 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:00 29[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' up-client
    2019-04-04 08:12:00 11[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:00 11[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:00 11[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:00 11[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:00 11[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.3.0/24) already set up
    2019-04-04 08:12:00 29[IKE] <Company_HQ_Data-1|67> establishing CHILD_SA Company_HQ_Data-5
    2019-04-04 08:12:00 29[ENC] <Company_HQ_Data-1|67> generating CREATE_CHILD_SA request 3 [ SA No KE TSi TSr ]
    2019-04-04 08:12:00 29[NET] <Company_HQ_Data-1|67> sending packet: from BRANCH_IP[500] to HEADQUARTERS_IP[500] (1168 bytes)
    2019-04-04 08:12:00 16[NET] <Company_HQ_Data-1|67> received packet: from HEADQUARTERS_IP[500] to BRANCH_IP[500] (272 bytes)
    2019-04-04 08:12:00 16[ENC] <Company_HQ_Data-1|67> parsed CREATE_CHILD_SA response 3 [ SA No KE TSi TSr ]
    2019-04-04 08:12:00 16[IKE] <Company_HQ_Data-1|67> CHILD_SA Company_HQ_Data-5{450} established with SPIs ca4d8d41_i c2af15cd_o and TS 192.168.5.0/24 === 192.168.9.0/24
    2019-04-04 08:12:00 16[APP] <Company_HQ_Data-1|67> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:00 16[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting) ref_count: 1 to 2 ++ up ++ (192.168.5.0/24#192.168.9.0/24)
    2019-04-04 08:12:00 16[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 8 to 9 ++ up ++ (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:00 16[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) UID: 67 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:00 16[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' up-client
    2019-04-04 08:12:00 14[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:00 14[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:00 14[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.9.0/24) already set up
    2019-04-04 08:12:00 16[IKE] <Company_HQ_Data-1|67> establishing CHILD_SA Company_HQ_Data-3
    2019-04-04 08:12:00 16[ENC] <Company_HQ_Data-1|67> generating CREATE_CHILD_SA request 4 [ SA No KE TSi TSr ]
    2019-04-04 08:12:00 16[NET] <Company_HQ_Data-1|67> sending packet: from BRANCH_IP[500] to HEADQUARTERS_IP[500] (1168 bytes)
    2019-04-04 08:12:00 07[NET] <Company_HQ_Data-1|67> received packet: from HEADQUARTERS_IP[500] to BRANCH_IP[500] (272 bytes)
    2019-04-04 08:12:00 07[ENC] <Company_HQ_Data-1|67> parsed CREATE_CHILD_SA response 4 [ SA No KE TSi TSr ]
    2019-04-04 08:12:00 07[IKE] <Company_HQ_Data-1|67> CHILD_SA Company_HQ_Data-3{451} established with SPIs ca7212bd_i c4e85288_o and TS 192.168.5.0/24 === 192.168.1.0/24
    2019-04-04 08:12:00 07[APP] <Company_HQ_Data-1|67> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:00 07[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting) ref_count: 1 to 2 ++ up ++ (192.168.5.0/24#192.168.1.0/24)
    2019-04-04 08:12:00 07[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 9 to 10 ++ up ++ (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:00 07[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) UID: 67 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:00 07[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' up-client
    2019-04-04 08:12:00 12[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:00 12[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:00 12[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.1.0/24) already set up
    2019-04-04 08:12:00 07[IKE] <Company_HQ_Data-1|67> establishing CHILD_SA Company_HQ_Data-1
    2019-04-04 08:12:00 07[ENC] <Company_HQ_Data-1|67> generating CREATE_CHILD_SA request 5 [ SA No KE TSi TSr ]
    2019-04-04 08:12:00 07[NET] <Company_HQ_Data-1|67> sending packet: from BRANCH_IP[500] to HEADQUARTERS_IP[500] (1168 bytes)
    2019-04-04 08:12:00 31[NET] <Company_HQ_Data-1|67> received packet: from HEADQUARTERS_IP[500] to BRANCH_IP[500] (272 bytes)
    2019-04-04 08:12:00 31[ENC] <Company_HQ_Data-1|67> parsed CREATE_CHILD_SA response 5 [ SA No KE TSi TSr ]
    2019-04-04 08:12:00 31[IKE] <Company_HQ_Data-1|67> CHILD_SA Company_HQ_Data-1{452} established with SPIs cf038334_i c8e0861d_o and TS 192.168.5.0/24 === 192.168.7.0/24
    2019-04-04 08:12:00 31[APP] <Company_HQ_Data-1|67> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:00 31[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting) ref_count: 1 to 2 ++ up ++ (192.168.5.0/24#192.168.7.0/24)
    2019-04-04 08:12:00 31[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 10 to 11 ++ up ++ (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:00 31[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) UID: 67 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:00 31[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' up-client
    2019-04-04 08:12:00 19[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:00 19[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:00 19[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.7.0/24) already set up
    2019-04-04 08:12:00 31[IKE] <Company_HQ_Data-1|67> establishing CHILD_SA Company_HQ_Data-4
    2019-04-04 08:12:00 31[ENC] <Company_HQ_Data-1|67> generating CREATE_CHILD_SA request 6 [ SA No KE TSi TSr ]
    2019-04-04 08:12:00 31[NET] <Company_HQ_Data-1|67> sending packet: from BRANCH_IP[500] to HEADQUARTERS_IP[500] (1168 bytes)
    2019-04-04 08:12:00 18[NET] <Company_HQ_Data-1|67> received packet: from HEADQUARTERS_IP[500] to BRANCH_IP[500] (272 bytes)
    2019-04-04 08:12:00 18[ENC] <Company_HQ_Data-1|67> parsed CREATE_CHILD_SA response 6 [ SA No KE TSi TSr ]
    2019-04-04 08:12:00 18[IKE] <Company_HQ_Data-1|67> CHILD_SA Company_HQ_Data-4{453} established with SPIs c5db6266_i cdffea92_o and TS 192.168.5.0/24 === 192.168.2.0/24
    2019-04-04 08:12:00 18[APP] <Company_HQ_Data-1|67> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:00 18[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting) ref_count: 1 to 2 ++ up ++ (192.168.5.0/24#192.168.2.0/24)
    2019-04-04 08:12:00 18[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 11 to 12 ++ up ++ (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:00 18[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) UID: 67 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:00 18[APP] <Company_HQ_Data-1|67> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' up-client
    2019-04-04 08:12:00 27[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:00 27[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:00 27[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.2.0/24) already set up
    2019-04-04 08:12:00 15[IKE] <Company_HQ_Data-1|66> deleting IKE_SA Company_HQ_Data-1[66] between BRANCH_IP[172.16.5.2]...HEADQUARTERS_IP[172.16.5.1]
    2019-04-04 08:12:00 15[IKE] <Company_HQ_Data-1|66> sending DELETE for IKE_SA Company_HQ_Data-1[66]
    2019-04-04 08:12:00 15[ENC] <Company_HQ_Data-1|66> generating INFORMATIONAL request 156 [ D ]
    2019-04-04 08:12:00 15[IKE] <Company_HQ_Data-1|66> sending DELETE for IKE_SA Company_HQ_Data-1[66]
    2019-04-04 08:12:00 15[ENC] <Company_HQ_Data-1|66> generating INFORMATIONAL request 156 [ D ]
    2019-04-04 08:12:00 15[NET] <Company_HQ_Data-1|66> sending packet: from BRANCH_IP[500] to HEADQUARTERS_IP[500] (96 bytes)
    2019-04-04 08:12:00 18[IKE] <Company_HQ_Data-1|67> verifying peer certificate
    2019-04-04 08:12:00 18[CFG] <Company_HQ_Data-1|67>   using trusted certificate "172.16.5.1"
    2019-04-04 08:12:00 18[IKE] <Company_HQ_Data-1|67> peer certificate successfully verified
    2019-04-04 08:12:00 23[NET] <Company_HQ_Data-1|66> received packet: from HEADQUARTERS_IP[500] to BRANCH_IP[500] (96 bytes)
    2019-04-04 08:12:00 23[ENC] <Company_HQ_Data-1|66> parsed INFORMATIONAL response 156 [ ]
    2019-04-04 08:12:00 23[IKE] <Company_HQ_Data-1|66> IKE_SA deleted
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting) ref_count: 2 to 1 -- down -- (192.168.5.0/24#192.168.201.0/24)
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 12 to 11 -- down -- (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) UID: 66 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' down-client
    2019-04-04 08:12:00 17[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:00 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:00 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.201.0/24) already set up
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting) ref_count: 2 to 1 -- down -- (192.168.5.0/24#192.168.3.0/24)
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 11 to 10 -- down -- (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) UID: 66 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' down-client
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting) ref_count: 2 to 1 -- down -- (192.168.5.0/24#192.168.9.0/24)
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 10 to 9 -- down -- (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) UID: 66 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:00 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' down-client
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting) ref_count: 2 to 1 -- down -- (192.168.5.0/24#192.168.1.0/24)
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 9 to 8 -- down -- (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) UID: 66 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' down-client
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting) ref_count: 2 to 1 -- down -- (192.168.5.0/24#192.168.7.0/24)
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 8 to 7 -- down -- (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) UID: 66 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' down-client
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [SSO] (sso_invoke_once) SSO is disabled.
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting) ref_count: 2 to 1 -- down -- (192.168.5.0/24#192.168.2.0/24)
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (ref_counting_remote) ref_count_remote: 7 to 6 -- down -- (BRANCH_IP#HEADQUARTERS_IP)
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) UID: 66 Net: Local BRANCH_IP Remote HEADQUARTERS_IP Connection: Company_HQ_Data Fullname: Company_HQ_Data-1
    2019-04-04 08:12:01 23[APP] <Company_HQ_Data-1|66> [COP-UPDOWN] (cop_updown_invoke_once) Tunnel: User '' Peer-IP '' my-IP '' down-client
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.3.0/24) already set up
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.9.0/24) already set up
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.1.0/24) already set up
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.7.0/24) already set up
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN][DB] (db_conn_info) hostname: 'Company_HQ_Data' result --> id: '1', mode: 'ntn', tunnel_type: '0', subnet_family:'0'
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec IKE for remotes (BRANCH_IP to HEADQUARTERS_IP) already set up
    2019-04-04 08:12:01 17[APP] [COP-UPDOWN] (do_cop_updown_invoke_once) !!SKIP!! IPsec SA for subnet (192.168.5.0/24 to 192.168.2.0/24) already set up
    2019-04-04 08:12:30 13[IKE] <Company_HQ_Data-1|67> sending DPD request
    2019-04-04 08:12:30 13[ENC] <Company_HQ_Data-1|67> generating INFORMATIONAL request 7 [ ]
    2019-04-04 08:12:30 13[NET] <Company_HQ_Data-1|67> sending packet: from BRANCH_IP[500] to HEADQUARTERS_IP[500] (96 bytes)
    2019-04-04 08:12:30 30[NET] <Company_HQ_Data-1|67> received packet: from HEADQUARTERS_IP[500] to BRANCH_IP[500] (96 bytes)
    2019-04-04 08:12:30 30[ENC] <Company_HQ_Data-1|67> parsed INFORMATIONAL response 7 [ ]
    

     

    Here's what my system log shows me:

    System.txt
    SYSTEM
    2019-04-04 08:12:01
    IPSec
    Terminated
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP terminated. (Remote: HQ_IP)
    17802
    SYSTEM
    2019-04-04 08:12:01
    IPSec
    Terminated
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP terminated. (Remote: HQ_IP)
    17802
    SYSTEM
    2019-04-04 08:12:01
    IPSec
    Terminated
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP terminated. (Remote: HQ_IP)
    17802
    SYSTEM
    2019-04-04 08:12:01
    IPSec
    Terminated
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP terminated. (Remote: HQ_IP)
    17802
    SYSTEM
    2019-04-04 08:12:01
    IPSec
    Terminated
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP terminated. (Remote: HQ_IP)
    17802
    SYSTEM
    2019-04-04 08:12:00
    IPSec
    Terminated
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP terminated. (Remote: HQ_IP)
    17802
    SYSTEM
    2019-04-04 08:12:00
    IPSec
    Established
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP established. (Remote: HQ_IP)
    17801
    SYSTEM
    2019-04-04 08:12:00
    IPSec
    Established
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP established. (Remote: HQ_IP)
    17801
    SYSTEM
    2019-04-04 08:12:00
    IPSec
    Established
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP established. (Remote: HQ_IP)
    17801
    SYSTEM
    2019-04-04 08:12:00
    IPSec
    Established
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP established. (Remote: HQ_IP)
    17801
    SYSTEM
    2019-04-04 08:12:00
    IPSec
    Established
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP established. (Remote: HQ_IP)
    17801
    SYSTEM
    2019-04-04 08:12:00
    IPSec
    Established
    Company_HQ_Data-1 - IPSec Connection Company_HQ_Data-1 between HQ_IP and BRANCH_IP established. (Remote: HQ_IP)
    17801

     

    And the Charon.log looks to be identical to my strongswan.log

  • In reply to LeetJN:

    Hi  

    Thank you for following up and sharing your logs.

    This behaviour could be related due to the default IKEv2 policies you are using, since both ends could be set to initiate the IPsec connection along with having re-initiate for their DPD peer unreachable action.

    Would it be possible to test by performing the following:

    • Duplicate the IKEv2 default policy you have been using on all of your XG devices.
    • Edit the cloned policy and IPsec connection for each site accordingly
      • For your HQ XG
        • Within the IPsec Connection configuration, the Gateway type is set to Respond only
        • For the duplicated IKEv2 IPsec policy, the Dead Peer Connection setting for When peer is unreachable is set to Disconnect
      • For your Branch XG's
        • Within the IPsec Connection configuration, the Gateway type is set to Initiate the connection
        • For the duplicated IKEv2 IPsec policy, the Dead Peer Connection setting for When peer is unreachable is set to Re-initiate

    Please keep me updated with your results.

    Regards,

  • In reply to FloSupport:

    HQ

    • Gateway was already set to respond only.
    • I duplicated the IKEv2 Policy and made sure the Dead Peer Connection setting for When peer is unreachable is set to Disconnect
    • I applied the duplicated IKEv2 to the IPSEC tunnel

    Branch

    • Gateway was already set to Initiate.
    • I duplicated the IKEv2 Policy and made sure the Dead Peer Connection setting for When peer is unreachable is set to Re-initiate
    • I applied the duplicated IKEv2 to the IPSEC tunnel.

    No change, every hour or so I get 5 disconnects and 5 reconnects with the same time stamp.

    Note: I strongly doubt this is related to any connection issues as well. The firewalls the Sophos XG replaced had IPSEC tunnels with the same dead peer settings for years and only went down when the internet was actually out at a location.

  • In reply to LeetJN:

    Hi  

    Thank you for testing and following up. At this point, I would advise to please raise a support case with us so that further investigation and troubleshooting can be performed with our support team.

    Please share your case number with me so that I can follow up accordingly.

    Regards,

  • In reply to FloSupport:

    8749017

     

    I was on the phone for a bit. The support agent enabled Support Access and escalated the issue.

  • In reply to LeetJN:

    Hi  

    Thank you for following up. I have left a note and will continue to monitor. Please don't hesitate to PM me directly if you had further questions or concerns.

    Cheers,

  • In reply to FloSupport:

    It was escalated and I've been going back and forth. Ultimately, the tech says those events are for rekeying.

     

    BUT, seems to be lost on the tech that I don't care what the events are for, they are flooding Sophos Central and my firewall logs with disconnect and reconnect events. THE EXACT SAME log entries for disconnect and reconnect events I actually care about. So when a tunnel goes down for real, the log entry is the same. I can't differentiate. I updated the ticket with a screenshot of what I'm taking about, I e-mailed my reps as well, and I haven't heard a thing from anyone.

     

    Bottom line, this is a bug and it has to be corrected. But for that to happen, Sophos needs to admit it's a bug and get it through the proper channels to be fixed. Either suppress the rekeying events in the system log, or change them to be something different so actual disconnection events are identifiable. After that is fixed, the Sophos Central issue will be fixed too.

  • In reply to LeetJN:

    Hi  

    Thank you for following up and for your ongoing participation during this process, I have been actively monitoring the activities on your support case.

    Your case has been escalated to our GES team to further investigate the behaviour you are experiencing. I'll continue to monitor, please don't hesitate to reach out to me directly if you had any questions or concerns in the meantime.

    Regards,

  • In reply to FloSupport:

    Just an update. This has been classified as a bug and I am eagerly awaiting the fix.

  • In reply to LeetJN:

    For context and tracking purposes of the community, the ID is (NR-1989).

  • In reply to FloSupport:

    Is there an ETA on the bug fix?

    I'm experiencing a very similar issue. 

  • In reply to LeeThomas:

    Hi  

    The issue is still under investigation by our team. Please stay tuned as I'll provide more updates when they become available.

    Thanks,

  • In reply to FloSupport:

    Hi Community,

    To follow up regarding  the fix for this (NR-1989) has since been released and has resolved the issue.

    Apologies for any inconveniences caused.

    Regards,

  • In reply to FloSupport:

    Hello,

    I think I have a similar problem between XG210 and Teltonika RUTX09 modems.

    Can you tell me how I can fix this problem? Sorry, I don't understand what mean "fix for this (NR-1989) has since been released and has resolved the issue". Do exist some a patch?


    Thanks.