IPSEC site-to-site VPN is up, but no traffic passing

Hi Patrick,

Reconfigure the IPSec policy on both the end. Post the screenshot of the configuration and let us know if the traffic is forwarded through either end via IPSec. To monitor the packet communication for IPSec tunnel refer https://community.sophos.com/kb/en-US/115702 .

Thanks

Hi Sachin, 

Apologies for the late reply. I did not receive an email notification that a response had been posted so i didn't read your reply until today. 

An update of where this problem is at: 

I have a case open with your tech support team and it is ongoing now for some time.

I updated to latest firmware on both UTMs (9.405-5) and reconfigured the gateways and connections on both ends. We know my config works because it does connect and traffic does flow sometimes. And it actually can work consistently at times. It can work all day, wit h me disconnecting and re-connecting several times without issue. 

However it will still randomly show the original problem (tunnel comes up, but no traffic allowed through). I have tried to get your tech team to log in remotely and see it in action, but unfortunately they haven't had the chance as yet. 

So at this point: 

- we know config is correct as it does work sometimes
- problem still randomly happens (tunnel will come up but no traffic whatsoever will go through either end)
- When it does happen, there doesn't seem to be a way to get it to behave again (have tried reboots, disconnects, reconnects, etc). I just leave it for a few hours or until next day and the find it suddenly works again. 
- problem does not happen with SSL tunnels (they come up and allow traffic every single time I have tested)

If you have any other suggestions, please let me know. This problem is a weird one, and i suspect it will take a while to figure out, so any help will be much appreciated. 

regards, 

Patrick

Hi Sachin, 

Apologies for the late reply. I did not receive an email notification that a response had been posted so i didn't read your reply until today. 

An update of where this problem is at: 

I have a case open with your tech support team and it is ongoing now for some time.

I updated to latest firmware on both UTMs (9.405-5) and reconfigured the gateways and connections on both ends. We know my config works because it does connect and traffic does flow sometimes. And it actually can work consistently at times. It can work all day, wit h me disconnecting and re-connecting several times without issue. 

However it will still randomly show the original problem (tunnel comes up, but no traffic allowed through). I have tried to get your tech team to log in remotely and see it in action, but unfortunately they haven't had the chance as yet. 

So at this point: 

- we know config is correct as it does work sometimes
- problem still randomly happens (tunnel will come up but no traffic whatsoever will go through either end)
- When it does happen, there doesn't seem to be a way to get it to behave again (have tried reboots, disconnects, reconnects, etc). I just leave it for a few hours or until next day and the find it suddenly works again. 
- problem does not happen with SSL tunnels (they come up and allow traffic every single time I have tested)

If you have any other suggestions, please let me know. This problem is a weird one, and i suspect it will take a while to figure out, so any help will be much appreciated. 

regards, 

Patrick

Hi Patrick,

If you look into the ESPdump captures, as referred in my previous post you can verify whether the UTM is forwarding the packets and if the request is responded or not via both ends. If the communication is one-sided, you can verify the IPSec connection with a different ISP line on both ends. I am sure this can help.

Thanks

Hi Sachin, 

The ESPdump is helpful. It is getting more information: 

- when the problem presents itself, ESPdump on the HQ side sees traffic coming from the Remote side (ICMP tests done from a computer on the remote lan to a computer at the HQ end)

- however, ESPdump does not show any traffic leaving the HQ side to the remote side (either the replies from the above ping test, or a separate ping test from an computer in the HQ lan to a computer in the Remote lan)

So it looks like when the problem presents, the HQ utm is deciding not to send traffic down its tunnel. 

When it does work, i can see traffic both sides using ESPdump, so I'm assuming that puts the problem on the HQ utm end (SG 330) 

I checked to see if a route had been created and there is one in the route table (i.e. 10.25.0.0 Use Interface eth1 (Ext WAN int)) 

Any other suggestions on what i can test?

Hi Patrick,

Amazing, now check if anything is dropped via HQ UTM. Check #1 from this amazing guide by balfson.

Thanks

Hi Sachin, 

Thanks. I have checked: 

- Instrusion Prevention is Off on both ends

- Application Control is not being used

- I have wateched the HQ firewall logs and not noticed any dropped traffic destined for Remote office

Any other suggestions?

Please keep in mind that this exact same config will randomly decide to work. I could come in tomorrow morning and find it working. I can then turn it off/on several times and it will remain working.

I'm thinking at this point, we've managed to figure out that it is the HQ end and it is related to the IPSEC tunnel. It does not seem to be sending traffic down the tunnel (proven with the espdump). What else can i check to see why it would not be sending traffic down the tunnel? 

The most disappointing thing about this whole experience is how Sophos Support has been. We have "Premium" support on both our UTMs and it's been weeks since i first logged the issue. To date i've only received first level support and the time in between responses there could be days. Only for him to confirm "yes, something is odd here". That was a week ago. It was "escalated" but i am yet to hear from anyone. 

I have looked through other threads and I notice similar experiences. In particular, ones where something that was working before a firmware update is suddenly broken. That screams firmware bug and you would think Sophos would be on it straight away to assist. I am seeing a trend of things breaking after updates and then little assistance from Sophos to resolve - really shakes my confidence in the product and makes me wonder what we are getting paying for "premium" support. 

Hi Patrick, 

We regret the poor experience you had with Support. Can you please provide me the case# so I can take a look at the history and try to find the required solution?

Meanwhile, go to IPSec> Debug, and select the IKE Debugging flags. Verify the logs if any suspicious information is captured for disconnection over IPSec, you can also send me an instance of the debug logs when you face the disconnection.  I would like to look into the logs from both ends HO and BO. 

Thanks

Hi Sachin, 

The case no is #6417303. Yesterday I did manage to get a call from another tech (funnily enough not long after i posted here). He did make quite a few log captures of things he thought would be useful and has gone away to look them over. The first level tech that responded also took a number of log captures as well. So i would say you should be able to see quite a lot of captures in the case history. 

regards, 

Patrick

Hi Patrick,

The case is with the escalation team and the engineer can call you in 8am – 6pm AEST Mon-Fri. Please drop a mail about your availability so that he can reach you. I am following up with the case#.

Thanks

Hi Sachin, 

Yes, thanks. I am aware that it has been escalated and have given them my availability. 

Just to update you with the latest development (and i will try to continue updating here in case other people have similar issues): 

I've had to roll ahead with my project and put the remote utm into production. The site-to-site is connected with an SSL vpn which seems to be doing ok. We would much rather have it on IPSEC so i will continue with trying to get this issue resolved. With both units now in production sites, it will make it a lot harder to troubleshoot and give your engineers access but we could not wait as there is no predictable time frame for a solution. 

We do have a standby utm which i can use to recreate the issue with another branch site. I will most likely set that up and use that when Sophos engineers get in touch. I will give them all this same details so they know what has changed since the start of this case. 

Patrick, you commented that IPS was off.  Did you actually look at the logs recommended in my post that Sachin linked to?

The only thing that Premium UTM Support buys you is the ability to open a Support Ticket yourself and to be able to call Support to open a ticket.  "Premium Sophos Support" that offers the kind of response that you wanted is over US$10K per year.  Sophos doesn't make this clear and most uninformed Sophos resellers aren't aware of the expensive service.

None of my clients uses the ability to open their own support tickets - they all call/email me.  For no extra charge, I look at their box and then open a ticket with Sophos Support.  If I can immediately resolve the issue in the time it normally takes to open a ticket, I do so for no extra charge.  I estimate that this "costs" me less than 15 minutes per year per client.

Cheers - Bob