This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Traffic not reaching internal network from MPLS after hardware replacement.

I've searched and read other posts about this, but nothing seems to be working for me. Hoping to get some more input and maybe point me in the right direction.

We're replacing the network hardware for a company that has 9 total locations. They are all currently connected via MPLS with the HQ location being the only site that has internet. All other sites feed out through the HQ site. They have Cisco 3800's are each location and we're replacing those with Sophos SG units. We've already successfully replaced the devices at the HQ location and everything is working correctly. We've tried replacing two separate branch locations and have run into the same issue.

The 3800's at each branch location have basic configurations. They have two active interfaces, one for the LAN and one for MPLS. The default route is setup to run through the MPLS interface at HQ (Sophos HQ's MPLS interface - 192.168.0.1).

When we swapped in the Sophos at a branch location today, the WAN Interface was designated as the MPLS interface (192.168.0.6/24), and the LAN was setup for the internal network (192.168.20.1/24). From inside the branch's network, we have no access to anything on the HQ Network and have no internet. When looking at the HQ firewall, we see traffic hitting the firewall and being allowed through, but it seems like traffic isn't returning.

From the HQ side, we can access the branch firewall on the MPLS IP and on the internal LAN IP, but can't access any device beyond the firewall. We can see traffic being dropped when the traffic is destined for the firewall specifically, but we don't see any drops, allows, or anything in the logs when we try and access something beyond the firewall. We verified all firewall rules on the branch firewall are open and allowed, and even set an any/any/any with logging enabled to try and see some entries, but nothing. We also tried it with and without masq rules, but had the same problem (our masq rule was set for internal LAN --> MPLS interface)

I contacted the ISP and had them check the MPLS routing and they said everything looked good. I feel like the issue is on their side and likely has something to do with the fact the hardware changed and something on their end isn't respecting that, but it could still be something with the branch Sophos. When we swap the Cisco back in, everything continues to work, so I don't feel like it's a configuration on the HQ firewall.. When I check arp tables on the HQ firewall, it has some entries for the branch sites internal LAN, but they are listed as "incomplete". After the cisco swap back in, the arp tables display the Cisco MAC correctly.

I've setup MPLS connections betore with Sophos and everything was pretty simple and worked, but we've spent the last two days on this and I'm not really sure what the issue is. Any ideas?



This thread was automatically locked due to age.
Parents
  • check (or post) the routing tables from HQ and branch firewall.

    try traceroute though and from HQ and branch Firewall.

    Are you able to ping HQ MPLS and LAN interface IP from branch and from firewall?


    Dirk

    Systema Gesellschaft für angewandte Datentechnik mbH  // Sophos Platinum Partner
    Sophos Solution Partner since 2003
    If a post solves your question, click the 'Verify Answer' link at this post.

  • Dirk,

    Routing tables at HQ are working properly since all communication works properly once the Cisco at branch is put back in. No changes required at HQ to make the communication work, just swapped back in the Cisco.

    At the branch, we didn't have any static routes to start since the gateway is the MPLS interface at HQ, but we tried adding those in with an interface route for 192.168.24.0/24 (HQ LAN) --> Eth2 interface on Branch (192.168.0.6). Still no change.

    I'll have to double check the pings from Branch firewall to HQ's interfaces. I believe we were able to. From the Branch internal LAN, we could not ping/access them initially. Once I crated a MASQ rule, I could the ping 192.168.0.1 (MPLS interface on HQ), but there was still no traffic flowing for internal LAN.

    We may have access again tonight to try troubleshooting, but we had to place the Cisco back in to keep the site online. I'll see about running a traceroute/ping tests from all locations.

  • Hi neighbor and welcome to the UTM Community!

    What do you learn from doing #1 in Rulz (last updated 2019-04-17) in the HQ and branch UTMs?

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Hey Bob,

    We did check the firewall logs and didn't see any hits on the branch Sophos's logs. I verified all other protections were disabled, but did not check any of their logs. Unfortunately I can't check them right now since we rolled back to the Cisco.

    On the HQ side, I just reviewed the application control log and the Intrusion Prevention log. There was no ATP log.

    The App Control log just has entries for "failed to parse DNS RR in answer.....". These logs are present before and after our swap.

    The intrusion prevention log has some entries, but nothing relating to any of the IPs I was working with at the branch location.

    The firewall log on the HQ side shows traffic coming from my Branch location, shows traffic headed towards the branch location. The Branch firewall log never shows the incoming traffic.

    I'm trying to get the Branch Sophos back online so I can at least look at the logs.

  • Just checked on the Branch Firewall, and there are no entries in the Intrusion Prevention, Application Control, or ATP logs.

  • When you switched to the UTM, did you "bounce" the ISP's modem or otherwise ask the ISP to clear the ARP table in their last-hop router?

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Bob,

    i did contact the ISP and asked them to look at it and specifically mentioned ARP. Two different technicians seemed really unconcerned about the ARP tables and said it wouldn't be a problem. I did ask one technician to reboot the device at the Branch location, but I think he only ended up bouncing the port, not actually rebooting. Subsequent techs couldn't figure out how to check uptime to confirm.

    One thing they did tell us is that we're more than welcome to just yank the power to reboot it. That's going to be one of my steps to perform when we try and do this cutover again. We may have to do the same thing on the HQ side, but I'm hoping not.

    Right now, I'm preparing to run tcpdump's on both firewalls and wiresharking on two PCs at each side to try and understand where traffic dies. I've confirmed routes again on both devices and everything else looks good.

    Specific question though - In a MPLS environment like this, would I need MASQ, or should that be disabled since it's essentially just a LAN extension?

  • Specific question though - In a MPLS environment like this, would I need MASQ, or should that be disabled since it's essentially just a LAN extension?

    Is there a diagram for that?  Even just a picture of a hand-drawn one with IPs and subnets noted would help.  That would help us to understand how the UTM "sees" the MPLS connection.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Bob,

    Below is a diagram with IP's detailed and also some notes on the firewall rules we have created. I don't believe MASQ rules are needed on the Branch side since we're not trying to NAT through that IP, but not 100%. I believe in previous MPLS setups we haven't had to do that.

  • No masquerading needed.  That looks perfect to me - done by a strong network engineer.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
Reply
  • No masquerading needed.  That looks perfect to me - done by a strong network engineer.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
Children
  • Just wanted to provide an update on this. After testing again we still weren't able to get communication going properly. What we ended up having to do is to change the route on both the HQ side and the branch side to being gateway routes instead of interface routes. On the HQ side, the route for 192.168.20.0/24 was directed to that Branch's interface IP (192.168.0.6). On the branch side, it was the reverse. The gateway route for 192.168.24.0/24 was set to 192.168.0.1.

    The default route on the Branch should have been enough to work after changing the route on HQ, but I don't believe it did. However, we may have just created that route to be redundant on the Branch side.

    It's interesting that with the Cisco's in place, the interface route on the HQ worked for the Branch location, but not with the Sophos.

    Thank you for your assistance on this.