Open IPv6 Issues / questions

Hi Ben, please see my answers inline below:

Ben said:

- will the fix for issue NUTM-7187 be included with 9.5?

 [BL]: The fix for NUTM-7187 is not included in this current UTM 9.5 beta version. We are actively working on the fix right now though, so as soon as we have a confirmed fix it will be included in a subsequent release.

- is there a fix in the works for IPv6 Connections where the WAN Port is supposed to use an address out of the delegated prefix? Currently users of such ISPs do not get any IPv6 address. (for esxample KPN netherlands)

[BL]: This should be supported today, unless the ISP is doing both stateless & stateful. Is that the case for you? If so, we are fixing that as part of NUTM-7187 as well.

- what about the ability to change/edit the UID for IPv6 Delegation Requests?

[BL]: Unfortunately this isn't part of this 9.5 release.

- what about long standing feature requests such as 6tunnel integration, lets encrypt - is that on the roadmap? Users, myself included had high hopes for 9.5 but this seems to be more than a maintance release.

[BL]: Lets Encrypt is on our current roadmap, but it's mainly planned as a WAF feature. As for 6tunnel integration, it's currently not planned for any specific release.

 

thank you in advance.

 

Hi Bobby,

Normally the ISPs router will then request /48 prefix and use a /64 from that prefix for the wan interface and a /64for the lan interface. So there are no other global ipv6 addresses than the ones from that /48.

On the Sophos UTM, in my case I will only receive a link local IPv6 address via PPPoE. Using a tcpdump I have verified the UTM is not sending out a prefix request after the PPPoE has been established. Is it waiting for a advertised IPv6 address for the WAN interface first before it will do this? Because in this case it will never get it... And thus a IPv6 prefix will never be requested.

If you want to have a look at my Sophos VM, or need some tcpdumps of the PPPoE setup let me know!

Rene

Hi,

as long as Prakash is ok with that ill provide you with the patch. I am running on 9.4, i think Rene is running on 9.5 beta. Since the patch only patches the ipv6 watchdog files (i think) you should be OK with 9.5 beta.

Hi SanderRutten,

You are free to get the rpm from Ben.

However, please note that this fix was only verified for 9.411-3.1. But, as Ben said, it should (I am 'pretty' sure it would) work if installed over 9.5 Beta too.

-Prakash

One way to find out ;-)

Hi Prakash,

Good and bad news...

The good news is, I am now getting an IPv6 prefix! YAAY

The bad news. IPv6 traffic is not working yet :( 
In the interfaces overview I am not seeing my link local address anymore on the pppoe wan interface. This was shown before.

Log still shows I get it, but ifconfig does not show this local LL address.  
2017:04:19-19:41:55 gateway pppd-pppoe[27608]: local LL address fe80::3567:15be:c320:d03e
2017:04:19-19:41:55 gateway pppd-pppoe[27608]: remote LL address fe80::2a31:52ff:fe59:9fa6

I am also missing an IPv6 default route.

-- UPDATE
After reverting to the previous version I see the ipv6 LL address again in the interfaces overview, but it does not show up in ifconfig.

Using both versions I am able to ping the remote LL
# ping6 -I ppp0 fe80::2a31:52ff:fe59:9fa6
PING fe80::2a31:52ff:fe59:9fa6(fe80::2a31:52ff:fe59:9fa6) from fe80::8445:1d69:66c2:b895 ppp0: 56 data bytes
64 bytes from fe80::2a31:52ff:fe59:9fa6: icmp_seq=1 ttl=64 time=1.14 ms
64 bytes from fe80::2a31:52ff:fe59:9fa6: icmp_seq=2 ttl=64 time=1.08 ms

-- UPDATE2

After setting a default route connectivity works! :D
# route add -A inet6 default gw fe80::2a31:52ff:fe59:9fa6 dev ppp0

I also noticed the "Internet IPv6" network object is not bound to any interface.

René

Hi René,

Thanks for spending time verifying the fix and for your inputs.

FYI... A few other modules underwent IPv6 related fixes and code restructuring for 9.5 Beta (not just the ipv6_watchdog).

The problems you are seeing could be because of certain missing interdependencies (assuming you installed the latest 9.411 ipv6_watchdog fix over 9.5Beta).

If this is not the case, I will need to take a look at the packet capture (just before and after the pppoe interface is enabled) and also the logs from /var/log  to understand the problem in more detail. Would it be possible for you to provide us the same (via Sophos support maybe)?

However, it could be about a week or so before I can actually work on a fix again (sorry about that..)

Regards,

Prakash

i will test the patch with the 9.5 beta soon (tm), will report back on that also.

edit: test with 9.5 Beta, looking good so far.

sent you the patch earlier ago, please let us know if it works for you, 

works here on 9.4 (current) and 9.5 beta

Hi Prakash,

When referring to previous version in my last post I mean the previous patched persion. That showed the LL address, so i think it should work correclty in 9.5.

Regarding the missing default route. How is this generated in the watchdog script? Is it using the address of the RAs? In this case we don't have those and the default should point to the remote link local received via pppoe.

Thank you for all the work so far! If you are able to fix the default route creation I think my IPv6 is fully working! :)

Let me know if you still need some captures.

René

A little bit lost, is probably the correct term :+ I had it working for a few minutes, but somehow I've lost IPv6 connectivity and can't get it back... Even after a factory reset.

My first attempt when I got it working: clean 9.5 beta, installed the patch, restarted the watchdog service, enabled IPv6.
Enabled IPv6 Default Gateway on my WAN interface, added a static IPv6 address to my LAN interface, and created a Prefix Advertisement for my LAN.
I didn't create a new default route myself, it was added automatically. Client received an IPv6 address in my /64 range, and I was able to go to ipv6-test.com. (I did notice that my clients IPv6 address was actually the LAN's interface address, not the clients privacy extension address ??)
When checking under Interfaces, the WAN interface showed DEFAULT GW <IPv4> | fe80::2a0:a50f:fc78:5530
For some reason it now only displays DEFAULT GW <IPv4> | :: 
Like it doesn't have a gateway anymore.

I forgot to write down some log/commands in my first attempt, so these are from my second attempt.
/var/log/ipv6.log
2017:04:19-23:15:15 router ipv6_watchdog[8749]: Installing default route via fe80::2a0:a50f:fc78:5530 for interface ppp0(ifidx 12)

Seems fine to me.

ip -6 route
default via fe80::2a0:a50f:fc78:5530 dev ppp0 proto ra metric 1024 expires 1578sec hoplimit 64

Seems also fine, the same LL address I got on my opnsense firewall.

Getting a little bit late now, so I will test again tomorrow.

I have not seen the "Installing default route" in my logs.

Attached are my PPPoE and IPv6 log files.

logfiles_20170420085819.zip

I have not seen the "Installing default route" in my logs.

Attached are my PPPoE and IPv6 log files.

logfiles_20170420085819.zip

Checked my logs as well, and they are quite the same.

So.. I decided to start over. No factory reset yet, just config-wise.
Removed all my IPv6 settings on my interfaces, deleted Prefix for my LAN, disabled IPv6.
Re-applied the patch (I know, shouldn't matter), restarted the watchdog service, and reconnected the PPPoE connection.

Gave my LAN a static address 2001:981:9D6E:1::1
Re-created the Prefix Advertisement for my LAN.
Enabled the "IPv6 Default GW" on my WAN. (2017:04:20-12:10:38 router ipv6_watchdog[10859]: Installing default route via fe80::2a0:a50f:fc78:5530 for interface ppp0(ifidx 19))
WAN interface shows: DEFAULT GW 194.109.5.175 | ::
After reconnecting the WAN: DEFAULT GW 194.109.5.175 | fe80::2a0:a50f:fc78:5530

No working connection :(
So, why not try a reboot, solves everything right? Appearantly it does :O After a reboot I could imediately ping and surf to IPv6 hosts.

I saw something new in the ipv6.log: Started dhclient6 -P
Let's hope it keeps working now!
This is also in René's logfile, but I didn't see it in mine before. Attached are my ipv6.log and pppoe.log. Everything before 12:15 is before the reboot, and after 12.15 is after the reboot.

There is one thing I already noticed, my clients are using my LAN's address 2001:981:9D6E:1::1 as outgoing address. Is this expected behavior? Security wise? Or should a client always use their temporary privacy extension address? 

Logs.zip

I tried rebooting, disabling IPv6 etc. but no matter what I try I am not seeing the following lines in my log:

2017:04:20-12:18:18 router ipv6_watchdog[4332]: Installing default route via fe80::2a0:a50f:fc78:5530 for interface ppp0(ifidx 10)
2017:04:20-12:18:18 router ipv6_watchdog[4332]: JSON "ppp0", "unknown", "up", { "gateway6": "fe80::2a0:a50f:fc78:5530", "network6":"1"}
2017:04:20-12:18:18 router ipv6_watchdog[4332]: RA flags changed for interface ppp0(ifidx 10): SENT,READY -> SENT,RCVD,READY

Somehow the creation of the default route does not work for me.

If you do a reconnect, will it keep working? 

You made me break my IPv6 :( :+
It was working untill I reconnected my PPPoE. A reboot didn't solve it this time, and right now deleting my IPv6 settings and re-applying them didn't solve it either. 
So.. eh, weird. Maybe more time this weekend.

i can only advice to record connects with tcpdump for later wireshark dissection, was able to find problems really fast through that. 

say your pppoe interface is eth1, 

# tcpdump -i eth1 pppoes and ip6 -v -w /home/login/eth1-pppoes-datestamp.pcap

i did that with all patches and before so i could see the changes.

edit: fixed a typo in the tcpdump, its "ip6" not "ipv6"

Somehow it also a relief that you can reproduce my problem ;)

If you just add the default route manually it should work for now:
# route add -A inet6 default gw <remote LL> dev ppp0

Regarding the issue that everything was using 2001:981:9D6E:1::1. Are you doing some NAT translation?

Hi All,

   My name is Duc Le. I am a new member of NSG.

   I've been working with Prakash on this issue for a bit of time before his vacation.

   Since I am a newbie on this issue, I just wonder if you (rklomp and SanderRutten)

   can do the followings (A):

   1) Running the system (UTM with Prakash patch) as vanilla as possible. This is to reduce noise.

       Only concentrate on IPv6 address problem. (this is only a reminder, please ignore if you already. Thanks)

   2) Capture tcpdump

   3) Capture UTM log files (ipv6 and pppoe) and other logs if you think they are needed

   4) Capture system log (dmesg)

   5) Capture these: ifconfig -a, route table info, ps -elaf

   6) Misc Info as you think needed

    I suspected there might be some sort of race condition as to sometimes you see "Installing default route"

    and sometimes not.

   That's all for now. Please let me know your comments, ideas or questions. Thanks.

Regards

Le

P.S. I prefer to be called Le. Thanks.

Hi Ben, rklomp & SanderRutten,

Thank you for your continued efforts to help us troubleshoot & test this issue.

Le is another member on our team, and he'll be taking over this issue from Prakash since Prakash went on holiday earlier this week. Prakash handed over the issue before he left, but please forgive us if Le asks for some redundant information as he gets up to speed on the issue.

If I followed the thread correctly, I think Ben's issues are completely resolved with Prakash's latest patch, but rklomp & SanderRutten still has the issue of missing default route?

Thanks again for all your time & effort in helping us improve the product!

Hi Bobby,

Thank you for taking the time to troubleshoot and help on this issue so quickly!
You are correct, for SanderRutten and me the only issue left is the missing default route.

Duc Le I am away this weekend, but will try to make some more captures next week. See my previous post for pppoe and ipv6 logs.

Thanks for the efforts!

Regards,

René

Hello Le,

I'm writing down every steps I did, just to make sure I didn't do something wrong. This will result in a long text, sorry :)

• Factory reset 9.5 beta (9.470-14)
• Performed the basic system setup
  • Changed LAN to 10.0.0.0/16, enabled DHCP
  • WAN setup, configured as DSL PPPoE
• Created 1 firewall rule: Any source using any protocol to any destination. Keep it simple for now :P
• Enabled SSH
  • Uploaded ep-ipv6-watchdog-9.40-4.gce64053.i686.rpm, and installed it (rpm -Uhv --force ep-ipv6-watchdog-9.40-4.gce64053.i686.rpm)
  • Restarted watchdog: /var/mdw/scripts/ipv6_watchdog restart
  • Reconnect PPPoE interface

Wanted to run the tcpdump before enabling IPv6, the command Ben posted gave a syntax error. Appearantly the filter option wasn't ipv6, but ip6.
tcpdump -i eth1 pppoes and ip6 -v -w /home/login/eth1-pppoes-datestamp.pcap

• Interfaces & Routing
  • IPv6: Enabled IPv6.
• Interfaces & Routing > Interfaces: Gave my LAN interface the address 2001:981:9D6E:1::1/64
• Interfaces & Routing > IPv6: Created a LAN IPv6 Prefix Advertisement.
• Interfaces & Routing > Interfaces: Enabled the option "IPv6 Default GW" for the WAN interface.
  Interface status now shows:
  External (WAN) on eth1 [80.100.129.64/32]
  MTU 1492 · DEFAULT GW 194.109.5.175 | ::

Seems that there is no IPv6 gateway, but the ipv6.log shows:
Installing default route via fe80::2a0:a50f:fc78:5530 for interface ppp0(ifidx 13)

router:/home/login # ip -6 route
2001:981:9d6e:1::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth1 proto kernel metric 256
fe80::/64 dev eth0 proto kernel metric 256
fe80::/10 dev ppp0 metric 1
fe80::/10 dev ppp0 proto kernel metric 256
default via fe80::2a0:a50f:fc78:5530 dev ppp0 proto ra metric 1024 expires 1727sec hoplimit 64

I'm not able to ping google.com from client or UTM over IPv6.

• Reconnected my PPPoE connection

The interface status now shows:
External (WAN) on eth1 [80.100.129.64/32]
MTU 1492 · DEFAULT GW 194.109.5.175 | fe80::2a0:a50f:fc78:5530

Although it now shows my linklocal address, I'm not able to ping. At this point I stopped tcpdump and rebooted the UTM.
Started tcpdump again (eth1-pppoes-datestamp2.pcap), after the reboot.

router:/home/login # ip -6 route
2001:981:9d6e:1::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth1 proto kernel metric 256
fe80::/10 dev ppp0 metric 1
fe80::/10 dev ppp0 proto kernel metric 256
default via fe80::2a0:a50f:fc78:5530 dev ppp0 proto ra metric 1024 expires 1559sec hoplimit 64

Still looks the same as before. But just like a previous attempt, since the reboot I can connect to external IPv6 hosts. (As mentioned before: My clients are using the LAN's IPv6 address 2001:981:9D6E:1::1 and not their own address)

Interface & Routing > IPv6: The connectivity shows:
Native over External (WAN)
Delegated Prefix: 2001:981:9d6e::/48

I don't believe I have seen this before. Of course I forgot to check earlier.

Time to break stuff. Started tcpdump again (eth1-pppoes-datestamp3.pcap)
Reconnected my PPPoE interface, and ip -6 route output is the same as above. But I cannot ping anymore.
Rebooted the UTM again, and IPv6 is still broken. And so far I have not been able to get IPv6 up and running again since I have no clue what triggers something to get it working :)

Went back to Interface & Routing > IPv6: The connectivity still shows: Delegated Prefix: 2001:981:9d6e::/48

Summary:
IPv6 assigned by ISP: 2001:981:9D6E::/48
LAN prefix: 2001:981:9D6E:1::/64
Remote linklocal address fe80::2a0:a50f:fc78:5530 (Genexis fiber modem)

Attached are the tcpdump files and logfiles (pppoe.log, ipv6.log, boot.log, kernel.log, system.log).

Logfiles:
eth1-pppoes-datestamp.pcap = Fresh enabled IPv6, but not working
eth1-pppoes-datestamp2.pcap = IPv6 Working, capture started after a reboot
eth1-pppoes-datestamp3.pcap = IPv6 stopped working after reconnecting PPPoE

ifconfig.txt: Output of ifconfig -a. Included it only once, there was no difference (Except packets) during working or broken IPv6.
ps.txt: Output of ps -elaf when IPv6 was working.
ps_broken_ipv6.txt: Output after reconnecting PPPoE.

1781.Logs.zip

Hi Bobby,

my problem is completly solved, thank you again for the swift responses of you and your team :) 

there are some improvement that could be made to ipv6, lots of great suggestions on the feature request section, as i do not know how much effort you can currently put in please allow me to just name a few

- Sophos RED Server IPv6 to IPv6

- "6tunnel" integration to translate incoming WAN IPv6 to IPv4 internally (For the WAF for example)

- NatPDv6 (translate IPv6 subnets to internal IPv6 subnets via NAT)

- adjust the licence count on IPv6 Adresses. Most OSs pull more than one single IPv6 and use up the 50 IPs addresses really fast, it makes it REALLY hard to test IPv6 or use on a day to day basis without having licence alerts all the time ;)

final big question: will the XG gain from all these improvements also? Last time i tried iPv6 was completly broken on the XG in this scenario.