Open IPv6 Issues / questions

Hi Ben, please see my answers inline below:

Ben said:

- will the fix for issue NUTM-7187 be included with 9.5?

 [BL]: The fix for NUTM-7187 is not included in this current UTM 9.5 beta version. We are actively working on the fix right now though, so as soon as we have a confirmed fix it will be included in a subsequent release.

- is there a fix in the works for IPv6 Connections where the WAN Port is supposed to use an address out of the delegated prefix? Currently users of such ISPs do not get any IPv6 address. (for esxample KPN netherlands)

[BL]: This should be supported today, unless the ISP is doing both stateless & stateful. Is that the case for you? If so, we are fixing that as part of NUTM-7187 as well.

- what about the ability to change/edit the UID for IPv6 Delegation Requests?

[BL]: Unfortunately this isn't part of this 9.5 release.

- what about long standing feature requests such as 6tunnel integration, lets encrypt - is that on the roadmap? Users, myself included had high hopes for 9.5 but this seems to be more than a maintance release.

[BL]: Lets Encrypt is on our current roadmap, but it's mainly planned as a WAF feature. As for 6tunnel integration, it's currently not planned for any specific release.

 

thank you in advance.

 

Hi Bobby,

Normally the ISPs router will then request /48 prefix and use a /64 from that prefix for the wan interface and a /64for the lan interface. So there are no other global ipv6 addresses than the ones from that /48.

On the Sophos UTM, in my case I will only receive a link local IPv6 address via PPPoE. Using a tcpdump I have verified the UTM is not sending out a prefix request after the PPPoE has been established. Is it waiting for a advertised IPv6 address for the WAN interface first before it will do this? Because in this case it will never get it... And thus a IPv6 prefix will never be requested.

If you want to have a look at my Sophos VM, or need some tcpdumps of the PPPoE setup let me know!

Rene

sent you the patch earlier ago, please let us know if it works for you, 

works here on 9.4 (current) and 9.5 beta

Hi Prakash,

When referring to previous version in my last post I mean the previous patched persion. That showed the LL address, so i think it should work correclty in 9.5.

Regarding the missing default route. How is this generated in the watchdog script? Is it using the address of the RAs? In this case we don't have those and the default should point to the remote link local received via pppoe.

Thank you for all the work so far! If you are able to fix the default route creation I think my IPv6 is fully working! :)

Let me know if you still need some captures.

René

A little bit lost, is probably the correct term :+ I had it working for a few minutes, but somehow I've lost IPv6 connectivity and can't get it back... Even after a factory reset.

My first attempt when I got it working: clean 9.5 beta, installed the patch, restarted the watchdog service, enabled IPv6.
Enabled IPv6 Default Gateway on my WAN interface, added a static IPv6 address to my LAN interface, and created a Prefix Advertisement for my LAN.
I didn't create a new default route myself, it was added automatically. Client received an IPv6 address in my /64 range, and I was able to go to ipv6-test.com. (I did notice that my clients IPv6 address was actually the LAN's interface address, not the clients privacy extension address ??)
When checking under Interfaces, the WAN interface showed DEFAULT GW <IPv4> | fe80::2a0:a50f:fc78:5530
For some reason it now only displays DEFAULT GW <IPv4> | :: 
Like it doesn't have a gateway anymore.

I forgot to write down some log/commands in my first attempt, so these are from my second attempt.
/var/log/ipv6.log
2017:04:19-23:15:15 router ipv6_watchdog[8749]: Installing default route via fe80::2a0:a50f:fc78:5530 for interface ppp0(ifidx 12)

Seems fine to me.

ip -6 route
default via fe80::2a0:a50f:fc78:5530 dev ppp0 proto ra metric 1024 expires 1578sec hoplimit 64

Seems also fine, the same LL address I got on my opnsense firewall.

Getting a little bit late now, so I will test again tomorrow.

I have not seen the "Installing default route" in my logs.

Attached are my PPPoE and IPv6 log files.

logfiles_20170420085819.zip

Checked my logs as well, and they are quite the same.

So.. I decided to start over. No factory reset yet, just config-wise.
Removed all my IPv6 settings on my interfaces, deleted Prefix for my LAN, disabled IPv6.
Re-applied the patch (I know, shouldn't matter), restarted the watchdog service, and reconnected the PPPoE connection.

Gave my LAN a static address 2001:981:9D6E:1::1
Re-created the Prefix Advertisement for my LAN.
Enabled the "IPv6 Default GW" on my WAN. (2017:04:20-12:10:38 router ipv6_watchdog[10859]: Installing default route via fe80::2a0:a50f:fc78:5530 for interface ppp0(ifidx 19))
WAN interface shows: DEFAULT GW 194.109.5.175 | ::
After reconnecting the WAN: DEFAULT GW 194.109.5.175 | fe80::2a0:a50f:fc78:5530

No working connection :(
So, why not try a reboot, solves everything right? Appearantly it does :O After a reboot I could imediately ping and surf to IPv6 hosts.

I saw something new in the ipv6.log: Started dhclient6 -P
Let's hope it keeps working now!
This is also in René's logfile, but I didn't see it in mine before. Attached are my ipv6.log and pppoe.log. Everything before 12:15 is before the reboot, and after 12.15 is after the reboot.

There is one thing I already noticed, my clients are using my LAN's address 2001:981:9D6E:1::1 as outgoing address. Is this expected behavior? Security wise? Or should a client always use their temporary privacy extension address? 

Logs.zip

I tried rebooting, disabling IPv6 etc. but no matter what I try I am not seeing the following lines in my log:

2017:04:20-12:18:18 router ipv6_watchdog[4332]: Installing default route via fe80::2a0:a50f:fc78:5530 for interface ppp0(ifidx 10)
2017:04:20-12:18:18 router ipv6_watchdog[4332]: JSON "ppp0", "unknown", "up", { "gateway6": "fe80::2a0:a50f:fc78:5530", "network6":"1"}
2017:04:20-12:18:18 router ipv6_watchdog[4332]: RA flags changed for interface ppp0(ifidx 10): SENT,READY -> SENT,RCVD,READY

Somehow the creation of the default route does not work for me.

If you do a reconnect, will it keep working? 

You made me break my IPv6 :( :+
It was working untill I reconnected my PPPoE. A reboot didn't solve it this time, and right now deleting my IPv6 settings and re-applying them didn't solve it either. 
So.. eh, weird. Maybe more time this weekend.

i can only advice to record connects with tcpdump for later wireshark dissection, was able to find problems really fast through that. 

say your pppoe interface is eth1, 

# tcpdump -i eth1 pppoes and ip6 -v -w /home/login/eth1-pppoes-datestamp.pcap

i did that with all patches and before so i could see the changes.

edit: fixed a typo in the tcpdump, its "ip6" not "ipv6"

Somehow it also a relief that you can reproduce my problem ;)

If you just add the default route manually it should work for now:
# route add -A inet6 default gw <remote LL> dev ppp0

Regarding the issue that everything was using 2001:981:9D6E:1::1. Are you doing some NAT translation?

Hi All,

   My name is Duc Le. I am a new member of NSG.

   I've been working with Prakash on this issue for a bit of time before his vacation.

   Since I am a newbie on this issue, I just wonder if you (rklomp and SanderRutten)

   can do the followings (A):

   1) Running the system (UTM with Prakash patch) as vanilla as possible. This is to reduce noise.

       Only concentrate on IPv6 address problem. (this is only a reminder, please ignore if you already. Thanks)

   2) Capture tcpdump

   3) Capture UTM log files (ipv6 and pppoe) and other logs if you think they are needed

   4) Capture system log (dmesg)

   5) Capture these: ifconfig -a, route table info, ps -elaf

   6) Misc Info as you think needed

    I suspected there might be some sort of race condition as to sometimes you see "Installing default route"

    and sometimes not.

   That's all for now. Please let me know your comments, ideas or questions. Thanks.

Regards

Le

P.S. I prefer to be called Le. Thanks.

Hi All,

   My name is Duc Le. I am a new member of NSG.

   I've been working with Prakash on this issue for a bit of time before his vacation.

   Since I am a newbie on this issue, I just wonder if you (rklomp and SanderRutten)

   can do the followings (A):

   1) Running the system (UTM with Prakash patch) as vanilla as possible. This is to reduce noise.

       Only concentrate on IPv6 address problem. (this is only a reminder, please ignore if you already. Thanks)

   2) Capture tcpdump

   3) Capture UTM log files (ipv6 and pppoe) and other logs if you think they are needed

   4) Capture system log (dmesg)

   5) Capture these: ifconfig -a, route table info, ps -elaf

   6) Misc Info as you think needed

    I suspected there might be some sort of race condition as to sometimes you see "Installing default route"

    and sometimes not.

   That's all for now. Please let me know your comments, ideas or questions. Thanks.

Regards

Le

P.S. I prefer to be called Le. Thanks.

Hi Ben, rklomp & SanderRutten,

Thank you for your continued efforts to help us troubleshoot & test this issue.

Le is another member on our team, and he'll be taking over this issue from Prakash since Prakash went on holiday earlier this week. Prakash handed over the issue before he left, but please forgive us if Le asks for some redundant information as he gets up to speed on the issue.

If I followed the thread correctly, I think Ben's issues are completely resolved with Prakash's latest patch, but rklomp & SanderRutten still has the issue of missing default route?

Thanks again for all your time & effort in helping us improve the product!

Hi Bobby,

Thank you for taking the time to troubleshoot and help on this issue so quickly!
You are correct, for SanderRutten and me the only issue left is the missing default route.

Duc Le I am away this weekend, but will try to make some more captures next week. See my previous post for pppoe and ipv6 logs.

Thanks for the efforts!

Regards,

René

Hello Le,

I'm writing down every steps I did, just to make sure I didn't do something wrong. This will result in a long text, sorry :)

• Factory reset 9.5 beta (9.470-14)
• Performed the basic system setup
  • Changed LAN to 10.0.0.0/16, enabled DHCP
  • WAN setup, configured as DSL PPPoE
• Created 1 firewall rule: Any source using any protocol to any destination. Keep it simple for now :P
• Enabled SSH
  • Uploaded ep-ipv6-watchdog-9.40-4.gce64053.i686.rpm, and installed it (rpm -Uhv --force ep-ipv6-watchdog-9.40-4.gce64053.i686.rpm)
  • Restarted watchdog: /var/mdw/scripts/ipv6_watchdog restart
  • Reconnect PPPoE interface

Wanted to run the tcpdump before enabling IPv6, the command Ben posted gave a syntax error. Appearantly the filter option wasn't ipv6, but ip6.
tcpdump -i eth1 pppoes and ip6 -v -w /home/login/eth1-pppoes-datestamp.pcap

• Interfaces & Routing
  • IPv6: Enabled IPv6.
• Interfaces & Routing > Interfaces: Gave my LAN interface the address 2001:981:9D6E:1::1/64
• Interfaces & Routing > IPv6: Created a LAN IPv6 Prefix Advertisement.
• Interfaces & Routing > Interfaces: Enabled the option "IPv6 Default GW" for the WAN interface.
  Interface status now shows:
  External (WAN) on eth1 [80.100.129.64/32]
  MTU 1492 · DEFAULT GW 194.109.5.175 | ::

Seems that there is no IPv6 gateway, but the ipv6.log shows:
Installing default route via fe80::2a0:a50f:fc78:5530 for interface ppp0(ifidx 13)

router:/home/login # ip -6 route
2001:981:9d6e:1::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth1 proto kernel metric 256
fe80::/64 dev eth0 proto kernel metric 256
fe80::/10 dev ppp0 metric 1
fe80::/10 dev ppp0 proto kernel metric 256
default via fe80::2a0:a50f:fc78:5530 dev ppp0 proto ra metric 1024 expires 1727sec hoplimit 64

I'm not able to ping google.com from client or UTM over IPv6.

• Reconnected my PPPoE connection

The interface status now shows:
External (WAN) on eth1 [80.100.129.64/32]
MTU 1492 · DEFAULT GW 194.109.5.175 | fe80::2a0:a50f:fc78:5530

Although it now shows my linklocal address, I'm not able to ping. At this point I stopped tcpdump and rebooted the UTM.
Started tcpdump again (eth1-pppoes-datestamp2.pcap), after the reboot.

router:/home/login # ip -6 route
2001:981:9d6e:1::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth1 proto kernel metric 256
fe80::/10 dev ppp0 metric 1
fe80::/10 dev ppp0 proto kernel metric 256
default via fe80::2a0:a50f:fc78:5530 dev ppp0 proto ra metric 1024 expires 1559sec hoplimit 64

Still looks the same as before. But just like a previous attempt, since the reboot I can connect to external IPv6 hosts. (As mentioned before: My clients are using the LAN's IPv6 address 2001:981:9D6E:1::1 and not their own address)

Interface & Routing > IPv6: The connectivity shows:
Native over External (WAN)
Delegated Prefix: 2001:981:9d6e::/48

I don't believe I have seen this before. Of course I forgot to check earlier.

Time to break stuff. Started tcpdump again (eth1-pppoes-datestamp3.pcap)
Reconnected my PPPoE interface, and ip -6 route output is the same as above. But I cannot ping anymore.
Rebooted the UTM again, and IPv6 is still broken. And so far I have not been able to get IPv6 up and running again since I have no clue what triggers something to get it working :)

Went back to Interface & Routing > IPv6: The connectivity still shows: Delegated Prefix: 2001:981:9d6e::/48

Summary:
IPv6 assigned by ISP: 2001:981:9D6E::/48
LAN prefix: 2001:981:9D6E:1::/64
Remote linklocal address fe80::2a0:a50f:fc78:5530 (Genexis fiber modem)

Attached are the tcpdump files and logfiles (pppoe.log, ipv6.log, boot.log, kernel.log, system.log).

Logfiles:
eth1-pppoes-datestamp.pcap = Fresh enabled IPv6, but not working
eth1-pppoes-datestamp2.pcap = IPv6 Working, capture started after a reboot
eth1-pppoes-datestamp3.pcap = IPv6 stopped working after reconnecting PPPoE

ifconfig.txt: Output of ifconfig -a. Included it only once, there was no difference (Except packets) during working or broken IPv6.
ps.txt: Output of ps -elaf when IPv6 was working.
ps_broken_ipv6.txt: Output after reconnecting PPPoE.

1781.Logs.zip

Hi Bobby,

my problem is completly solved, thank you again for the swift responses of you and your team :) 

there are some improvement that could be made to ipv6, lots of great suggestions on the feature request section, as i do not know how much effort you can currently put in please allow me to just name a few

- Sophos RED Server IPv6 to IPv6

- "6tunnel" integration to translate incoming WAN IPv6 to IPv4 internally (For the WAF for example)

- NatPDv6 (translate IPv6 subnets to internal IPv6 subnets via NAT)

- adjust the licence count on IPv6 Adresses. Most OSs pull more than one single IPv6 and use up the 50 IPs addresses really fast, it makes it REALLY hard to test IPv6 or use on a day to day basis without having licence alerts all the time ;)

final big question: will the XG gain from all these improvements also? Last time i tried iPv6 was completly broken on the XG in this scenario.

I think the last line in your route table shows your default route:

default via fe80::2a0:a50f:fc78:5530 dev ppp0 proto ra metric 1024 expires 1559sec hoplimit 64

So it seems it should be working for you.

There is the magic word: Should :) It should indeed work. Except it isn't :( Or at least not always, it breaks very easily. 
If I get it working again I will keep my hands of the settings for a while, and see if it keeps working. Or maybe it breaks anyway and has the reconnect/reboot no influence at all.

@SanderRutten: Thanks so much for your help and log files. I am looking at it and will update ASAP. Thanks.

@Ben: Thanks

Quick summary of findings in the SanderRutten latest logs:

1) The default route over the ppp0/eth1 is there in the route table. So it is OK for now.

My view: As IPv6 allows multiple default gateways. This is OK even though there is no RA.

TBD: Keep an eye if the default route for ppp0 is missing. This is a no-no.

2) Ping from 2001:981:9d6e:1::1 to 2a00:1450:400e:801::200e (I assumed this is the ping that you are concerned about):

i) In the very first capture, packet# 304 failed to get ping-reply   // This is changed configuration

ii) In the second capture (datestamp2), the packet#620 and packet #621 succeeds   // This is reboot

iii) In the third capture. packet#189 failed to get a reply // After reconnect PPPoE

My view: This looks quite strange since most of the stuff (system/UTM) looked OK

To Do: See (3)

3) Also, I saw in the log the following message (system log): dhclient6: send_packet6: Operation not permitted

My view: more than likely it is iptables rule. This might impact the (2) above as well

To Do: SanderRutten

a) Please dump out the iptables when it is working and when it is not working and include them in this thread.

Sorry that I forgot about the iptables stuff. I made the mistake of making the assumption that it is OK. I just want to make sure this time

b) When ping fails, can you check the stattisctis of sending and receiving on the eth1 to see they are going up

4) Rebind issue 

My view: to be deal with once 2 is out of the way

5) The ps look fines

Thanks a lot SanderRutten for your help and sorry for my mistake of overlooking the iptables stuff.

Also please let me know your ideas, comments or questions. Thanks again.

Hi Le,

Attached are the iptables rules (iptables --list) 

iptables_ipv6_disabled.txt - IPv6 completely disabled
iptables_ipv6_enabled.txt - IPv6 enabled and configured. Not working.
iptables_ipv6_reconnected.txt - Reconnected PPPoE, not working.
iptables_ipv6_reboot.txt - The magic reboot got my IPv6 up and running again.

Not much difference in each output.

Just like before, if I now recoonect my PPPoE, or even reboot, it doesn't work anymore unless I disable IPv6 for a while and reconfigure everything, and reboot.

dhclient6 the cause?
One thing I just noticed in yesterdays ipv6.log:

2017:04:22-08:42:56 router ipv6_watchdog[4280]: Started dhclient6 -P (pid 4858)
And a few minutes later:
2017:04:22-08:51:15 router ipv6_watchdog[4280]: dhclient6 (pid 4858) has died

So this made me think.. Was my connection working during that time? Did I reconnect my PPPoE connection?
Since I currently have a working IPv6 connection after the reboot, I cheched the log:

2017:04:23-10:16:58 router ipv6_watchdog[4290]: Started dhclient6 -P (pid 4851)
Reconnected PPPoE, and my IPv6 connectivity was broken again. In the log:
2017:04:23-10:39:17 router ipv6_watchdog[4290]: dhclient6 (pid 4851) has died

This could be a problem.
Can I try to start it manually? I found the binary in /var/sec/chroot-dhcpc/usr/sbin/ but -P is not a valid option?

iptables.zip

Hi SandyRutten,

    Thanks so much for your help. It is appreciated.

    You are right in that we need to deal with "dhclient6 has died" and I will look into it.

    But in the meantime, the DHCPv6 and ICMPv6 traffics seems to be dropped as you can see

    from your previous 1781.Logs.zip in the captures (first and third) that did not work, there is

    no DHCPv6 traffics captured at all; while the second capture had DHCPv6 in it and we did get a prefix

    from DHCPv6 Server. So it is more probable that it is the iptables involved.

    Sorry to say it was my mistake not to mention to dump out both IPv4 and IPv6 iptables.

    Can you do the followings:

    1) Dump out the ip6tables (both working and non-working) and attach them to this thread

    2) Clear out the ip6tables (ip6tables -F) and try ping6 again

    Thank You so much and sorry for my omission. Thanks again.