This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

XG430 10G Flexi Port link not coming up - replacement of the module

During HA-Rebuild and Upgrade to 19.5.1 I noticed a log in System logs: "PortA4 up" on the peer node at a time when it was unexpected.

I then noticed on the physical interface, that the LED of that Port were off while the other 3 were blinking.

It is 10G Copper with SFP Plus Cable.

I replaced the cable with the same result. After 15 or more minutes I checked the device again and the port was up and running - as can be seen from the below output and screenshots.

I checked the Troubleshooting tests for network issues, but I found no log information or debug procedere else than this one: https://support.sophos.com/support/s/article/KB-000036345?language=en_US

Are there useful logs to analyze this?

This was the port status when it was off:

PortA3    Link encap:Ethernet  HWaddr C8:4F:86:FC:00:0D  
          inet6 addr: fe80::ca4f:86ff:fefc:d/64 Scope:Link
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:772884 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1243396 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:447511125 (426.7 MiB)  TX bytes:782974630 (746.7 MiB)

PortA4    Link encap:Ethernet  HWaddr C8:4F:86:FC:00:0D  
          inet6 addr: fe80::ca4f:86ff:fefc:e/64 Scope:Link
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

and than when it was on.

--

XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Primary# ifconfig PortA4
PortA4    Link encap:Ethernet  HWaddr C8:4F:86:FC:00:0D
          inet6 addr: fe80::ca4f:86ff:fefc:e/64 Scope:Link
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:7259882 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8158434 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5353399207 (4.9 GiB)  TX bytes:7343072503 (6.8 GiB)

XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Primary# ifconfig PortA3
PortA3    Link encap:Ethernet  HWaddr C8:4F:86:FC:00:0D
          inet6 addr: fe80::ca4f:86ff:fefc:d/64 Scope:Link
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:9608423 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6768176 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7102495310 (6.6 GiB)  TX bytes:1499533574 (1.3 GiB)

The 4 Ports are in a LACP LAG.

That is the problematic port:
XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Primary# ethtool PortA4
Settings for PortA4:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x0000000f (15)
                               drv probe link timer
        Link detected: yes

One other Port:

XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Primary# ethtool PortA3
Settings for PortA3:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x0000000f (15)
                               drv probe link timer
        Link detected: yes

--



This thread was automatically locked due to age.
  • The ethtool output: was this while it was already online again? 

    BTW is your xmit Hash policy correct? Likely it does not match with your switch. 

    __________________________________________________________________________________________________________________

  • yes. unfortunately I did not collect it while it was down.

    The ethtool output: was this while it was already online again? 

    will check about xmit policy on switch side.

  • it would be OK from switch-side. But that is only loadbalancing - should have nothing to do with the port issue.

    Can you please tell if this is still true?

    NC-94073
    • SFOS 19.0.0 GA-Build317 (19.0.0.317) [Tupai]
    • NoRelease
    • XGS BSP
    XGS 10G interface not working when interface speed is set to Auto-negotiation (Physical or LAG)

    Issue: XGS 10G interface is not working when interface speed is set to Auto-negotiation (Physical or LAG)

    Affected Product: Only XGS hardware with 10G interface

    Set interface speed to Manual 10000 Mbps - Full-Duplex (Applicable for Physical, LAG interfaces).

  • Yes it is still true, but with a firmware update the Link state was fixed to 10 gbe, as you can see above. 

    __________________________________________________________________________________________________________________

  • Why does it autoneg when the interface is set to 10g full? Is that a correct behaviour?

    did a ethtool -r PortA4 and that causes the port to go down for a while.

    G430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Auxiliary# ethtool -M PortA4
    Supported interface speed: Supports auto-negotiation:   No
    XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Auxiliary# ethtool -r PortA4
    XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Auxiliary# ethtool -M PortA4
    Supported interface speed:                              1000fd -> 1000baseT/Full


    XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Auxiliary# ethtool PortA4
    Settings for PortA4:
            Supported ports: [ ]
            Supported link modes:   1000baseT/Full
                                    1000baseKX/Full
                                    10000baseT/Full
                                    1000baseX/Full
                                    10000baseSR/Full
                                    10000baseLR/Full
            Supported pause frame use: Symmetric
            Supports auto-negotiation: Yes
            Supported FEC modes: Not reported
            Advertised link modes:  1000baseT/Full
                                    1000baseKX/Full
                                    10000baseT/Full
                                    1000baseX/Full
                                    10000baseSR/Full
                                    10000baseLR/Full
            Advertised pause frame use: No
            Advertised auto-negotiation: Yes
            Advertised FEC modes: Not reported
            Speed: Unknown!
            Duplex: Unknown! (255)
            Port: Other
            PHYAD: 0
            Transceiver: internal
            Auto-negotiation: off
            Supports Wake-on: g
            Wake-on: g
            Current message level: 0x0000000f (15)
                                   drv probe link timer
            Link detected: no

    XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Auxiliary# ethtool PortA1 | grep uto-neg
            Supports auto-negotiation: No
            Advertised auto-negotiation: No
            Auto-negotiation: off
    XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Auxiliary# ethtool PortA2 | grep uto-neg
            Supports auto-negotiation: No
            Advertised auto-negotiation: No
            Auto-negotiation: off
    XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Auxiliary# ethtool PortA3 | grep uto-neg
            Supports auto-negotiation: No
            Advertised auto-negotiation: No
            Auto-negotiation: off
    XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Auxiliary# ethtool PortA4 | grep uto-neg
            Supports auto-negotiation: Yes
            Advertised auto-negotiation: Yes
            Auto-negotiation: off

    the switching team reported the port is set to auto neg on switch side. as this is a bad setting in this situation they'll try to change it. looks like the port group needs to be rebuild for that as the port speed cannot be changed in a port group.

  • So the colleagues have changed the port group on the switch to use 10G full static and still this one flexi Port out of four is doing some auto negotiate in  opposite to the adapter setting: auto neg: off

    Can we convince PortA4 to do no auto negotiation?


    XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Auxiliary# ethtool PortA4 | grep -i auto-neg
            Supports auto-negotiation: Yes
            Advertised auto-negotiation: Yes
            Auto-negotiation: off

  • I am mixing something up: The bug only involved the XGS Flexiport Module of 1U Appliances. You are not affected by the bug. The Bug was due the firmware of the Flexi Port itself, which is not applicable to your scenario, as you have an other Flexi Port module. 

    __________________________________________________________________________________________________________________

  • our XG430 is a 1U machine.

    ethtool -i PortA4
    driver: i40e
    version: 2.10.19.82
    firmware-version: 5.05 0x800028ac 0.0.0
    expansion-rom-version:
    bus-info: 0000:02:00.3
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: yes

    do we have other tools to check the firmware version you're pointing at?

    With v18 we started having issues with lag members in auto speed mode that we did not have before. I mentioned in the link to my old post above. That's when we changed it for the firewall side but forgot to change the switches to match that setting.

    Or do you suggest opening a case? If not a software bug, I suspect something faulty on this port.

    btpw: that is the status now after some minutes in faulty state:

    XG430_WP02_SFOS 19.5.1 MR-1-Build278 HA-Auxiliary# ethtool PortA4
    Settings for PortA4:
            Supported ports: [ FIBRE ]
            Supported link modes:   10000baseT/Full
            Supported pause frame use: Symmetric
            Supports auto-negotiation: No
            Supported FEC modes: Not reported
            Advertised link modes:  10000baseT/Full
            Advertised pause frame use: No
            Advertised auto-negotiation: No
            Advertised FEC modes: Not reported
            Speed: 10000Mb/s
            Duplex: Full
            Port: Direct Attach Copper
            PHYAD: 0
            Transceiver: internal
            Auto-negotiation: off
            Supports Wake-on: g
            Wake-on: g
            Current message level: 0x0000000f (15)
                                   drv probe link timer
            Link detected: yes

  • Again: It is only affected the particular SKU, which only the XGS 1U supports. 

    This particular module has a firmware bug, which does not support Auto-Neg. 

    You are not effected by this problem, as you cannot have the affected hardware in the first place. 

    __________________________________________________________________________________________________________________

  • ok - I did not notice the S in you writing XGS. Thanks for your help so far!