Advisory: Support Portal Maintenance. Login is currently unavailable, more info available here.

RED not working

Hello,

we did an firmware update on an XG 550 Cluster last weekend.

Initially the RED 50  was running but after a colleague did some changes on another RED 15W it is disconnected now for unknown reasons.This was on Monday around 16:50. Currently the RED50 is the only device that is connected. Two other devices (one of them the RED 15W mentioned above) are offline.

Below you see the logfile of the device when it disconnected the first time:

Mon Jan 11 16:50:13 2021 REDD INFO command 'PORTSTATE 1E04,1004,1004,1004,1E04'
Mon Jan 11 16:50:13 2021 REDD INFO PORTSTATE LAN1: 1Gb/s,LAN2: Down,LAN3: Down,LAN4: Down
Mon Jan 11 16:50:13 2021 REDD INFO command 'PING 0 uplink=WAN1 uplinkstate=0 wan1_ip=192.168.0.128'
Mon Jan 11 16:50:13 2021 REDD INFO PING remote_tx=0 local_rx=0 diff=0
Mon Jan 11 16:50:13 2021 REDD INFO PONG local_tx=0
Mon Jan 11 16:50:29 2021 REDD INFO command 'PORTSTATE 1E04,1004,1004,1004,1E04'
Mon Jan 11 16:50:29 2021 REDD INFO A3400E6BF2EA04B/REDs Pilz is now disconnected
Mon Jan 11 16:50:29 2021 REDD INFO device is disconnected.
Mon Jan 11 16:50:35 2021 REDD INFO server: New connection from 84.130.87.219 (cipher ECDHE-RSA-AES256-GCM-SHA384), rev1
Mon Jan 11 16:50:36 2021 REDD INFO connected OK, pushing config
Mon Jan 11 16:50:36 2021 REDD INFO A3400E6BF2EA04B/REDs Pilz is now disconnected
Mon Jan 11 16:50:36 2021 REDD INFO device is disconnected.
Tue Jan 12 09:53:27 2021 REDD ERROR server: generate device config for red A3400E6BF2EA04B failed: Failed to get red device cert pem File /conf/certificate/internalcerts/red_device_A3400E6BF2EA04B.pem does not exists
Tue Jan 12 09:59:42 2021 REDD ERROR server: generate device config for red A3400E6BF2EA04B failed: Failed to get red device cert pem File /conf/certificate/internalcerts/red_device_A3400E6BF2EA04B.pem does not exists
Tue Jan 12 09:59:52 2021 REDD ERROR Failed to register on provisioning service: Failed to register RED device on provisioning server: Curl command to register utm on the provisioning server failed with error 1792
Tue Jan 12 10:00:01 2021 REDD ERROR server: generate device config for red A3400E6BF2EA04B failed: Failed to get red device cert pem File /conf/certificate/internalcerts/red_device_A3400E6BF2EA04B.pem does not exists

After one of my collegues did some troubleshooting we see now this:

Failed to register RED device on provisioning server: Curl command to register utm on the provisioning server failed with error 1792
Tue Jan 12 11:03:12 2021 REDD ERROR server: generate device config for red A3400E6BF2EA04B failed: Failed to get red device cert pem File /conf/certificate/internalcerts/red_device_A3400E6BF2EA04B.pem does not exists
Tue Jan 12 11:03:16 2021 REDD ERROR Failed to delete certificates: Deleting RED device cert from filesystem failed: file /conf/certificate/internalcerts/red_device_A3400E6BF2EA04B.pem does not exist
Tue Jan 12 11:11:10 2021 REDD ERROR server: generate device config for red A3400E6BF2EA04B failed: Failed to get red device cert pem File /conf/certificate/internalcerts/red_device_A3400E6BF2EA04B.pem does not exists
Tue Jan 12 11:11:15 2021 REDD ERROR Failed to register on provisioning service: Failed to register RED device on provisioning server: Curl command to register utm on the provisioning server failed with error 1792
Tue Jan 12 11:11:22 2021 REDD ERROR Failed to delete certificates: Deleting RED device cert from filesystem failed: file /conf/certificate/internalcerts/red_device_A3400E6BF2EA04B.pem does not exist
Tue Jan 12 12:16:36 2021 REDD INFO Forcing regeneration of RED certificates
Tue Jan 12 12:16:37 2021 REDD ERROR Failed to create tunnel: Failed: /bin/red2ctl create 3 reds3
Tue Jan 12 12:17:15 2021 REDD INFO Forcing regeneration of RED certificates
Tue Jan 12 14:18:58 2021 REDD ERROR Failed to notify red service: red_server is not running
Tue Jan 12 14:18:58 2021 REDD ERROR Failed to notify red server (enable_device_event)
Wed Jan 13 09:44:41 2021 REDD ERROR Failed to notify red service: red_server is not running
Wed Jan 13 09:44:41 2021 REDD ERROR Failed to notify red server (enable_device_event)

The Log red.log shows some kind of cycle:

Wed Jan 13 19:26:47 2021 REDD ERROR: DIE --------------------------------------------------------
Wed Jan 13 19:26:47 2021 REDD ERROR: Undefined subroutine &Red::Util::Error called at /lib32/perl/site_perl/5.20.1/Red/Util.pm line 141.

Wed Jan 13 19:26:47 2021 REDD ERROR: Trace begun at /lib32/perl/site_perl/5.20.1/Red/Util.pm line 55
Red::Util::print_stacktrace at /bin/red_server.pl line 55
main::__ANON__('Undefined subroutine &Red::Util::Error called at /lib32/perl/site_perl/5.20.1/Red/Util.pm line 141.^J') called at /lib32/perl/site_perl/5.20.1/Red/Util.pm line 141
Red::Util::mask2bits(undef) called at /lib32/perl/site_perl/5.20.1/Red/CSC.pm line 305
Red::CSC::get_split_networks('AUS_VPN_Pool_10.242.6.0\24') called at /lib32/perl/site_perl/5.20.1/Red/Util.pm line 768
Red::Util::generate_device_config('A360188498A5089') called at /bin/red_server.pl line 157

Undefined subroutine &Red::Util::Error called at /lib32/perl/site_perl/5.20.1/Red/Util.pm line 141.
Wed Jan 13 19:26:47 2021 REDD INFO: client: (Re-)loading device configurations
Wed Jan 13 19:26:47 2021 REDD INFO: client: shutdown requested, killing clients
Wed Jan 13 19:26:47 2021 REDD INFO: client: exiting
Wed Jan 13 19:26:51 2021 REDD INFO: server: Using RED firmware in /content/redfw/
Wed Jan 13 19:26:51 2021 REDD INFO: server: RED15(w) fw version set to 10224
Wed Jan 13 19:26:51 2021 REDD INFO: server: RED20 fw version set to 1-1111-c753026a5-fda4803
Wed Jan 13 19:26:51 2021 REDD INFO: server: RED50 fw version set to 10224
Wed Jan 13 19:26:51 2021 REDD INFO: server: RED60 fw version set to 1-1111-c753026a5-fda4803
Wed Jan 13 19:26:51 2021 REDD INFO: Startup server - waiting 5 seconds ...
Wed Jan 13 19:26:51 2021 REDD INFO: Startup client - waiting 5 seconds ...
Wed Jan 13 19:26:56 2021 REDD INFO: server: Configuration uploader process starting
Wed Jan 13 19:26:56 2021 REDD INFO: server: (Re-)loading device configurations
Wed Jan 13 19:27:04 2021 REDD ERROR: WARN -------------------------------------------------------
Wed Jan 13 19:27:04 2021 REDD ERROR: Use of uninitialized value in concatenation (.) or string at /lib32/perl/site_perl/5.20.1/Red/CSC.pm line 305.

Wed Jan 13 19:27:04 2021 REDD ERROR: Trace begun at /lib32/perl/site_perl/5.20.1/Red/Util.pm line 55
Red::Util::print_stacktrace at /bin/red_server.pl line 61
main::__ANON__('Use of uninitialized value in concatenation (.) or string at /lib32/perl/site_perl/5.20.1/Red/CSC.pm line 305.^J') called at /lib32/perl/site_perl/5.20.1/Red/CSC.pm line 305
Red::CSC::get_split_networks('AUS_VPN_Pool_10.242.6.0\24') called at /lib32/perl/site_perl/5.20.1/Red/Util.pm line 768
Red::Util::generate_device_config('A360188498A5089') called at /bin/red_server.pl line 157

Wed Jan 13 19:27:04 2021 REDD ERROR: DIE --------------------------------------------------------

The device itself is in a boot cycle.

Yesterday my colleague did also an Pattern Update:

Before:



One can clearly see that this was 



What is the issue here? 
What needs to be done to get this up and running again?

Any help appreciated ....

Parents
  • Hi ,

    Thank you for reaching out to the Community! 

    Can you tell us more about the changes your colleague made that caused this issue? 

    Did you try to delete the offline RED and reconfigure them on the firewall? 

    Thanks,

     

     
    Harsh Patel (H_Patel)

    Community Support Engineer | Sophos Technical Support
    Support Videos | Product Documentation | @SophosSupport | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' button.

  • It talked to my colleague this morning. He deleted and recreated the RED on the firewall on Tuesday around 12:17. The firewall was running in split mode before and is now configured in unified mode. 


    Tue Jan 12 12:16:37 2021 REDD ERROR Failed to create tunnel: Failed: /bin/red2ctl create 3 reds3
    Tue Jan 12 12:17:15 2021 REDD INFO Forcing regeneration of RED certificates


    Tue Jan 12 14:18:58 2021 REDD ERROR Failed to notify red service: red_server is not running
    Tue Jan 12 14:18:58 2021 REDD ERROR Failed to notify red server (enable_device_event)

    He also send me some screenshots where one can see that the RED tries to connect to the IP Address the firewall is listening to.

    Is there any need to do a reset on the RED in order to reintialize? If yes what needs to be done?

Reply
  • It talked to my colleague this morning. He deleted and recreated the RED on the firewall on Tuesday around 12:17. The firewall was running in split mode before and is now configured in unified mode. 


    Tue Jan 12 12:16:37 2021 REDD ERROR Failed to create tunnel: Failed: /bin/red2ctl create 3 reds3
    Tue Jan 12 12:17:15 2021 REDD INFO Forcing regeneration of RED certificates


    Tue Jan 12 14:18:58 2021 REDD ERROR Failed to notify red service: red_server is not running
    Tue Jan 12 14:18:58 2021 REDD ERROR Failed to notify red server (enable_device_event)

    He also send me some screenshots where one can see that the RED tries to connect to the IP Address the firewall is listening to.

    Is there any need to do a reset on the RED in order to reintialize? If yes what needs to be done?

Children