Last night at 22:58:16 ET all of our RED devices began disconnecting and are unable to reconnect.
We've made no recent configuration changes to our network or Sophos UTMs.
We have 37 REDs of various models, many of which have been in use for several years.
This is a summary of the logged events from the initial disconnection to the current looping errors:
2022:07:28-22:58:16 gateway-2 red_server[34459]: R20001GHKVGDX50: command '{"data":{"message":"Failed to send keepalive frame: Trying to send PING but expecting PONG to receive first","type":"RUNTIME_ERROR_OCCURRED"},"type":"DISCONNECT"}'
2022:07:28-22:58:16 gateway-2 red_server[34459]: R20001GHKVGDX50: Disconnecting: RUNTIME_ERROR_OCCURRED, Failed to send keepalive frame: Trying to send PING but expecting PONG to receive first
2022:07:29-08:22:09 gateway-2 red_server[43473]: SELF: New connection from XX.XX.43.9 with ID R20001GHKVGDX50 (cipher AES256-GCM-SHA384), rev1
2022:07:29-08:22:09 gateway-2 red_server[43473]: SELF: no such client: R20001GHKVGDX50
2022:07:29-08:22:09 gateway-2 red_server[43473]: R20001GHKVGDX50: Sending json message {"data":{},"type":"DEVICE_NOT_BOUND_TO_UTM"}
Failed attempts to resolve:
- Toggling the devices enabled status
- Enabling and disabling tunnel compression
- Changing UTM hostname
- Physically restarting RED devices
We have a critical case open with Sophos, but it has been over an hour with no contact from them.
This thread was automatically locked due to age.
