Sophos Central Endpoint and SEC: Computers fail/hang on boot after the Microsoft Windows April 9, 2019 update. Please follow knowledge base article 133945

Learn about the Benefits of Multi-Factor Authentication (MFA). Turn your MFA on now!

Outage on MySophos and Partner Portal. You may contact Sophos Support through Phone.

Sophos WLAN AP55 Firmware / Recover - HOWTO: Unbrick your Sophos AP!

Hello,

I have fixed my AP55 over the serial connection (called cosole port at the AP).

Sophos doesn't provide a recovery tool to fix a broken AP55, you must send it back! That was not a solution for me, so I try to bring it back to life! I have droped my AP55 from the AP-List on the UTM, but then it doesn't come up again. It stays weak green, no blinking, nothing.

Then I have found the following post:

 

https://community.sophos.com/products/unified-threat-management/f/wireless-security/56598/accesspoint-howto-troubleshoot-on-console

 

But the interesting thing ist the following:

 

For troubleshooting issues, I needed to know some more Informations about why an AP 55 was not getting discovered by my Sophos UTM.  
Therefore I decided to connect my Notebook to the AP's Console Port to figure out what can be done via Console Port.  
Because those Informations could be helpful for anybody else, here you are. 

Connection can be established, using 115200 Baud. There is no login Password required. Just press enter.

 

So, you buy a RJ-45 to serial-connector and plug the serial-connector in your PC.

I have done that with a USB-to-Serial-Converter. A cheap adpater from amazon. But with Windows 10, the device doesn’t come up. I found the following driver:

 

http://www.totalcardiagnostics.com/support/Knowledgebase/Article/View/92/20/prolific-usb-to-serial-fix-official-solution-to-code-10-error

 

http://www.totalcardiagnostics.com/files/PL2303_64bit_Installer.exe

 

An then, tada. You are able to connect to the AP through the adapter.

 

I have connected me to the AP with putty, You must switch to serial connection and add the Baudrate: 115200. The right COM-Port can you see in the Windows Device Manager. In the empty window you must press enter. When you pull out the power from the AP, you can see U-Boot comes up an try to load the OpenWRT-Firmware, that SOPHOS deploys for the APs.

But in my case, the following problem occured:

 

U-Boot 1.1.4-gcb612594 (Dec 23 2016 - 12:50:03)

ELX version: 1.0.0

 

7679WSC - Scorpion 1.0DRAM:

sri

Scorpion 1.0

ath_ddr_initial_config(178): (32bit) ddr2 init

tap = 0x00000003

Tap (low, high) = (0x5, 0x1b)

Tap values = (0x10, 0x10, 0x10, 0x10)

128 MB

Flash Manuf Id 0xc2, DeviceId0 0x20, DeviceId1 0x18

Flash [MX25L12845E] sectors: 256

Flash: 16 MB

In:    serial

Out:   serial

Err:   serial

Net:   ath_gmac_enet_initialize...

athrs_sgmii_res_cal: cal value = 0xe

Fetching MAC Address from 0x87fed9ec

ath_gmac_enet_initialize: reset mask:c02200

Scorpion ---->8035 PHY*

AR8035 PHY reg init

: cfg1 0x80000000 cfg2 0x7114

eth0: 00:aa:bb:cc:dd:00

AR8035 found!

[0:4]Phy ID 4d:d072

Port 0, Neg Success

eth0 up

eth0

Setting 0x18116290 to 0x458ba14f

Hit any key to stop autoboot:  0

## Booting image at 9f070000 ...

   Image Name:   MIPS OpenWrt Linux-3.18.11

   Created:      2016-12-23  12:57:39 UTC

   Image Type:   MIPS Linux Kernel Image (gzip compressed)

   Data Size:    7132027 Bytes =  6.8 MB

   Load Address: 80060000

   Entry Point:  80060000

   Verifying Checksum at 0x9f070040 ...Bad Data CRC

Speed is 1000T

 

The firmware is corrupt, that is the problem.

It takes me one day to find the right solution, but I don’t want to show you the whole shit i tried. So, only the interesting things.

 

First we must make a little network from the PC to the Sophos AP.

On the PC or an other device you must provide a TFTP-Server and a DHCP-Server.

I use the following tools:

 

http://www.dhcpserver.de/cms/

https://sourceforge.net/projects/tftp-server/

 

The problem is the AP knows rests of the network configuartion. When you download the firmware to the AP, It only takes it from the IP, that it shows up.

 

Back in the putty-serial-session, you must stop U-Boot to load the corrupt image.

At this step at the boot, press any key:

Setting 0x18116290 to 0x458ba14f

Hit any key to stop autoboot:  0

 

Then you are in the U-Boot-Bootloader, where the magic happends.

 

Type help to see the possible commands:

 

ath> help

?       - alias for 'help'

autoscr - run script from memory

base    - print or set address offset

bdinfo  - print Board Info structure

boot    - boot default, i.e., run 'bootcmd'

bootd   - boot default, i.e., run 'bootcmd'

bootelf - Boot from an ELF image in memory

bootm   - boot application image from memory

bootp   - boot image via network using BootP/TFTP protocol

bootvx  - Boot vxWorks from an ELF image

cmp     - memory compare

coninfo - print console devices and information

cp      - memory copy

crc32   - checksum calculation

dhcp    - invoke DHCP client to obtain IP/boot params

echo    - echo args to console

erase   - erase FLASH memory

ethreg    - S26 PHY Reg rd/wr  utility

exit    - exit script

flinfo  - print FLASH memory information

go      - start application at address 'addr'

help    - print online help

iminfo  - print header information for application image

itest   - return true/false on integer compare

loop    - infinite loop on address range

md      - memory display

compute MD5 message digestmii     - MII utility commands

mm      - memory modify (auto-incrementing)

mtest   - simple RAM test

mw      - memory write (fill)

nfs     - boot image via network using NFS protocol

nm      - memory modify (constant address)

pci     - list and access PCI Configuration Space

ping    - send ICMP ECHO_REQUEST to network host

pll cpu-pll dither ddr-pll dither - Set to change CPU & DDR speed

pll erase

pll get

printenv- print environment variables

progmac - Set ethernet MAC addresses

protect - enable or disable FLASH write protection

rarpboot- boot image via network using RARP/TFTP protocol

reset   - Perform RESET of the CPU

run     - run commands in an environment variable

saveenv - save environment variables to persistent storage

sendmagic       - (usage) send/broadcast MAGIC PACKET to network host

                - <timeout> timeout for response

                - <retry> number of times magic to be sent to network host

                - <devid_base_addr> baseaddr of sector containing devid

                - <devid_len> offset to base addr

                - <offset_to_baseaddr> offset to base addr

sendsts - send status of firmware recovery process

                - <stscode> 0 - send apstate, non-zero - send specified statuscode

setenv  - set environment variables

sleep   - delay execution for some time

test    - minimal test like /bin/sh

tftpboot- boot image via network using TFTP protocol

version - print monitor version

 

We want to know the IP, from where the AP expect the firmware, so type:

 

Tftpboot

 

ath> tftpboot

Speed is 1000T

dup 1 speed 1000

Using eth0 device

TFTP from server 192.168.99.8; our IP address is 192.168.99.9

Filename 'uImage_AP100'.

Load address: 0x81000000

Loading: T T T T T T T T T T T T T T

 

My AP wants to download it from 192.168.99.8 (the firmware must named exactly like the „Filename“ above).

 

Now you must setup the DHCP-Server and the TFTP-Server on your PC to the IP-Range that the AP wants and connect the AP to the NIC where you put the DHCP-Server on. Give the TFTP-Server the address that the AP wants to have (in my case 192.168.99.8).

 

You can download all the firmware files for the APs from your UTM. Connect via WinSCP to the UTM (you must enable shell access in the WebAdmin), connect with the loginuser.

Go to /etc/wireless/firmware and download AP55.uimage

Copy your firmware in your TFTP-Server-Root-Directory on your PC and name it like the AP it wants to have. In my case: uImage_AP100 (without filetype!)

 

 

Then type the following in the putty-serial-connection:

ath> tftpboot

Speed is 1000T

dup 1 speed 1000

Using eth0 device

TFTP from server 192.168.99.8; our IP address is 192.168.99.9

Filename 'uImage_AP100'.

Load address: 0x81000000

Loading: #################################################################

         #################################################################

         #################################################################

         ############################

done

Bytes transferred = 7132091 (6cd3bb hex)

Now you have the Image on the AP at the address: 0x81000000

Now erase the flash memory.

You must calculate the right memory spaces in hex. We have made it for the AP55, for other APs it can be different. You can flash other APs with this procedure, but with other memory spaces and other firmwares. ;-)

I show you how we calculate from where to where we must erase the flash memory.

Type:

ath> bdinfo

boot_params = 0x87F7BFB0

memstart    = 0x80000000

memsize     = 0x08000000

flashstart  = 0x9F000000

flashsize   = 0x01000000

flashoffset = 0x00029CD4

ethaddr     = 00:00:AA:BB:CC:DD

ip_addr     = 192.168.99.9

baudrate    = 115200 bps

You can see the flashsize, this is important. When you boot the AP, and it ends up with bad-checksum error, you can see the memory address where the AP wants to find the boot-image, look here:

ath> boot

## Booting image at 9f070000 ...

   Image Name:   MIPS OpenWrt Linux-3.18.11

   Created:      2016-12-23  12:57:39 UTC

So you must add 0x9f070000 plus 0x01000000 with a hex-calculator. With the Windows Calc you can do that, the result is: A0070000

Type the following:

ath> era 0x9f070000 0xA0070000

Erasing flash...

First 0x7 last 0xff sector size 0x10000                                      255

Erased 249 sectors

So, now you are ready to flash the firmware image to flash memory, that we put with TFTP at the address:

0x81000000

0x6cd3bb is the size of the image. That info we get from TFTP-copy-process at the end, watch above.

0x9f070000 is the address where U-Boot want to find the image, you can see it above at the moment of the boot.

More infos for U-Boot:

www.denx.de/.../UBootCmdGroupFlash

ath> cp.b 0x81000000 0x9f070000 0x6cd3bb

Copy to Flash...

 Copy 7132091 [0x6cd3bb] byte to Flash... write addr: 9f070000

Done

Now you are ready to go. Type boot and have fun, now the following must be appear:

ath> boot

## Booting image at 9f070000 ...

   Image Name:   MIPS OpenWrt Linux-3.18.11

   Created:      2016-12-23  12:57:39 UTC

   Image Type:   MIPS Linux Kernel Image (gzip compressed)

   Data Size:    7132027 Bytes =  6.8 MB

   Load Address: 80060000

   Entry Point:  80060000

   Verifying Checksum at 0x9f070040 ...OK

   Uncompressing Kernel Image ... OK

 

Starting kernel ...

Later you can find it at Sophos UTM as a new AP and manage it.

I hope, I can help a manny people with this HOW-TO.

Sorry for my english and grammar failures. I type this fast and with a german Microsoft WORD..

Enjoy your new AP! :)

 

 

 

 

 

  • Hello guys,

     

    first of all thanks to Mr. Dobbermann for his work in discovering this method. It is known to some support members that there are ways, including this

    and that you can make it work. This is also somewhat a form of tinkering so one information has to be added here which is important:

    This method can work, but always keep in mind that this is not an approved way according to our support guidelines. So if you do this,

    it can lead to a problematic support status when it comes to issues following this treatment and when you open a case, we have to know

    this has been done.

     [EDIT]

    [I will, however ask the guys responsible for approving such processes to look at this and test it through so we can maybe turn this into

    an official KBA and get this approved. Until then : please proceed at your own risk.]

     

    => I have just received the information that we can not approve this as this would open a way to install custom firmware on the device

    and therefore leads to an FCC violation.  Which in turn means that we do not approve this method. So if you're having issues it is best to contact

    Support first.

     

    Regards,

    Cheerok

  • Hello Robin,

    Thank you for your explanation, I've just revived my AP55 from a crashed firmware upgrade!

    Cordialement,

    Thierry

  • Now I have a AP15 with a broken firmware. The recovery program is not compatible with this AP.

    So maybe in the next few days, I'm writing a manual to recover this AP through the serial-console-connection on the motherboard of the AP15. You must open the AP for this and use a adpater like this: https://www.amazon.de/gp/product/B0757FQ5CX/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

    to make a serial connection to your computer and flash the firmware at the same way as the AP55.

    At the moment only one YouTube-Video exists to recover the AP15, but in this case he unsolder the EEPROM to flash the firmware. But this way is much too complex for all.

    https://www.youtube.com/watch?v=xyvU6sALPQA&t=316s

     

    Maybe with this manual it is possible to recover all Sophos APs that are don't covered by the original provided tool from Sophos. - as long as the have a serial-connection on the motherboard. I don't know if all models have a pin-out, maybe at a few models you must soldering a pin-bar to acces the serial-connection. But the AP15 has this bar.

    And when you can't recover the AP through this, you have the option of JTAG - but this is very complex. The AP15 has a JTAG-header.

    With the tool from Sophos you can only recover AP 10 / 30 / 50.

     

    Most of my information, I get from: https://www.dd-wrt.com/wiki/index.php/Serial_Recovery

    So thanks to DDWRT for the good informations!

     

    So, stay tuned!

     

    Greetings,

     

    Robin

  • In reply to RobinDobbermann:

    Hello everybody,

     

    I have made a video, how to flash the Sophos AP 15 (and maybe others).

    And after this video, you follow my guide above and you´re Sophos AP is ready to go.

     

    So, good luck and I hope to hear from you!

     

    www.youtube.com/watch

     

    Greetings,

    Robin

  • In reply to RobinDobbermann:

    My AP55 asking for username/password now.

     

    any idea?

  • In reply to Usbkey:

    It can't asking you for a password or username, when it boots up, it shows you the bootloader U-Boot and boot the image.. When and where should the Sophos AP ask you this?

    When you can, post a picture.

     

    BTW:

     

    Recovery for the AP15 was succesfull! Easy as the AP55.

  • In reply to RobinDobbermann:

    When it fully boot up.

    I was trying to modify the allowed channels to utiliza 80Mhz channels but coudl not login.

     

    The login prompt is simple just like below.

    OpenWrt:>

  • In reply to Usbkey:

    Okay, but this what you tryin to make is not covered by my guide.. Your AP boots up and you want to modify settings in OpenWRT. I show the Receovery of a AP, and not how to modify the AP without the UTM.

  • In reply to RobinDobbermann:

    understood, just want to see if you know the password.

     

    According to Sophos engineers, all Sophos AP only use 40Mhz channels for enterprise deployment so I could not archive 866Mbps connection speed with both AP55/AP55C

     

    There is a file allowed_channels at /tmp once you login and I am trying to add new channels to that file.

     

    now you know what I am trying to do.

     

    Thanks

  • Thanks! Worked exactly like your description!

  • Thanks so much! This worked for me too!

    I didn't used the recommended TFTP Server because the DHCP Server Tool brings a TFTP Server too.

    Because I have had some Problems to bring my DHCP Server to live here the configuration of my ini-file:

     

    [GENERAL]
    SUBNETMASK=255.255.255.0
    ROUTER_0=192.168.99.1
    DNS_0=192.168.99.1

    [SETTINGS]
    IPBIND_0=192.168.99.8
    IPPOOL_0=192.168.99.10-50

    [TFTP-SETTINGS]
    EnableTFTP=1
    ROOT=D:\red-ap_recoveryapp\dhcpsrv2.5.2\wwwroot
    WritePermission=0

    [HTTP-SETTINGS]
    EnableHTTP=1
    ROOT=D:\red-ap_recoveryapp\dhcpsrv2.5.2\wwwroot