Sophos Central Endpoint and SEC: Computers fail/hang on boot after the Microsoft Windows April 9, 2019 update. Please follow knowledge base article 133945
Learn about the Benefits of Multi-Factor Authentication (MFA). Turn your MFA on now!
Outage on MySophos and Partner Portal. You may contact Sophos Support through Phone.
We'd love to hear about it! Click here to go to the product suggestion community
Hello,
I have fixed my AP55 over the serial connection (called cosole port at the AP).
Sophos doesn't provide a recovery tool to fix a broken AP55, you must send it back! That was not a solution for me, so I try to bring it back to life! I have droped my AP55 from the AP-List on the UTM, but then it doesn't come up again. It stays weak green, no blinking, nothing.
Then I have found the following post:
https://community.sophos.com/products/unified-threat-management/f/wireless-security/56598/accesspoint-howto-troubleshoot-on-console
But the interesting thing ist the following:
For troubleshooting issues, I needed to know some more Informations about why an AP 55 was not getting discovered by my Sophos UTM. Therefore I decided to connect my Notebook to the AP's Console Port to figure out what can be done via Console Port. Because those Informations could be helpful for anybody else, here you are. Connection can be established, using 115200 Baud. There is no login Password required. Just press enter.
So, you buy a RJ-45 to serial-connector and plug the serial-connector in your PC.
I have done that with a USB-to-Serial-Converter. A cheap adpater from amazon. But with Windows 10, the device doesn’t come up. I found the following driver:
http://www.totalcardiagnostics.com/support/Knowledgebase/Article/View/92/20/prolific-usb-to-serial-fix-official-solution-to-code-10-error
http://www.totalcardiagnostics.com/files/PL2303_64bit_Installer.exe
An then, tada. You are able to connect to the AP through the adapter.
I have connected me to the AP with putty, You must switch to serial connection and add the Baudrate: 115200. The right COM-Port can you see in the Windows Device Manager. In the empty window you must press enter. When you pull out the power from the AP, you can see U-Boot comes up an try to load the OpenWRT-Firmware, that SOPHOS deploys for the APs.
But in my case, the following problem occured:
U-Boot 1.1.4-gcb612594 (Dec 23 2016 - 12:50:03)
ELX version: 1.0.0
7679WSC - Scorpion 1.0DRAM:
sri
Scorpion 1.0
ath_ddr_initial_config(178): (32bit) ddr2 init
tap = 0x00000003
Tap (low, high) = (0x5, 0x1b)
Tap values = (0x10, 0x10, 0x10, 0x10)
128 MB
Flash Manuf Id 0xc2, DeviceId0 0x20, DeviceId1 0x18
Flash [MX25L12845E] sectors: 256
Flash: 16 MB
In: serial
Out: serial
Err: serial
Net: ath_gmac_enet_initialize...
athrs_sgmii_res_cal: cal value = 0xe
Fetching MAC Address from 0x87fed9ec
ath_gmac_enet_initialize: reset mask:c02200
Scorpion ---->8035 PHY*
AR8035 PHY reg init
: cfg1 0x80000000 cfg2 0x7114
eth0: 00:aa:bb:cc:dd:00
AR8035 found!
[0:4]Phy ID 4d:d072
Port 0, Neg Success
eth0 up
eth0
Setting 0x18116290 to 0x458ba14f
Hit any key to stop autoboot: 0
## Booting image at 9f070000 ...
Image Name: MIPS OpenWrt Linux-3.18.11
Created: 2016-12-23 12:57:39 UTC
Image Type: MIPS Linux Kernel Image (gzip compressed)
Data Size: 7132027 Bytes = 6.8 MB
Load Address: 80060000
Entry Point: 80060000
Verifying Checksum at 0x9f070040 ...Bad Data CRC
Speed is 1000T
The firmware is corrupt, that is the problem.
It takes me one day to find the right solution, but I don’t want to show you the whole shit i tried. So, only the interesting things.
First we must make a little network from the PC to the Sophos AP.
On the PC or an other device you must provide a TFTP-Server and a DHCP-Server.
I use the following tools:
http://www.dhcpserver.de/cms/
https://sourceforge.net/projects/tftp-server/
The problem is the AP knows rests of the network configuartion. When you download the firmware to the AP, It only takes it from the IP, that it shows up.
Back in the putty-serial-session, you must stop U-Boot to load the corrupt image.
At this step at the boot, press any key:
Then you are in the U-Boot-Bootloader, where the magic happends.
Type help to see the possible commands:
ath> help
? - alias for 'help'
autoscr - run script from memory
base - print or set address offset
bdinfo - print Board Info structure
boot - boot default, i.e., run 'bootcmd'
bootd - boot default, i.e., run 'bootcmd'
bootelf - Boot from an ELF image in memory
bootm - boot application image from memory
bootp - boot image via network using BootP/TFTP protocol
bootvx - Boot vxWorks from an ELF image
cmp - memory compare
coninfo - print console devices and information
cp - memory copy
crc32 - checksum calculation
dhcp - invoke DHCP client to obtain IP/boot params
echo - echo args to console
erase - erase FLASH memory
ethreg - S26 PHY Reg rd/wr utility
exit - exit script
flinfo - print FLASH memory information
go - start application at address 'addr'
help - print online help
iminfo - print header information for application image
itest - return true/false on integer compare
loop - infinite loop on address range
md - memory display
compute MD5 message digestmii - MII utility commands
mm - memory modify (auto-incrementing)
mtest - simple RAM test
mw - memory write (fill)
nfs - boot image via network using NFS protocol
nm - memory modify (constant address)
pci - list and access PCI Configuration Space
ping - send ICMP ECHO_REQUEST to network host
pll cpu-pll dither ddr-pll dither - Set to change CPU & DDR speed
pll erase
pll get
printenv- print environment variables
progmac - Set ethernet MAC addresses
protect - enable or disable FLASH write protection
rarpboot- boot image via network using RARP/TFTP protocol
reset - Perform RESET of the CPU
run - run commands in an environment variable
saveenv - save environment variables to persistent storage
sendmagic - (usage) send/broadcast MAGIC PACKET to network host
- <timeout> timeout for response
- <retry> number of times magic to be sent to network host
- <devid_base_addr> baseaddr of sector containing devid
- <devid_len> offset to base addr
- <offset_to_baseaddr> offset to base addr
sendsts - send status of firmware recovery process
- <stscode> 0 - send apstate, non-zero - send specified statuscode
setenv - set environment variables
sleep - delay execution for some time
test - minimal test like /bin/sh
tftpboot- boot image via network using TFTP protocol
version - print monitor version
We want to know the IP, from where the AP expect the firmware, so type:
Tftpboot
ath> tftpboot
dup 1 speed 1000
Using eth0 device
TFTP from server 192.168.99.8; our IP address is 192.168.99.9
Filename 'uImage_AP100'.
Load address: 0x81000000
Loading: T T T T T T T T T T T T T T
My AP wants to download it from 192.168.99.8 (the firmware must named exactly like the „Filename“ above).
Now you must setup the DHCP-Server and the TFTP-Server on your PC to the IP-Range that the AP wants and connect the AP to the NIC where you put the DHCP-Server on. Give the TFTP-Server the address that the AP wants to have (in my case 192.168.99.8).
You can download all the firmware files for the APs from your UTM. Connect via WinSCP to the UTM (you must enable shell access in the WebAdmin), connect with the loginuser.
Go to /etc/wireless/firmware and download AP55.uimage
Copy your firmware in your TFTP-Server-Root-Directory on your PC and name it like the AP it wants to have. In my case: uImage_AP100 (without filetype!)
Then type the following in the putty-serial-connection:
Loading: #################################################################
#################################################################
############################
done
Bytes transferred = 7132091 (6cd3bb hex)
Now you have the Image on the AP at the address: 0x81000000
Now erase the flash memory.
You must calculate the right memory spaces in hex. We have made it for the AP55, for other APs it can be different. You can flash other APs with this procedure, but with other memory spaces and other firmwares. ;-)
I show you how we calculate from where to where we must erase the flash memory.
Type:
ath> bdinfo
boot_params = 0x87F7BFB0
memstart = 0x80000000
memsize = 0x08000000
flashstart = 0x9F000000
flashsize = 0x01000000
flashoffset = 0x00029CD4
ethaddr = 00:00:AA:BB:CC:DD
ip_addr = 192.168.99.9
baudrate = 115200 bps
You can see the flashsize, this is important. When you boot the AP, and it ends up with bad-checksum error, you can see the memory address where the AP wants to find the boot-image, look here:
ath> boot
So you must add 0x9f070000 plus 0x01000000 with a hex-calculator. With the Windows Calc you can do that, the result is: A0070000
Type the following:
ath> era 0x9f070000 0xA0070000
Erasing flash...
First 0x7 last 0xff sector size 0x10000 255
Erased 249 sectors
So, now you are ready to flash the firmware image to flash memory, that we put with TFTP at the address:
0x81000000
0x6cd3bb is the size of the image. That info we get from TFTP-copy-process at the end, watch above.
0x9f070000 is the address where U-Boot want to find the image, you can see it above at the moment of the boot.
More infos for U-Boot:
www.denx.de/.../UBootCmdGroupFlash
ath> cp.b 0x81000000 0x9f070000 0x6cd3bb
Copy to Flash...
Copy 7132091 [0x6cd3bb] byte to Flash... write addr: 9f070000
Done
Now you are ready to go. Type boot and have fun, now the following must be appear:
Verifying Checksum at 0x9f070040 ...OK
Uncompressing Kernel Image ... OK
Starting kernel ...
Later you can find it at Sophos UTM as a new AP and manage it.
I hope, I can help a manny people with this HOW-TO.
Sorry for my english and grammar failures. I type this fast and with a german Microsoft WORD..
Enjoy your new AP! :)
Hello guys,
first of all thanks to Mr. Dobbermann for his work in discovering this method. It is known to some support members that there are ways, including this
and that you can make it work. This is also somewhat a form of tinkering so one information has to be added here which is important:
This method can work, but always keep in mind that this is not an approved way according to our support guidelines. So if you do this,
it can lead to a problematic support status when it comes to issues following this treatment and when you open a case, we have to know
this has been done.
[EDIT]
[I will, however ask the guys responsible for approving such processes to look at this and test it through so we can maybe turn this into
an official KBA and get this approved. Until then : please proceed at your own risk.]
=> I have just received the information that we can not approve this as this would open a way to install custom firmware on the device
and therefore leads to an FCC violation. Which in turn means that we do not approve this method. So if you're having issues it is best to contact
Support first.
Regards,
Cheerok
Hello Robin,
Thank you for your explanation, I've just revived my AP55 from a crashed firmware upgrade!
Cordialement,
Thierry
Now I have a AP15 with a broken firmware. The recovery program is not compatible with this AP.
So maybe in the next few days, I'm writing a manual to recover this AP through the serial-console-connection on the motherboard of the AP15. You must open the AP for this and use a adpater like this: https://www.amazon.de/gp/product/B0757FQ5CX/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1
to make a serial connection to your computer and flash the firmware at the same way as the AP55.
At the moment only one YouTube-Video exists to recover the AP15, but in this case he unsolder the EEPROM to flash the firmware. But this way is much too complex for all.
https://www.youtube.com/watch?v=xyvU6sALPQA&t=316s
Maybe with this manual it is possible to recover all Sophos APs that are don't covered by the original provided tool from Sophos. - as long as the have a serial-connection on the motherboard. I don't know if all models have a pin-out, maybe at a few models you must soldering a pin-bar to acces the serial-connection. But the AP15 has this bar.
And when you can't recover the AP through this, you have the option of JTAG - but this is very complex. The AP15 has a JTAG-header.
With the tool from Sophos you can only recover AP 10 / 30 / 50.
Most of my information, I get from: https://www.dd-wrt.com/wiki/index.php/Serial_Recovery
So thanks to DDWRT for the good informations!
So, stay tuned!
Greetings,
Robin
In reply to RobinDobbermann:
Hello everybody,
I have made a video, how to flash the Sophos AP 15 (and maybe others).
And after this video, you follow my guide above and you´re Sophos AP is ready to go.
So, good luck and I hope to hear from you!
www.youtube.com/watch
My AP55 asking for username/password now.
any idea?
In reply to Usbkey:
It can't asking you for a password or username, when it boots up, it shows you the bootloader U-Boot and boot the image.. When and where should the Sophos AP ask you this?
When you can, post a picture.
BTW:
Recovery for the AP15 was succesfull! Easy as the AP55.
When it fully boot up.
I was trying to modify the allowed channels to utiliza 80Mhz channels but coudl not login.
The login prompt is simple just like below.
OpenWrt:>
Okay, but this what you tryin to make is not covered by my guide.. Your AP boots up and you want to modify settings in OpenWRT. I show the Receovery of a AP, and not how to modify the AP without the UTM.
understood, just want to see if you know the password.
According to Sophos engineers, all Sophos AP only use 40Mhz channels for enterprise deployment so I could not archive 866Mbps connection speed with both AP55/AP55C
There is a file allowed_channels at /tmp once you login and I am trying to add new channels to that file.
now you know what I am trying to do.
Thanks
Thanks! Worked exactly like your description!
Thanks so much! This worked for me too!
I didn't used the recommended TFTP Server because the DHCP Server Tool brings a TFTP Server too.
Because I have had some Problems to bring my DHCP Server to live here the configuration of my ini-file:
[GENERAL]SUBNETMASK=255.255.255.0ROUTER_0=192.168.99.1DNS_0=192.168.99.1
[SETTINGS]IPBIND_0=192.168.99.8IPPOOL_0=192.168.99.10-50
[TFTP-SETTINGS]EnableTFTP=1ROOT=D:\red-ap_recoveryapp\dhcpsrv2.5.2\wwwrootWritePermission=0
[HTTP-SETTINGS]EnableHTTP=1ROOT=D:\red-ap_recoveryapp\dhcpsrv2.5.2\wwwroot