Upgrading? Need advice? Let us know!

Hi Lil,

 That's a brave offer!

 I had a lot of problems upgrading from v3.0 to v4.0. I hasten to add that none of them were Sophos's fault!

Not only do we use a separate SQL server to host the database but  I also customised our setup to link different Active directory groups to the "Sophos DB Admins" role to deal with local jurisdictions.

 Yep, I really was asking for problems :smileyvery-happy:

This isn't a big issue at all (I recognise that my circumstance is a very, very unusual scenario) but I noticed that the installer didn't actually give any warnings when the SQL scripts to create the new upgraded database failed.  It took me a while to put the pieces together in my head :smileywink:. (or more accurately to remember I'd tweaked the AD groups, Doh!)

 With 20:20 hindsight it might be good to cater for those scripts failing and giving a diagnostic message. This isn't for fools like me (Although it might have saved me a few minutes :smileywink:). It's based on my general distrust for SQL Server/Express. It can be troublesome sometimes and having seen what the new console can do I feel it really deserves not be victim to Microsoft-isms.

 I'm trying to pitch this fairly. If someone wants to use an external SQL server than that's great that you support them. If they play around with it or 'tweak' their systems then it would be nice for you (and the more adventurous souls) to have a diagnostic message rather than have us both get confused and adding some bad tech. support requests :smileyhappy:

 That was a bit long! Sorry! It's just a thought!

Keith

Hi Keith,

In order to fully understand your experience during the upgrade, would you be able to clarify a couple of point to help us improve the process for the future?

When you first ran the installer on the database machine to create the new SOPHOS4 database in your SQL instance, did this succeed? Do you know if the new SOPHOS4 database was created ok at this stage?

When you then ran the installer on the management server to upgrade that component, did the Sophos Management Service fail to start due to:

1. the absence of the SOPHOS4 database, as a result of the first step failing?
2. The custom security permissions?

We'd like to understand whether the problem was either:

• the inability to create the SOPHOS4 database (in which case you should find an error in “%programfiles%\Sophos\Enterprise Console\DB\InstallDB.log”),
• the migration of data from the SOPHOS3 database to the newly created SOPHOS4 database
• or the failure of the management service to start due to incorrect permissions? 

Thanks,

Lil

Hello,

I just have to take you up on such a generous offer!

I need to move an Enterprise console 3.0.0.2321 from an existing 2003 x64 server to a new 2008 x64 server. I'm seeing a couple of useul guides, one for migration from x86 to x64, and one for migration between 32-bit servers. I have a couple of questions first, though:

- Do I need to upgrade the existing enterprise console to a later V3 build before I can migrate to V4?

- Can I use the 32-bit to 32-bit migration guide in a full 64-bit environment, or is there another document for this  ( http://www.sophos.com/support/knowledgebase/article/28276.html )?

Thank you for your support,

Hi,

No problems, we'll be sure to update our kba to reflect the upgrade of 64-bit servers.

In the meantime, please do follow article 28276 (please note that v4 is not supported on Windows Server 2008 R2); however, on a 64-bit installation: 

• the path to the Sophos programs is: C:\Program Files (x86)\Sophos
• the registry keys on 64-bit servers can be found at: "HKEY_LOCAL_MACHINE\Software\WOW6432Node\Sophos...".

There is no need to upgrade the existing Enterprise Console, however, because the new server is running Windows Server 2008, you will need to install Enterprise Console version 3.1.1 on it.

I would also advise that you disconnect the new server until all of the steps have been performed. If that is not possible, we recommend that you disconnect the old server once you have installed Enterprise Console 3.1.1 on the new server. Probably best to do it when you don't need access to the console for a while :).

Please also ensure that you take note of the following points before upgrading:

• These instructions apply only to installations that used the default settings. In particular, it does not apply to installations where the database and/or the management service are located on a different server to Enterprise Console.
• The hostname and IP address of the new server MUST match the original server’’’’s hostname and IP, otherwise you will have to redeploy the client software to your endpoints.
• The target server should be in the same domain as the existing one is/was.
• Ensure that you are running only one Console on the network at a time, otherwise you will experience problems.
• If you are migrating to a server running Windows Server 2008, please be aware that you will need to use a message relay for network communications if you have more than 5,000 endpoints. See Enterprise Console: configuring message relay computers for more information.
• Before you start, read the Warning about editing the windows registry.

Hope it all goes well!

Cheers,

Lil

Hmmm, upgrade process worked from v.3 to v.4 no real problems at all. Migrate of Lib Manager never worked and still doesn't. I got bland messages about the failure and submitted these to support who drew a blank. Ended up simply recreating the whole updating config from scratch. Updates used to occupy about 250Mb's of space for the entire warehouse. Now looking at about 4.3GB's because of all the CID variations I have need there own download warehouse. Not very pretty now and hideous when an update occurs. Kiss goodbye to the server for 20 minutes during an update :-( it's nearly always full 100% CPU everyday all day 24/7 from SophosUpdateMgr process. Had to knock off several CID's and roll back to Lib Manager to restore some process sanity.

I'd really like to see the SUM rewritten from the ground up thinking properly about CID distribution. No amount of manuals are going to help until that's done properly. Funny though 'cause v.3 was actually much more sensible at this even if it was cludgy.

Matt

Hi Matt,

What a great word -- cludgy!!

Seriously though, our developers would love to get more info from you about your environment -- both to understand why the migration failed and also to understand why SUM's choking the server during an update -- this is not behaviour that we've seen in our tests and we're taking it very seriously.

Obviously, you may be wary of publishing some of the details of your installation to the forums, so if you could switch on private messaging and mail me the following info, we can get someone from the SUM team to look at this:

• Support ticket number for your failed migration, so we can check the details and try to improve the migration process.

• How the SUM updating hierarchy is organised, e.g. one SUM or many SUMs in a hierarchy or many SUMs connecting to Sophos.

• Which products are subscribed

• How many CIDs are being distributed and to where (local machine or remote machine)

• What was included in the quoted figure of 250Mb, e.g. only the CIDs (all of them? Which products?)

• What is included in the quoted figure of 4.3Gb, e.g. only the SUM share, the remotely deployed CIDs, or the Sophos product set as a whole?

• What’’’’s the machine spec? (This would be useful to know for performance reasons, as the post says it’’’’s at 100% CPU for 20 minutes, so we could combine that with info on the number of CIDs being pushed)

If you need help switching on private messaging, please let me know!

Best regards,

Lil

Hi Matt,

As lilhavoc mentioned, we'd like to understand what your configuration is that is causing the performance problems you are seeing.

In the distribution tab of the SUM configuration, you can configure which subscriptions (which contain CIDs) are distributed to which locations. However, SUM does maintain a default distribution which contains all of the subscriptions. If you have a large number of subscriptions this will take a while to update. Is this what you are seeing?

Cheers,

John Reynolds

Hi Lil,

Actually for the moment, I'd rather this all stays in the public domain to assist other users rather than hide it all in the private messaging. I'm stunned that your developers aren't aware of the problems, looks like support isn't feeding back information.

I think the migration failed because of the complexity of the CID's but recreating new updates isn't that much hassle so rather than spend days firing support questions and logs as I started to do and have call references that never closed which I can PM you, I just gave up and manually configured the updates myself, which I found much easier anyway as your support staff seemed very green on the new version back last year when I upgraded, than the upgrade process itself.

So let's look at why this is all so 'cludgy'......................

Firstly, let's look closely at v.3 library manager and see exactly how that worked. I had a subscription to the databank and I simply ticked all the packages I want to download. In my case, this is win 9x, NT, x86, x64, Linux libc6, MacOSX . 6 Packages that gave a download warehouse of 250Mb's (approx) Can jump as high as 350Mb's during the month.  From this single download, I can then distribute CID's to my domains. Some CID's have just 9x and x86, others may have 9x, x86, NT and so on. There are 8 combinations but only one download is required from Sophos databanks. Each CID is copied out from the LibMan server to the CID locations across UNC paths (most are local 100 base LAN but there are 3 distributed across slower WAN links). During the copy process, DLLLoader.exe executes and checksums the CID folder before integrating the updated files. DLLLoader puts reasonable pressure on the server but although heavy is not overpowering (there do seem to be some issues with this process though as there is noticeably more CPU load on slower links which means the process is not threading correctly and chewing CPU - probably due to poor wait loops).

Now let's look at v.4 Update manager. Hmm, first problem; I no longer have the ability to specify the packages I want in each CID so I now have to create a subscription to match the packages for each CID. Each subscription downloads it's own warehouse of packages. So in my 8 combinations, I now download x86, 9x for every warehouse and then some with NT, some with NT+linux, some with NT+linux+MAC, some with NT+MAC etc. Giving 8 warehouses. Total download size 4.3GB's (yep, v.9 EPS is much larger than v.7 SAV - almost double by my calc). Remember I said that Dllloader.exe was pretty heavy on CPU. Well SophosUpdateMgr.exe makes that look stupid, it chews CPU like no other process invented. Take a look for yourself, look at task manager and see the amount of CPU used (note, CPU usage, not RAM - your support took a while to cotton on to that). Now add a second warehouse (or subscription) and look again at the CPU during and update. Add another and look again.

Life is so bad with EMC v.4 that I've rolled back to lib manager v.3 and reduced the warehouse count on lib manager to just 2 and dedicated an entire server to this process now. With 2 subscriptions, my server spends ~11 minutes of each 15 minutes between update cycles (your default) servicing 2 CID's. 1 CID contains 9x, x86, Linux and NT and a second just x86. If I add a 3rd subscription, the server is flat out 24/7/365 100% CPU dedicated to SophosLibraryMgr.exe. The two remaining CID's are local 100base LAN connections so no slow WAN involved - dread to think what that would be like. Support have told me to get a faster server to fix this. What I find astonishing is that I can run an MD5 checksum across a Sophos warehouse in around 30-50 seconds on this server whereas UpdateManager seems to chew the disk for around 30 seconds then spend the next 10 minutes or more eating CPU with little or no disk activity.

There are so many issues with the new update manager that I really feel it's unsuitable and not fit for purpose. E.g. take a simple feature I relied on. When I get an update on LibMan v.3, I get an email. The email tells me that I've had xyz.ide update. With Update manager, I get a nice notice in EM Console saying my warehouse last updated but I haven't a clue what got updated and if it was a minor or major update. If I submit a sample to Sophos, I want to know exactly when I'm covered by Sophos. The emails I get back from Labs say that xyz.ide will go up on the databank. But now I do not know when that has happened. LibMan v.3 emailed me, update manager v.4 doesn't tell me toffee!

I would absolutely love direct contact with your developers and cut out the Chinese whispers from support to labs. Let's get some real feedback into the hands of those that write the package and make progress getting Sophos sitting on top again. I've been a subscriber of Sophos for in excess of 12 years now watching it grow from v.1 SAV through v.9 EPS and never been as disappointed in a product as much as this latest release. Still, staying on topic here, lets fix Update Manager first.

Matt

Thanks so much for your rich feedback Matt.

At the moment, all I can suggest is that Support haven't heard this feedback a lot, perhaps because most people are using SUM to download one or two packages -- maybe most of them are on the recommended version of Windows package only? That's just a guess, but I will look into it further. And of course, I'm sure anyone else who is affected will now shower us with replies!  Please come out of the woodwork now, if you are!

I know that development are currently poring over your post (John R is a fellow Sophos employee), and will definitely be getting in touch to discuss this further.

Finally, may I please have your ticket number so that I can check into the case and see what areas of knowledge we need to fill in for support agents?

Many kind regards,

Lil