Failed to get SSL certificate | Cannot verify peer's SSL certificate, unknown CA | Caught Empty IOR string from iiopAddressesInIOR

Hello, I have a couple of servers which on the one hand have Sophos AV fully working, but on the other one they cannot be seen in SEC (Sophos Enterprise Console). After some investigations I found in logs this:

28.06.2018 11:58:47 1E04 W Failed to get certificate, retrying in 600 seconds
28.06.2018 12:08:47 1E04 I Getting parent router IOR from 10.183.173.88:8192
28.06.2018 12:08:47 1E04 I Getting a new router certificate...
28.06.2018 12:09:29 1E04 E Router::GetCertificate: Caught CORBA system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO

28.06.2018 12:20:11 1E04 W Failed to get certificate, retrying in 600 seconds
28.06.2018 12:30:11 1E04 I Getting parent router IOR from 10.183.173.88:8192
28.06.2018 12:30:11 1E04 I Getting a new router certificate...
28.06.2018 12:32:39 1E04 W SSL connection alert, peer address 10.183.173.88
28.06.2018 12:32:39 1E04 W Cannot verify peer's SSL certificate, unknown CA
28.06.2018 12:32:39 1E04 E Router::ReportInvalidCertificate: Caught Empty IOR string from iiopAddressesInIOR
28.06.2018 12:32:39 1E04 I This computer is part of the domain EU
28.06.2018 12:32:39 1E04 E ACE_SSL (7964|7684) error code: 336134278 - error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
28.06.2018 12:33:00 1E04 E Router::GetCertificate: Caught CORBA system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO

Then I read through several articles and forums which raised some questions for which I couldnt have been able to find answers yet.

1.) How are "ParentAddress" and "ParentPort" (found in HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Sophos\Messaging System\Router) related to the IOR port?

2.) What is "http://www2.parc.com/istl/projects/ILU/parseIOR/" used for? I tried to get there IOR but then the page said "Your IOR is misformed. It must begin with either "IOR:" or "IOR2:", and then have an even number of hex digits." It seems as if the IOR wasnt correct.

3.) There are "pkc" and "pkp" missing under HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Sophos\Remote Management System\ManagementAgent\Private which should be crucial for a server . How I can get "pkc" and "pkp" back? (There is also no NotifyClientUpdate infound in HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Sophos\Messaging System\Router)

4.) All the necessary ports (80,8192,8194) are opened. I can telnet the destination point without a problem. IOR is shown when telneting 8192. How come the router catching empty IOR string then?

5.) Finally. How can it be this whole issue fixed? I won't be able to do it without someone's help.

I would be realy greatful if someone knew what to do, because I have already ran out of all ideas. Thank you.

  • Hi Maros,

    I did some digging and reached out to expert team members regarding your questions. This is the info I was able to gather:

    1)ParentAddress=Address of the Console Server
    ParentPort=Port 8192

    2)ParseIOR=We use this to parse the IOR, which is encoded routing information that is sent when a Sophos-protected machine is contacted over port 8192. We use that information to establish a connection over port 8194. The format goes "IOR:21654894564651ewe84561e89w7r9we84rwe561rwe897re98r4ew32r1we31r"

    3) pkc/pkp keys - These are the certificates that the client receives from the Console Server. They will only exist if the RMS handshake went through.

    4) If you're getting "certificate verify failed" it's possible the cac.pem or the mrinit.conf file keys are wrong.

    5) Please follow this article to fix your issue (NOTE: Those steps will help you create a script on the Console server, but do not run it there, instead run it on the Endpoint).

    5a) Additionally, if problems persists, you could use this article to test connectivity over por 8194 with openssl.

    Please let me know if this information answers your questions, or if additional assistance is required.

    Regards,

  • In reply to Barb@Sophos:

    Hi Borb, thank you for your great answer! But unfortunately it seems the created script hasnt done anything. Or maybe i did something wrong.

    The RMS section is OK i think, there is nothing to it. I found the paths and checked "RMS required".

    The Sophos Patch Agent section seems to be a bit confusing. Is Management server address an adress of the Sophos relay which my endpoint server connects to? If yes, I have it OK. I didnt change Management server port since i think 80 should be good. I used the script with Force configuration checked and unchecked but the result were same.

    Finally, the last section Script configuration options section. I didnt toucht it at all.

    The result is that neither cac.pem nor mrinit.conf had any data changed, so it seems as if the script didnt do anything.

    Did i use the tool correctly?

    This is how ClientMRInit.log looks like. As if something was really happening there during the time when the script ran.

    03.04.2018 13:57:17 F244 I SOF: C:\WINDOWS\TEMP/ClientMRInit-20180403-115717.log
    03.04.2018 13:57:17 F244 D ClientMRInit installing
    03.04.2018 13:57:17 F244 D mrfile=`MRInit.conf`
    cafile=`cac.pem`
    filepath=`C:\Program Files (x86)\Sophos\Remote Management System\`
    rtrname=`Router`
    logpath=`C:\WINDOWS\TEMP`
    03.04.2018 13:57:17 F244 I Opening initialisation file: C:\Program Files (x86)\Sophos\Remote Management System/MRInit.conf
    03.04.2018 13:57:17 F244 I Opening root certificate initialisation file: C:\Program Files (x86)\Sophos\Remote Management System/cac.pem
    03.04.2018 13:57:17 F244 I Intelligent updating is: Off
    03.04.2018 13:57:17 F244 E MRInitData failed with exception: CAccessFailureException:CACertificate not found
    03.04.2018 13:57:17 F244 D Old certificate not present, using new.
    03.04.2018 13:57:17 F244 T New Message Router identity key is present.
    03.04.2018 13:57:17 F244 T New Managed Application identity key is present.
    03.04.2018 13:57:17 F244 T New Management Agent identity key is present.
    03.04.2018 13:57:17 F244 D CheckParentAddress( `*** NOT SET ***`->`10.128.99.125,NAHVMMS03.am.boehringer.com` )
    03.04.2018 13:57:17 F244 D IsThisComputer[10.128.99.125,NAHVMMS03.am.boehringer.com]
    03.04.2018 13:57:17 F244 D Found 4 addresses
    03.04.2018 13:57:17 F244 D Just use new parent
    03.04.2018 13:57:17 F244 I Parent router IOR port: 8192
    03.04.2018 13:57:17 F244 I New router IOR port: 8192
    03.04.2018 13:57:17 F244 I Setting router service arguments: "-ORBListenEndpoints iiop://:8193/ssl_port=8194"
    03.04.2018 13:57:17 F244 I ClientMRInit successful exit

    Interestingly, the same log on another server which have the same problem looks really different.

    25.10.2017 06:34:15 07F8 I SOF: C:\WINDOWS\TEMP/ClientMRInit-20171025-103415.log
    25.10.2017 06:34:15 07F8 D ClientMRInit updating
    25.10.2017 06:34:15 07F8 D mrfile=`MRInit.conf`
    cafile=`cac.pem`
    filepath=`C:\Program Files (x86)\Sophos\Remote Management System\`
    rtrname=`Router`
    logpath=`C:\WINDOWS\TEMP`
    25.10.2017 06:34:15 07F8 I Opening initialisation file: C:\Program Files (x86)\Sophos\Remote Management System/MRInit.conf
    25.10.2017 06:34:15 07F8 I Opening root certificate initialisation file: C:\Program Files (x86)\Sophos\Remote Management System/cac.pem
    25.10.2017 06:34:15 07F8 I Intelligent updating is: Off
    25.10.2017 06:34:15 07F8 D CA certificates are the same, no action taken.
    25.10.2017 06:34:15 07F8 I Message Router identity keys match.
    25.10.2017 06:34:15 07F8 I Managed Application identity keys match.
    25.10.2017 06:34:15 07F8 I Management Agent identity keys match.
    25.10.2017 06:34:15 07F8 D CheckParentAddress( `10.128.99.126,NAHVMMS04.am.boehringer.com`->`10.183.173.5,inhas60989.eu.boehringer.com` )
    25.10.2017 06:34:15 07F8 D IsThisComputer[10.183.173.5,inhas60989.eu.boehringer.com]
    25.10.2017 06:34:15 07F8 D Found 3 addresses
    25.10.2017 06:34:15 07F8 I Connection cache size for endpoint will be set to 10 , NumSenderThreads will be set to 3, NumORBThreads will be set to 4
    25.10.2017 06:34:15 07F8 I Parent router ports match, no action taken: 8192
    25.10.2017 06:34:15 07F8 I Router IOR ports match, no action taken: 8192
    25.10.2017 06:34:15 07F8 D Router service args are the same (-ORBListenEndpoints iiop://:8193/ssl_port=8194), no change.
    25.10.2017 06:34:15 07F8 D Apply operating in update-only mode
    25.10.2017 06:34:15 07F8 I ClientMRInit successful exit

  • In reply to Maros Goc:

    Hi Maros,

    I got additional updates for you:

    Looks our KB needs to be updated (we will work on that): 

    The cac.pem and mrinit.conf should point to the following location:
    C:\Program Files (x86)\Sophos\Enterprise Console\   

    Also, Force Configuration should be selected for RMS,  and ignore the Patch agent part entirely.   

    Let me know if how it goes. 

    Regards,

  • In reply to Barb@Sophos:

    Thanks for the clarification Barb, but there is a problem. I have no Enterprise Console folder there. Maybe I forgot to mention that our machines have no common Windows installed but Windows Server 2008 R2 or Windows Server 2016. I dont know if it makes a difference for you but maybe it does.

  • Hello Maros Goc,

    first of all, you are using SESC so the Central forum is not the right one - please join the Endpoint Security and Control group and move your post there (I'd have moved it but I can't before you have joined this group).

    IOR is shown when telneting 8192
    using the
    10.183.173.88 address? You have to take the whole response starting with IOR: and paste it in the parse box - please note that if you copy it from the cmd window it's broken up into several lines and contains additional CRLFs and this is why the parser complains, so make it one single string first.

    Just saw your new post:
    I have no Enterprise Console folder there
    where's there? You should run the EMU (guess this is what you are referring to) on the management server (SEC) - this folder and the files are there.

    Christian

  • In reply to QC:

    Hi Christian,

    oh my god, what a silly am i... I have misinterpreted Barb's hint. I should have used the Migration utility on the relay and then the script on the endpoint. It seems I am a bit overworked lol. I will try it immediately.

    And I will move the topic to the place you suggested. And also yes, i have already noticed that copying an IOR from CMD creates gaps, so that is why it didnt work.

    Thank you very much :)

  • In reply to Maros Goc:

    Hello Maros Goc,

    oops, I see I've missed the reference to the relay - naturally it doesn't have the Enterprise Console folder.
    cac.pem should be the same on all machines, the mrinit.conf should be taken from the (configured) CID the endpoints will update from.

    Christian

  • In reply to QC:

    Hi,

    now I am completely confused. :D

    Barb said "Force Configuration should be selected for RMS", but in the manual  (https://community.sophos.com/kb/en-us/116737) there is "WARNING: This can damage the SEC Server or a Relay if run with this option. This script should not be run with force on either of these two servers as it can cause high amounts of damage."

    You said that cac.pem should be the same on all machines and the mrinit.conf should be taken from the CID. So it means that i dont have to create a script and what I should do is just to copy the mrinit.conf to the endpoint?

    But, if you say I should create the script, then i have a question: What is a "Management server adress"? Is it an adress of my endpoint or an adress of the relay where I am running the Migration utility?

  • In reply to Maros Goc:

    Hello Maros Goc,

    I'm not sure I understand your setup and what it is you think you have to do.

    You have the management server, the console works fine. It is protected (i.e. the Endpoint software installed). Are there already other endpoints where Sophos is installed and do they communicate as expected? I think so.

    You say Sophos is installed a couple of servers but they can't communicate. Where does the relay come into play here, why do you (think you) need it? Is 10.183.173.88 your management server or some other machine?
    You don't need the script if you add a relay to an existing setup, normally you need it if you install a completely new management server (or migrate the existing one) and can't or don't want to use Protect Computers or to reinstall the Endpoint software on your endpoints.

    Christian
    P.S.: please take the time to join the SESC group

  • In reply to QC:

    Christian,

    Sophos AV works fine on those servers, but they dont appear in the Sophos Enterprise Console and i have to find out why and fix it. I have been told that the issue can be caused by incorrect mrinit.conf and cac.pem and Barb suggested to use the Migration utility to fix the problem. And now I am collecting info how to use the utility the correct way (hoping it will fix the issue).

    That is basically all what I am doing.

  • In reply to Maros Goc:

    Hello Maros Goc,

    I have been told by whom? that the issue can be caused can or is - it's quite dangerous to apply a fix for a potential cause that's not applicable because the cause is another one (pardon the puns).

    Taking a closer look at your logs:
    In the original post the endpoint tries to connect to
    10.183.173.88 from where it allegedly receives an invalid IOR. What's the role of 10.183.173.88 - management server, relay or something else?

    The two ClientMRInit logs show:
    the first endpoint has not yet a Parent Address set, from the mrinit.conf it gets an IP and a FQDN: 10.128.99.125,NAHVMMS03.am.boehringer.com
    the second one had 10.128.99.126,NAHVMMS04.am.boehringer.com but from the new mrinit.conf it sets 10.183.173.5,inhas60989.eu.boehringer.com

    Three different addresses. Apparently relays are in use, naturally you have to use an mrinit.conf that has a correct relay (Parent Address) address. Another important point is the method used to apply the relay configuration. Usually this is done as described in section 1.2 in Configuring Message Relay Computers. If it has been done this way and the endpoint updates from another CID with a different configuration changes applied with the ReInit.vbs will get reverted. 

    So: You can use the EMU if an incorrect Parent Address is the cause of the communication problems, and you use the correct mrinit.conf, and the incorrect address hasn't been set because the endpoint is updating from a wrong (configured with an inappropriate mrinit.conf) CID. 

    Christian

  • In reply to QC:

    Hi,

    10.183.173.88 is an IP of the relay which one of the servers connects to during the installation process or when updating.

    How can i set a ParentAdress for the first endpoint?

  • In reply to Maros Goc:

    Hello Maros Goc,

    then the address in your first post is the correct one.

    Hm, looking again at the initial log - the Caught Empty IOR string might be a red herring as the preceding line says: Cannot verify peer's SSL certificate, unknown CA.
    Are other endpoints successfully using this relay?

    Christian

  • In reply to QC:

    It is a common relay used by many servers with Sophos AV. Yes they work with the relay without a problem. I am curious why "it" catches empty IOR string when during telneting 8192 the IOR string is clearly shown.

  • In reply to Maros Goc:

    Hello Maros Goc,

    I don't know the logic and workflow involved, as said I think the Caught Empty IOR string is a red herring: The IOR is parsed, the addresses (there can be more than one) are tried, and if a connection succeeds the IOR is logged. If all SSL connection attempts fail the IOR is considered empty - while actually it's an SSL error.

    As other endpoints works I'd reinstall (just install over the existing installation) the Sophos Endpoint software on one of the servers (BTW - which Windows Server version?) This should reset it to a state that is known to work on others. If the error is still there it should immediately be noticeable.

    Christian