KnowBe4 RanSim - CryptoGuard didn't stop InsideCryptor

Question

I recently run KnowBe4's ransomware simulator on my desktop. Whilst it appears that Sophos did better than most of their competitors it still missed InsideCryptor. I'm looking for an explanation on how this could happen. Is it a configuration problem? An issue with the simulator or did CryptoGuard drop the ball and Sophos is working on this?

This thread was automatically locked due to age.

PeterM · Answer

Hi everyone, 
 The RanSim tool is not a realistic test, while some of what it includes does match real world threats they have also invented some new techniques that create misleading and inconsistent results (as demonstrated by the comments above) and importnatly aren't seen in actual malware. Most AV vendors create detections specifically for the RanSim tool even if what they are doing wouldn't even be considered malicious in the real world, this makes the results from this tool relatively pointless as it doesn't accurately reflect how good a security product is against real world threats. 
 Testing AV products is a good idea, however you may find it a lot easier if you just read the reports from the independent 3rd parties that actually create the testing standards and are internationally accepted by the Security industry. 
 Our CTO published an article on this recently: https://news.sophos.com/en-us/2018/10/23/fair-rigorous-transparent-and-collaborative-cybersecurity-product-testing-is-good-for-customers-and-good-for-the-industry/ 
 Or you can skip to some recent 3rd party results that tested Sophos alongside other leading AV vendors: https://news.sophos.com/en-us/2019/07/01/more-fantastic-test-results-for-sophos-intercept-x/ 
 Hope this helps.

Badrobot · Answer

I have to apologize I believe I looked in the wrong folder the first time, the files for the 3 tests did get encrypted, not everything was reported in Central either, it was on the computer in terms of notifications or prompts as I was running the tests but not in the events tab on Sophos on the endpoint or in Central. I also had a little more time this afternoon to run more efficient tests so to speak, the pains of managing multiple IT support aspects again I do apologize. 
 So as stated above the test results from the Reflective Injector, Virlock Variant and Injector show as vulnerable and did have the files encrypted. What confuses me a bit is that as I was watching the tests run and I did run the tests three times to ensure no mistakes is the prompts Sophos Central throws up. 
 At one point the prompt shows- 
 
 Which shows the attack as seen as Generic Malware however this would have been the attack tested as the Virlock Variant. In this case the file has already been encrypted when the prompt shows, the attack also shows as Mal/Generic-S, which according to Sophos is "Mal/Generic-S is a name used by Sophos products when detecting a threat via the cloud using Sophos Live Protection." And is not listed as Ransomware. 
 MalGeneric-S Found Here: https://www.sophos.com/en-us/threat-center/threat-analyses/viruses-and-spyware/Mal~Generic-S/detailed-analysis.aspx 
 Now on the tests it did block and had no encrypted files in the folders Sophos would throw up a prompt such as &ldquo;Ransomware Blocked&rdquo; and the associated folder did not have encrypted files within, which is great because that is what should happen right? 
 Now add to this while I have been working on this reply and checking on things, I have noticed that Sophos is still working even though the tests have completed on the RamSim application. For example, a few minutes after the application completed. In which RanSim showed test folders 1, 10 & 14 as vulnerable and the files were encrypted. Sophos was able to revert 3 files in both 10 & 14 back to CSV files. When you look in the Sophos app and the events tab on the computer you can also see this as- 
 HPmal/RanSim-B partially cleaned up. 
 Now here is where things just don't add up, I ran the tests or overall RanSim application 3 times, I received 18 notification emails and 18 alerts in Sophos Central, all of which were detected as Ransomware, in each case it is the same 6 aspects that are detected. 
 There is no notification for the Mal/Generic-S which is odd because I see Sophos flag it as I showed in the above screen shot and did also see more than 6 pop up prompts during each test. This really does not add up since RanSim runs a total of 18 tests so I should be seeing 54 emails and 54 alerts in Sophos Central. That is unless some of these tests are being blocked by updates or patched software and as such Sophos does not have to do anything anyway. 
 So that is about it, I am going to dive into the Threat Analysis Center after this to see what I can learn about a Ransomware attack, but I wanted to add the above so I had a more accurate description of the testing I have seen between RanSim and Sophos. 
 A few other notes that I want to mention based on responses from RanSim and or Sophos- 
 I did see mention of hardware playing a factor so I would like to add that I am running a i7-6500U processor with 12GB of RAM and an SSD, I was not over utilizing any resources when I ran the test. 
 Sophos did respond to my support case as well on this, some of this would answer these questions so I felt I should add it to this post- 
 Regarding the Virlock test. This is an exploit that we've protected against since 2013. However, KnowBe4 does a variation of this exploit that's more of a simulation than an actual exploit. Since nothing is actually being exploited in Windows Intercept X is not triggered. However, we are working on detecting KnowBe4's specific test, and this will be released in an upcoming release. 
 Regarding the Injector and ReflectiveInjector vulnerabilities. The description appears to simply be an exploit know a Process Hollowing, which is another type of exploit we do protect against and have for a while. While I've not seen that particular vulnerability flag, after speaking to some of my colleagues in our Global Escalations team, we do believe it to be similar to the Virlock exploit test. 
 
 I know Sophos and Knowbe4 have both been a part of this debate and each side is weighing their opinion and I do appreciate the concerns from all parties regarding this and want to thank everyone for helping with this forum post! 
 If possible I am wondering 2 things- 
 One aspect I would like to know is one, is it possible to run these tests individually? And another is why is the reporting the on-device level not matching with the management system or Central? i.e. Why am I not seeing all the attempts in Sophos Central?

David Coombe · Answer

Looks like I get to answer my own question. 
 
 I spoke to a Sophos engineer and he said that CyberGuard actually did stop it and all the files were spared. KnowBe4's sim reported it wrong. They have reached out to KnowBe4 and are waiting for them to fix their simulator.

Simeon Lewis · Answer

I ran the same test today and found similar issues. We have endpoint protection and Intercept X. 
 The tool is called Ransim from KnowBe4 and simulates 16 types of ransomware. I'd be interested if this is a problem with the Ransim tool or if InterceptX isn't seeing them>

David Coombe · Answer

Hi Simeon, 
 A quick way to see if Sophos worked or not is to check to see if the files are encrypted or not. The path is on the right hand column.

Badrobot · Answer

I would like to add that I agree with PeterM, after running my own tests while the simulator was running I found that the simulator did not encrypt any files for the tests it claimed I was vulnerable to but also required administrator privileges to even run many aspects of the test which in many cases the average users does not even have. Sophos also flagged all 16 attacks in Sophos Central which again does not add up to the report I was getting. I did reach out to the reps at knowb4 regarding these results as well, thinking maybe this is something they were not aware of and asking if it is possible that they are false positives and or have they proven their app against Sophos security apps and the response I got was- 
 
 We have a support document that covers issues with false positives (towards the end), should be what you're looking for https://support.knowbe4.com/hc/en-us/articles/229040167-RanSim

Stu Sjouwerman · Answer

Here is some feedback from KnowBe4: The original poster, who originally noted InsideCryptor not being blocked, uses a very old version of Ransim (1.0.3.4). Since then, several things have changed in the code and those changes also reflect in the results. I would advise them to give it a try with the latest version. 
 The rest of the people did use the latest version in which InsideCryptor appear to be blocked, as that person from Support states (he probably used the latest version as well), but there are other scenarios that are not blocked (the support guy doesn't mention anything about them - he only says "it" referring to InsideCryptor). 
 About the inconsistencies, badrobot got slightly different results from Simeon Lewis, but the results are not that different. From the screenshots one can see that in both cases Injector, ReflectiveInjector and VirlockVariant are not blocked. On the Simeon's machine, the miner also ran fine and one false positive scenario was blocked. 
 We've seen such results form time to time with certain AVs, but it is not a problem in Ransim. The detection methodologies used by AV engines, the performance/processing and I/O load on the machine on which the test occurs play a major in the outcome of the test. To give you an example, we saw a few cases when Windows Defender flagged a particular scenario several minutes after the test was over and attempted to quarantine it, but the executable was already erased from disk and the files remained encrypted. Another time, on the same machine, the same Ransim setup, Windows Defender didn't complain at all on the same scenario. 
 In conclusion, based on the somewhat limited information on that thread, we don't really see an with Ransim. Moreover, as one of the participants said, everyone can have a look at the files (they can even use their own files) and see if they are encrypted or not after the test. 
 Hope this clarifies, Warm regards, Stu