EAP2 noticeable improvements and observations

I think the memory usage will creep back up as Big_Buck commented above. I do however see a huge improvement in load average.  

I am usually pretty harsh on sophos for not listening but someone actually listened and tuned the load average. Since v15, I have never seen XG running like a normal linux box with low load averages when the firewall is sitting there idle. Bravo to the developers and PM who actually listened to our feedback and made an effort to make things better.

Regards

Bill

But.  It does not mean its bad for so much anyway.  Logic dictates OS flushes it only when necessary.  Or maybe.  What I am most concerned is the turn around.  If the OS dumps & reads memory on disk constantly, then, THAT means there's not enough.  What I see here, is that it fills slowly over many days.  Which suggest it fills because the OS (or applications) does not detect any flush is necessary.  Much like a teenager with his bed-room. I'll clean it only when I won't be able to close the door anymore.

Paul Jr

But.  It does not mean its bad for so much anyway.  Logic dictates OS flushes it only when necessary.  Or maybe.  What I am most concerned is the turn around.  If the OS dumps & reads memory on disk constantly, then, THAT means there's not enough.  What I see here, is that it fills slowly over many days.  Which suggest it fills because the OS (or applications) does not detect any flush is necessary.  Much like a teenager with his bed-room. I'll clean it only when I won't be able to close the door anymore.

Paul Jr

An update on the the graphs - memory and load. There is no noticeable increase in memory even after adding 2 more clienteless users in IPv6. Further updating and adding mail rules to allow for the MBP certificate issue with one of my mail accounts.

What drives memory usage up is having the GUI open. 

Ian

EAP2 has been running for a couple of days now.

1/. ram usage is still down - 60% down form over 70%

2/. load is still down, way below 17.5.9 values

3/. categories are not being identified

4/. report summary missing values.

5/. GUI much more responsive.

6/.I have email scanning working for one mail account even on the iPad.

7/. mail still throwing up odd mail address info, same as 17.5.x

8/. mail still has trouble determining how many times a message should be scanned and reported on

9/. I am running IP4 and IPv6 on all interfaces - will be the subject of another thread.

Ian

Memory has been quite here too ...  CPU down as well.

Paul Jr

Hi Ian, by any chance have you enabled DPI rules, application control and IPS? I haven't rolled my own snort daemon in a long time but usually the more traffic you process via snort, the more cpu and memory gets used.

You can use memcap in snort so that snort doesn't go above a certain amount of ram but in a regular IDP/IPS snort drops packets when it gets overwhelmed. That is by design and to process more traffic, you can add more cpu and ram to process more traffic. Here is the basic layout

1. OpenAppID (application detection) doesn't need all the intrusion protection signatures so can run easily with low memory and very low cpu usage. Usually no packets are dropped and there is enough memory and cpu power to ID the apps.

2. Intrustion protection takes a lot of ram and processing power. That is why we use only a limited amount of signatures. Also the quality of signature matters when processing intrusion protection rules. You can drop a few packets here and there if the traffic gets overwhelming but if sized correctly, this is not a problem. Memory is limited in a XG type device but newer processors usually do fairly well.

3. Now we come to DPI, its running its own set of signatures? I am not sure how it is implemented as I haven't had time to look into it deeply. But you pretty much cannot drop any packets and the memory is limited in a XG type device so the throughput will depend on how fast is the processor.

Now run all 3 tasks above at the same time. I haven't done any testing with everything running at the same time but for snort to process all this traffic at the same time without choking will be interesting to see.

Regards

Bill

Hi Bill,

at this stage I am only running DPI in both IP4 and IPv6. I do find the functionality limited when compared to the WEB Proxy, though still fine tuning which rules I can use in Application and DPI.

The DPI does not use the exception list setup for the web proxy, it has its own exception list which will need to be updated when v18 goes GA to restore the scanning of all the failed sites.

Again having two different lists of exceptions to manage is not a well though out process.

Ian

Hi folks,

memory usage is slowly creeping up, after 3 days now showing 64%.

Daily reports are almost as if the reporting system is reporting on the day before, not yesterday because the categorisation is now working or something changed in a lookup table.

I click on unclassified in the GUI which shows 30, but the expanded view shows 124 unclassified technology items.

Ian

Any major show stoppers or stability issues with EAP2 that you wouldn’t recommend using it for a home environment? I’d like to help test v18, but it would be on my home network in place of my current v17.

Hi Shred,

I haven't tried my full home network since EAP1. I am only using DPI in my test network

EAP2 appears to be stable and I suspect most of my issues are to do with my ISP changing something in their mail servers overnight and Sophos Home Premium blocking sites.

My test network does not have any interLAN rules or VLAN.

If you use MAC or iPads or iPhones mail scanning for imaps is still problematic. I was able to get Imap mail working but my ISP broke something overnight and now devices that did work no longer work. I Have some certificate errors, my created issue which is proving a pain to fix.

Ian

I just upgraded to EAP2 on my home network. This is running on a Qotom Q355G4: Intel i5-5250U (AES-NI) w/ 4 GB RAM. CPU utilization is hanging around 3-6% but memory utilization is ~90%. Previously on V17, I was seeing about 65% memory utilization so this seems like a fairly significant jump. I even tried reducing the amount of signatures on my IPS policies (removed ~2,000 signatures) but the memory usage remains the same. I also disabled all 'Content scanning' and 'Web proxy' options in my firewall rules (I was running Decrypt & Scan HTTPS in v17) but still the same memory utilization. My home network consists of 11 Apple devices (iMac, iPhones, iPads, Watches, HomePod, Apple TVs) and about 30+ other random "IOT" type devices (smart home devices, smart TVs, sensors, printers, cameras, picture frames , thermostats, raspberry pis, etc.).

First question:

I noticed all of my firewall rules that were migrated had a "FW#X_migrated_NAT_Rule" as well for MASQ. I simply cloned each firewall rule and deleted the original one to get rid of all the migrated NAT rules, then created a NAT rule:

• Original source: Any
  • Translated source: MASQ
• Original destination: Any
• Original service: Any
• Inbound interface: Port1 (LAN)
• Outbound interface: Port2 (WAN)

I did that for IPv6 as well as it appears MASQ is still required.

Is that an "acceptable" way to setup the MASQ rule for a home network? Seems to work fine, but just want to make sure I'm not screwing anything up.

Second question:

On v17, I had a firewall rule for Any source/destination/service that scanned IMAP, IMAPS, POP3, POP3S, SMTP and SMTPS since I'm running the Email service as a transparent proxy (not MTA mode). I noticed now in v18, you can select those email protocols within any firewall rule so I deleted the rule I had previously and simply selected everything under 'Scan email content' for the firewall rule my devices with email utilize (also, added the appropriate email services when prompt). Is that the proper method for setting up email scanning in v18?

Overall, the web GUI seems to load pages quicker than v17 and everything seems to be working fine so far. I'm concerned about the huge jump in memory utilization, especially since I'm only ~10% from being maxed out. I'll continue to run it for a bit without any SSL/TLS inspection rules to make sure everything runs fine then start incorporating some SSL/TLS inspection.

Hi Shred,

the NAT thing is a pain, I left them as was at the time.

Your mail setup is similar to mine, but only used IMAPS and SMTP/S.

Memory usage has increased significantly and from various reports within the EAP forum a 4gb system will struggle and with as many devices that you have you will continue to struggle.

My current memory usage is around 65% on  6gb system with 6 active addresses (3 clients) and 11 firewall rules (6 IP4 and 5 IPv6) and two NAT rules using DPI.

IPv6 still requiring a NAT is a failure to understand by Sophos dev teams. Yes, NAT on IPv6 is an option but should not be mandatory.

Ian