ips.log filling up disk

Can you share a df -h output here?

And please an ls -lh /log/ 

Because generally speaking, the Log Rotation should hit here and override the Log File to prevent this. 

Thank you for your respone. Please be aware that I have removed ips.log file couple of hours ago because its size was 70GB+ and it caused /var partition usage become 100%. Also LoggingDaemon was stopped. Removing ips.log resolved these issues temporarily but since then I've stopped the IPS service. If I start it again, same log will be generated again and same issues will reiterate.

Here is the output you asked:

XG210_WP03_SFOS 19.5.4 MR-4-Build718# df -h

Filesystem                Size      Used Available Use% Mounted on

none                      1.5G      1.2M      1.4G   0% /

none                      3.8G    512.0K      3.8G   0% /dev

none                      3.8G      3.8G     60.0K 100% /tmp

none                      3.8G     14.6M      3.8G   0% /dev/shm

/dev/boot               126.2M     31.0M     92.5M  25% /boot

/dev/mapper/mountconf

                        954.9M     88.0M    863.0M   9% /conf

/dev/content             11.2G    487.0M     10.7G   4% /content

/dev/var                 87.0G     14.5G     72.5G  17% /var

XG210_WP03_SFOS 19.5.4 MR-4-Build718# ls -lh /log/

-rw-r--r--    1 root     0              0 Feb 26 18:34 VPN.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 WINGc.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 WINGu.log

-rw-r--r--    1 root     0          42.8M Mar  4 12:19 access_server.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 apache.log

-rw-r--r--    1 root     0          16.1M Mar  4 12:19 apache_access.log

-rw-r--r--    1 root     0              0 Feb 27 08:59 apiparser.log

-rw-r--r--    1 root     0          12.1K Feb 27 08:59 apiparser.log-20240227_085953.gz

-rw-r--r--    1 root     0            836 Mar  4 05:42 app-feedback.log

-rw-r--r--    1 root     0         614.9K Mar  4 11:47 appcached.log

-rw-r--r--    1 root     0          15.2M Mar  4 12:19 applog.log

-rw-r--r--    1 root     0             42 Mar  4 12:19 async_auth.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 atop.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 av.log

-rw-r--r--    1 root     0           2.3M Mar  4 10:01 avd.log

-rw-r--r--    1 root     0           1.2M Mar  4 12:18 awarrenhttp.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 awarrenhttp_access.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 awarrenmta.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 awarrenmta_debug.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 awarrensmtp.log

-rw-r--r--    1 root     0          10.2K Feb 26 19:57 awed.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 bad_mail_detector.log

-rw-r--r--    1 root     0           1.0K Mar  4 10:42 bgpd.log

-rw-r--r--    1 root     0            716 Feb 26 19:57 bwm.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 catUpdateLog

-rw-r--r--    1 root     0              0 Feb 26 18:34 catUpdateLog1

-rw-r--r--    1 root     0            375 Feb 26 19:59 catUpdateLog11

-rw-r--r--    1 root     0              0 Feb 26 18:34 centralmanagement.log

-rw-r--r--    1 root     0         282.3K Mar  4 12:16 charon.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 chromebook-sso-backend.log

-rw-r--r--    1 root     0            334 Feb 26 18:55 clientless_access.log

-rw-r--r--    1 root     0              0 Feb 26 19:56 confdbstatus.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 crreportdb.log

-rw-r--r--    1 root     0          59.9M Mar  4 12:19 csc.log

-rw-r--r--    1 root     0           8.5M Mar  3 00:00 cschelper.log

-rw-r--r--    1 root     0           6.3M Feb 28 00:00 cschelper.log-20240228_000015.gz

-rw-r--r--    1 root     0              0 Feb 26 18:34 csd.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 ctasd.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 ctipd.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 ctsyncd.log

-rw-r--r--    1 root     0           5.5K Feb 26 19:58 czt.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 dbcleanup.log

-rw-r--r--    1 root     0           1.9K Feb 27 09:04 ddc.log

-rw-r--r--    1 root     0           3.6M Mar  4 12:18 dgd.log

-rw-r--r--    1 root     0           3.9K Feb 26 19:45 dhcpd.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 dhcpd6.log

-rw-r--r--    1 root     0         111.6K Mar  4 12:15 dnsd.log

-rw-r--r--    1 root     0          79.8M Mar  4 12:19 dnsgrabber.log

-rw-r--r--    1 root     0         278.6K Mar  1 23:53 dnsgrabber.log-20240301_235302.gz

-rw-r--r--    1 root     0              0 Feb 26 18:57 dropbear.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 eacd.log

-rw-r--r--    1 root     0              0 Feb 26 18:33 entity.log

-rw-r--r--    1 root     0          40.0K Mar  4 08:44 error_log.log

-rw-r--r--    1 root     0           1.3K Feb 29 08:16 exim_mail_client.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 filesync.log

-rw-r--r--    1 root     0          37.7K Mar  4 10:50 firewall_rule.log

-rw-r--r--    1 root     0           4.3M Mar  4 12:18 fqdnd.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 fqdndebug.log

-rw-r--r--    1 root     0            148 Mar  2 03:00 fstrim.log

-rw-r--r--    1 root     0            920 Feb 26 19:57 ftpproxy.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 fwcm-eventd.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 fwcm-heartbeatd.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 fwcm-updaterd.log

-rw-r--r--    1 root     0          32.0K Mar  4 08:44 fwlog.log

-rw-r--r--    1 root     0          22.0K Mar  2 23:58 fwmgmt.log

-rw-r--r--    1 root     0           6.1M Mar  4 12:19 garner.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 ha_pair.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 ha_tunnel.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 hbtrust.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 heartbeatd.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 hostapd.log

-rw-r--r--    1 root     0              0 Feb 26 18:33 hotspotd.log

-rw-r--r--    1 root     0         417.9K Mar  4 11:50 httplogd.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 hwaccel.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 hwmon.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 iaasd.log

-rw-r--r--    1 root     0           2.3M Mar  4 10:51 ips.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 ipsec.log

drwxr-xr-x    2 root     0           4.0K Feb 26 18:57 ipsec_conn

-rw-r--r--    1 root     0           1.5K Feb 26 19:58 ipsec_monitor.log

-rw-r--r--    1 root     0          20.8M Mar  4 12:14 iview.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 l2tpd.log

-rw-r--r--    1 root     0          13.9K Feb 26 19:58 lcd.log

-rw-r--r--    1 root     0           6.9K Feb 27 00:00 legacyconversion.log

-rw-r--r--    1 root     0          28.0K Mar  3 19:58 licensing.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 logrotate.log

-rw-r--r--    1 root     0           5.9K Feb 26 19:57 mdev.log

-rw-r--r--    1 root     0           7.1K Feb 26 18:56 migration.log

-rw-r--r--    1 root     0          59.9K Feb 26 19:56 migrationhash.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 mrouting.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 msync.log

drwxr-xr-x    2 root     0           4.0K Feb 26 18:33 nSXLd

-rw-r--r--    1 root     0         243.2K Mar  4 07:57 nSXLd.log

-rw-r--r--    1 root     0          23.8K Mar  4 12:16 nasm.log

-rw-r--r--    1 root     0          15.8K Mar  3 17:22 nat_rule.log

-rw-r--r--    1 root     0         174.2K Mar  4 09:04 networkd.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 npu-startup.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 npu-startup.log.prev

-rw-r--r--    1 root     0              0 Feb 26 18:34 npu_syslog.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 ntpclient.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 oauth_sso_captive.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 oauth_sso_userportal.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 oauth_sso_webadmin.log

-rw-r--r--    1 root     0              0 Feb 26 19:58 openvpn-status.log

-rw-------    1 root     0              0 Mar  4 12:19 openvpn-status0.log

-rw-r--r--    1 root     0             72 Feb 26 19:55 ospf6d.log

-rw-r--r--    1 root     0            449 Feb 26 19:55 ospfd.log

-rw-r--r--    1 root     0           2.2M Mar  4 12:18 osquery.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 pimd.log

-rw-r--r--    1 root     0           4.5K Feb 29 12:00 pktcapd.log

-rw-r--r--    1 root     0           8.9M Mar  4 12:18 postgres.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 pptpvpn.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 radvd.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 raid.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 readapplicationapi.log

-rw-r--r--    1 root     0            373 Mar  4 12:18 readobject.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 red.log

drwxr-xr-x    2 root     0           4.0K Mar  1 10:42 redis

-rw-r--r--    1 root     0           4.1K Mar  3 16:04 regen_client_bundle.log

-rw-r--r--    1 root     0          61.6K Mar  4 08:50 reportdb.log

-rw-r--r--    1 root     0           3.5K Feb 26 20:08 reportmigration.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 reverseproxy.log

-rw-r--r--    1 root     0            550 Feb 26 18:55 ripd.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 routinghelper.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 sac-feedback.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 sandbox_reportd.log

-rw-r--r--    1 root     0           1.8K Feb 26 19:57 sandboxd.log

-rw-r--r--    1 root     0         152.0K Mar  3 18:17 sasi.log

-rw-------    1 root     0              0 Feb 26 18:33 sessiontbl.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 setup_vf_dpdk.log

-rw-r--r--    1 root     0           3.9K Feb 29 12:00 sig_upgrade.log

-rw-r--r--    1 root     0         165.3K Mar  4 12:18 sigdb.log

-rw-r--r--    1 root     0           5.9K Feb 26 18:56 sigmigration.log

-rw-r--r--    1 root     0           6.0K Feb 26 19:58 skein.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 smbnetfs.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 smtpd_error.log

-rw-r--r--    1 root     0           1.6M Mar  4 12:19 smtpd_main.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 smtpd_panic.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 smtpd_reject.log

-rw-r--r--    1 root     0            619 Feb 26 19:45 snireport.log

-rw-r--r--    1 root     0            813 Feb 26 19:58 snmpd.log

-rw-r--r--    1 root     0            695 Feb 26 19:58 sophos-central.log

-rw-r--r--    1 root     0           7.9K Mar  4 12:18 sshd.log

-rw-r--r--    1 root     0          73.9M Mar  4 12:19 sslvpn.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 ssod.log

-rw-r--r--    1 root     0            651 Feb 26 19:57 staticd.log

-rw-r--r--    1 root     0           1.5K Feb 26 19:58 strongswan-monitor.log

-rw-r--r--    1 root     0         286.3K Mar  4 12:16 strongswan.log

-rw-r--r--    1 root     0            488 Feb 26 19:56 strongswan_migration.log

-rw-r--r--    1 root     0              0 Feb 26 18:33 sync.log

-rw-r--r--    1 root     0          77.5K Feb 26 19:55 sysinit.log

-rw-r--r--    1 root     0           1.3M Mar  4 08:45 syslog-ng.log

-rw-r--r--    1 root     0           3.6M Mar  4 12:19 syslog.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 test_snort_config.log

-rw-r--r--    1 root     0           2.1K Mar  4 05:26 tlsreport.log

-rw-r--r--    1 root     0           6.4K Mar  4 00:30 tmclient.log

-rw-r--r--    1 root     0          44.6M Mar  4 12:19 tomcat.log

-rw-r--r--    1 root     0            752 Feb 29 12:00 tracelog.txt

-rw-r--r--    1 root     0         924.3K Mar  4 12:15 u2d.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 u2d_airgap.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 uma.log

-rw-r--r--    1 root     0          18.6K Mar  4 10:01 up2date_av.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 utm_info.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 validation.log

-rw-r--r--    1 root     0           6.9M Mar  4 12:16 validationError.log

-rw-r--r--    1 root     0         160.5K Feb 27 14:16 validationError.log-20240227_141712.gz

-rw-r--r--    1 root     0         142.1K Feb 27 20:57 validationError.log-20240227_205712.gz

-rw-r--r--    1 root     0         139.9K Feb 28 05:07 validationError.log-20240228_050727.gz

-rw-r--r--    1 root     0              0 Feb 26 18:34 vhost.log

-rw-r--r--    1 root     0           2.0K Mar  2 09:06 vpncertificate.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 waagent.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 waf_timer.log

-rw-r--r--    1 root     0          61.2K Feb 26 19:57 warren.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 wc_remote.log

-rw-r--r--    1 root     0              0 Feb 26 18:33 webproxy.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 wifiauth.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 xfrmi.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 xgs-healthmond.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 xgs-host.log

-rw-r--r--    1 root     0              0 Feb 26 18:34 xgs-npu-fw.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 xgs-npu-serial.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 xgs-pport-wait.log

-rw-r--r--    1 root     0           5.7K Mar  3 23:58 zebra.log

-rw-r--r--    1 root     0              0 Feb 26 18:57 ztna-connector.log

So this looks fine to me.

Can you check ls -lh /tmp/ as well? 

It looks fine because ips.log file size is not large. If I start IPS service again, after couple of days, it will take up all the disk space. I started ips service for 2 minutes and it generated 2.3 MB of log!

/tmp high usage is because of my mistake. I had tried to move ips.log to /tmp before removing the original file and it caused /tmp to fill up. I removed ips.log from there and now it's usage is back to normal. The original issue still persists.

XG210_WP03_SFOS 19.5.4 MR-4-Build718# df -h

Filesystem                Size      Used Available Use% Mounted on

none                      1.5G      1.2M      1.4G   0% /

none                      3.8G    512.0K      3.8G   0% /dev

none                      3.8G     13.1M      3.8G   0% /tmp

none                      3.8G     14.6M      3.8G   0% /dev/shm

/dev/boot               126.2M     31.0M     92.5M  25% /boot

/dev/mapper/mountconf

                        954.9M     88.0M    863.0M   9% /conf

/dev/content             11.2G    487.0M     10.7G   4% /content

/dev/var                 87.0G     14.5G     72.5G  17% /var

Can you monitor the IPS log? 

For example i have 163MB on the IPS.log but it rotates all the time. 

I've done it before. It will not rotate and it will grow up until disk space is full. Last time that I left the service in running state, file size reached to 70GB.

Could you create a support case for that? 

Unfortunately, we do not have active support subscription for this device.

 Farshid I am responding only to log rotate part. why is ips log coming continuously will need to be checked by different experts..

For log rotation not working - you probably have run into situation where log rotation  stopped for some reason.

Can you check the csc.log if it shows any instances of  "log_rotate:exec Failed".

If it shows you can restart log rotations by running following  from advanced shell.

/bin/logrotate "/static/logrotate/logrotate.conf" -s "/tmp/logrotate.status" -l "/log/logrotate.log"

If the log rotation continues to fail - we will need do more investigations with support access id available. -Shrikant

BTW - looking at the /log output above log rotation does seems to be working as of mar 1st.

-rw-r--r--    1 root     0         278.6K Mar  1 23:53 dnsgrabber.log-20240301_235302.gz

was the device restarted after you have seen 70GB ips.log.

how long does it take to ips.log to become 70GB in this device?

 Farshid I am responding only to log rotate part. why is ips log coming continuously will need to be checked by different experts..

For log rotation not working - you probably have run into situation where log rotation  stopped for some reason.

Can you check the csc.log if it shows any instances of  "log_rotate:exec Failed".

If it shows you can restart log rotations by running following  from advanced shell.

/bin/logrotate "/static/logrotate/logrotate.conf" -s "/tmp/logrotate.status" -l "/log/logrotate.log"

If the log rotation continues to fail - we will need do more investigations with support access id available. -Shrikant

BTW - looking at the /log output above log rotation does seems to be working as of mar 1st.

-rw-r--r--    1 root     0         278.6K Mar  1 23:53 dnsgrabber.log-20240301_235302.gz

was the device restarted after you have seen 70GB ips.log.

how long does it take to ips.log to become 70GB in this device?

I tried "cat /log/csc.log | grep "log_rotate" and I only see logs like following that I only assume it indicates log rotation is working like you have suggested.

MESSAGE   Mar 01 22:29:07Z  [log_rotate:28371]: opcode 'log_rotate': time taken: 0.075893232 seconds with return status: '200'

MESSAGE   Mar 01 22:29:57Z  [worker:28461]: {"request":{"method":"nopcode","name":"log_rotate","version":"1.0","type":"text","length":0}}

Anyway, I manually restarted log rotation with command you have provided and will check for results.

To answer your other questions:

This device has not restarted after removing large IPS file.

I'm not sure how much it takes because I have been notified regarding this issues today but I think it takes about a week. We reimaged this device 26th February and today I've seen 70GB ips.log file.

Farshid , Can you please send your access id in private message so we can look at the configuration and logs.

docs.sophos.com/.../index.html

I can't send PM to you, please change your setting to communicate with you. 

Can you send the `cat /var/tmp/debugp.conf` output to make sure ips is not running in debug mode?

Sorry for PM issue, fixed it.

there was no such file at path that you gave but I confirmed that IPS service is not running in debug mode with following:

XG210_WP03_SFOS 19.5.4 MR-4-Build718# service -S | grep ips

ips          RUNNING

I believe if it was running in debug mode the result would be: "ips         RUNNING,DEBUG"

BTW, I left the IPS service in running state since 6 hours ago and till now ips.log size is not increasing to unusual amount and it only takes 11MB but as SG001 suggested, maybe it works fine until some point and then during a short period fills up disk.