Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Thread Info
  • State Suggested Answer
  • +1 person also asked this people also asked this
  • Locked Locked
  • Replies 27 replies
  • Answers 1 answer
  • Subscribers 45 subscribers
  • Views 8243 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

ips.log filling up disk

Farshid

We have XG210 with SFOS 19.5.4. I've noticed ips.log filling up /var partition till there is no free space on disk and it causes device to boot into fail-safe mode. Stopping IPS service stops log file from growing but when I restart IPS service, this issue occurs again. Switching off IPS Protection button from Intrusion Prevention menu or removing IPS polices from all firewall rules does not resolve this issue and only stopping service stops it. Here is the log that reappears again and again in the log file and causes this issue:

2024-03-03T20:42:03.268400Z [28142]:DAQ:INFO:daq_nmsp.c:2342(daq_nmsp_get_pkt)--> jumbogram read failed. jumbo len: 0 rslot 11 rlen 2048

2024-03-03T20:42:03.268402Z [28142]:DAQ:INFO:daq_nmsp.c:2384(daq_nmsp_get_pkt)--> Exit --> DAQ Error -7

I've tried re-imaging SFOS but it didn't solve the problem. Is there anyway to solve this issue without need to stopping IPS?



This thread was automatically locked due to age.
Parents
  • 0

    Can you share a df -h output here?

    And please an ls -lh /log/ 

    Because generally speaking, the Log Rotation should hit here and override the Log File to prevent this. 

    __________________________________________________________________________________________________________________

  • 0 in reply to LuCar Toni

    Thank you for your respone. Please be aware that I have removed ips.log file couple of hours ago because its size was 70GB+ and it caused /var partition usage become 100%. Also LoggingDaemon was stopped. Removing ips.log resolved these issues temporarily but since then I've stopped the IPS service. If I start it again, same log will be generated again and same issues will reiterate.

    Here is the output you asked:

    XG210_WP03_SFOS 19.5.4 MR-4-Build718# df -h

    Filesystem                Size      Used Available Use% Mounted on

    none                      1.5G      1.2M      1.4G   0% /

    none                      3.8G    512.0K      3.8G   0% /dev

    none                      3.8G      3.8G     60.0K 100% /tmp

    none                      3.8G     14.6M      3.8G   0% /dev/shm

    /dev/boot               126.2M     31.0M     92.5M  25% /boot

    /dev/mapper/mountconf

                            954.9M     88.0M    863.0M   9% /conf

    /dev/content             11.2G    487.0M     10.7G   4% /content

    /dev/var                 87.0G     14.5G     72.5G  17% /var

    XG210_WP03_SFOS 19.5.4 MR-4-Build718# ls -lh /log/

    -rw-r--r--    1 root     0              0 Feb 26 18:34 VPN.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 WINGc.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 WINGu.log

    -rw-r--r--    1 root     0          42.8M Mar  4 12:19 access_server.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 apache.log

    -rw-r--r--    1 root     0          16.1M Mar  4 12:19 apache_access.log

    -rw-r--r--    1 root     0              0 Feb 27 08:59 apiparser.log

    -rw-r--r--    1 root     0          12.1K Feb 27 08:59 apiparser.log-20240227_085953.gz

    -rw-r--r--    1 root     0            836 Mar  4 05:42 app-feedback.log

    -rw-r--r--    1 root     0         614.9K Mar  4 11:47 appcached.log

    -rw-r--r--    1 root     0          15.2M Mar  4 12:19 applog.log

    -rw-r--r--    1 root     0             42 Mar  4 12:19 async_auth.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 atop.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 av.log

    -rw-r--r--    1 root     0           2.3M Mar  4 10:01 avd.log

    -rw-r--r--    1 root     0           1.2M Mar  4 12:18 awarrenhttp.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 awarrenhttp_access.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 awarrenmta.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 awarrenmta_debug.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 awarrensmtp.log

    -rw-r--r--    1 root     0          10.2K Feb 26 19:57 awed.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 bad_mail_detector.log

    -rw-r--r--    1 root     0           1.0K Mar  4 10:42 bgpd.log

    -rw-r--r--    1 root     0            716 Feb 26 19:57 bwm.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 catUpdateLog

    -rw-r--r--    1 root     0              0 Feb 26 18:34 catUpdateLog1

    -rw-r--r--    1 root     0            375 Feb 26 19:59 catUpdateLog11

    -rw-r--r--    1 root     0              0 Feb 26 18:34 centralmanagement.log

    -rw-r--r--    1 root     0         282.3K Mar  4 12:16 charon.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 chromebook-sso-backend.log

    -rw-r--r--    1 root     0            334 Feb 26 18:55 clientless_access.log

    -rw-r--r--    1 root     0              0 Feb 26 19:56 confdbstatus.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 crreportdb.log

    -rw-r--r--    1 root     0          59.9M Mar  4 12:19 csc.log

    -rw-r--r--    1 root     0           8.5M Mar  3 00:00 cschelper.log

    -rw-r--r--    1 root     0           6.3M Feb 28 00:00 cschelper.log-20240228_000015.gz

    -rw-r--r--    1 root     0              0 Feb 26 18:34 csd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 ctasd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 ctipd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 ctsyncd.log

    -rw-r--r--    1 root     0           5.5K Feb 26 19:58 czt.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 dbcleanup.log

    -rw-r--r--    1 root     0           1.9K Feb 27 09:04 ddc.log

    -rw-r--r--    1 root     0           3.6M Mar  4 12:18 dgd.log

    -rw-r--r--    1 root     0           3.9K Feb 26 19:45 dhcpd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 dhcpd6.log

    -rw-r--r--    1 root     0         111.6K Mar  4 12:15 dnsd.log

    -rw-r--r--    1 root     0          79.8M Mar  4 12:19 dnsgrabber.log

    -rw-r--r--    1 root     0         278.6K Mar  1 23:53 dnsgrabber.log-20240301_235302.gz

    -rw-r--r--    1 root     0              0 Feb 26 18:57 dropbear.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 eacd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:33 entity.log

    -rw-r--r--    1 root     0          40.0K Mar  4 08:44 error_log.log

    -rw-r--r--    1 root     0           1.3K Feb 29 08:16 exim_mail_client.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 filesync.log

    -rw-r--r--    1 root     0          37.7K Mar  4 10:50 firewall_rule.log

    -rw-r--r--    1 root     0           4.3M Mar  4 12:18 fqdnd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 fqdndebug.log

    -rw-r--r--    1 root     0            148 Mar  2 03:00 fstrim.log

    -rw-r--r--    1 root     0            920 Feb 26 19:57 ftpproxy.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 fwcm-eventd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 fwcm-heartbeatd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 fwcm-updaterd.log

    -rw-r--r--    1 root     0          32.0K Mar  4 08:44 fwlog.log

    -rw-r--r--    1 root     0          22.0K Mar  2 23:58 fwmgmt.log

    -rw-r--r--    1 root     0           6.1M Mar  4 12:19 garner.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 ha_pair.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 ha_tunnel.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 hbtrust.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 heartbeatd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 hostapd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:33 hotspotd.log

    -rw-r--r--    1 root     0         417.9K Mar  4 11:50 httplogd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 hwaccel.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 hwmon.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 iaasd.log

    -rw-r--r--    1 root     0           2.3M Mar  4 10:51 ips.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 ipsec.log

    drwxr-xr-x    2 root     0           4.0K Feb 26 18:57 ipsec_conn

    -rw-r--r--    1 root     0           1.5K Feb 26 19:58 ipsec_monitor.log

    -rw-r--r--    1 root     0          20.8M Mar  4 12:14 iview.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 l2tpd.log

    -rw-r--r--    1 root     0          13.9K Feb 26 19:58 lcd.log

    -rw-r--r--    1 root     0           6.9K Feb 27 00:00 legacyconversion.log

    -rw-r--r--    1 root     0          28.0K Mar  3 19:58 licensing.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 logrotate.log

    -rw-r--r--    1 root     0           5.9K Feb 26 19:57 mdev.log

    -rw-r--r--    1 root     0           7.1K Feb 26 18:56 migration.log

    -rw-r--r--    1 root     0          59.9K Feb 26 19:56 migrationhash.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 mrouting.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 msync.log

    drwxr-xr-x    2 root     0           4.0K Feb 26 18:33 nSXLd

    -rw-r--r--    1 root     0         243.2K Mar  4 07:57 nSXLd.log

    -rw-r--r--    1 root     0          23.8K Mar  4 12:16 nasm.log

    -rw-r--r--    1 root     0          15.8K Mar  3 17:22 nat_rule.log

    -rw-r--r--    1 root     0         174.2K Mar  4 09:04 networkd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 npu-startup.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 npu-startup.log.prev

    -rw-r--r--    1 root     0              0 Feb 26 18:34 npu_syslog.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 ntpclient.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 oauth_sso_captive.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 oauth_sso_userportal.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 oauth_sso_webadmin.log

    -rw-r--r--    1 root     0              0 Feb 26 19:58 openvpn-status.log

    -rw-------    1 root     0              0 Mar  4 12:19 openvpn-status0.log

    -rw-r--r--    1 root     0             72 Feb 26 19:55 ospf6d.log

    -rw-r--r--    1 root     0            449 Feb 26 19:55 ospfd.log

    -rw-r--r--    1 root     0           2.2M Mar  4 12:18 osquery.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 pimd.log

    -rw-r--r--    1 root     0           4.5K Feb 29 12:00 pktcapd.log

    -rw-r--r--    1 root     0           8.9M Mar  4 12:18 postgres.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 pptpvpn.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 radvd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 raid.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 readapplicationapi.log

    -rw-r--r--    1 root     0            373 Mar  4 12:18 readobject.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 red.log

    drwxr-xr-x    2 root     0           4.0K Mar  1 10:42 redis

    -rw-r--r--    1 root     0           4.1K Mar  3 16:04 regen_client_bundle.log

    -rw-r--r--    1 root     0          61.6K Mar  4 08:50 reportdb.log

    -rw-r--r--    1 root     0           3.5K Feb 26 20:08 reportmigration.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 reverseproxy.log

    -rw-r--r--    1 root     0            550 Feb 26 18:55 ripd.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 routinghelper.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 sac-feedback.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 sandbox_reportd.log

    -rw-r--r--    1 root     0           1.8K Feb 26 19:57 sandboxd.log

    -rw-r--r--    1 root     0         152.0K Mar  3 18:17 sasi.log

    -rw-------    1 root     0              0 Feb 26 18:33 sessiontbl.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 setup_vf_dpdk.log

    -rw-r--r--    1 root     0           3.9K Feb 29 12:00 sig_upgrade.log

    -rw-r--r--    1 root     0         165.3K Mar  4 12:18 sigdb.log

    -rw-r--r--    1 root     0           5.9K Feb 26 18:56 sigmigration.log

    -rw-r--r--    1 root     0           6.0K Feb 26 19:58 skein.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 smbnetfs.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 smtpd_error.log

    -rw-r--r--    1 root     0           1.6M Mar  4 12:19 smtpd_main.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 smtpd_panic.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 smtpd_reject.log

    -rw-r--r--    1 root     0            619 Feb 26 19:45 snireport.log

    -rw-r--r--    1 root     0            813 Feb 26 19:58 snmpd.log

    -rw-r--r--    1 root     0            695 Feb 26 19:58 sophos-central.log

    -rw-r--r--    1 root     0           7.9K Mar  4 12:18 sshd.log

    -rw-r--r--    1 root     0          73.9M Mar  4 12:19 sslvpn.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 ssod.log

    -rw-r--r--    1 root     0            651 Feb 26 19:57 staticd.log

    -rw-r--r--    1 root     0           1.5K Feb 26 19:58 strongswan-monitor.log

    -rw-r--r--    1 root     0         286.3K Mar  4 12:16 strongswan.log

    -rw-r--r--    1 root     0            488 Feb 26 19:56 strongswan_migration.log

    -rw-r--r--    1 root     0              0 Feb 26 18:33 sync.log

    -rw-r--r--    1 root     0          77.5K Feb 26 19:55 sysinit.log

    -rw-r--r--    1 root     0           1.3M Mar  4 08:45 syslog-ng.log

    -rw-r--r--    1 root     0           3.6M Mar  4 12:19 syslog.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 test_snort_config.log

    -rw-r--r--    1 root     0           2.1K Mar  4 05:26 tlsreport.log

    -rw-r--r--    1 root     0           6.4K Mar  4 00:30 tmclient.log

    -rw-r--r--    1 root     0          44.6M Mar  4 12:19 tomcat.log

    -rw-r--r--    1 root     0            752 Feb 29 12:00 tracelog.txt

    -rw-r--r--    1 root     0         924.3K Mar  4 12:15 u2d.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 u2d_airgap.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 uma.log

    -rw-r--r--    1 root     0          18.6K Mar  4 10:01 up2date_av.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 utm_info.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 validation.log

    -rw-r--r--    1 root     0           6.9M Mar  4 12:16 validationError.log

    -rw-r--r--    1 root     0         160.5K Feb 27 14:16 validationError.log-20240227_141712.gz

    -rw-r--r--    1 root     0         142.1K Feb 27 20:57 validationError.log-20240227_205712.gz

    -rw-r--r--    1 root     0         139.9K Feb 28 05:07 validationError.log-20240228_050727.gz

    -rw-r--r--    1 root     0              0 Feb 26 18:34 vhost.log

    -rw-r--r--    1 root     0           2.0K Mar  2 09:06 vpncertificate.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 waagent.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 waf_timer.log

    -rw-r--r--    1 root     0          61.2K Feb 26 19:57 warren.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 wc_remote.log

    -rw-r--r--    1 root     0              0 Feb 26 18:33 webproxy.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 wifiauth.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 xfrmi.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 xgs-healthmond.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 xgs-host.log

    -rw-r--r--    1 root     0              0 Feb 26 18:34 xgs-npu-fw.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 xgs-npu-serial.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 xgs-pport-wait.log

    -rw-r--r--    1 root     0           5.7K Mar  3 23:58 zebra.log

    -rw-r--r--    1 root     0              0 Feb 26 18:57 ztna-connector.log

  • 0 in reply to Farshid

    So this looks fine to me.

    Can you check ls -lh /tmp/ as well? 

    __________________________________________________________________________________________________________________

  • 0 in reply to LuCar Toni

    It looks fine because ips.log file size is not large. If I start IPS service again, after couple of days, it will take up all the disk space. I started ips service for 2 minutes and it generated 2.3 MB of log!

    /tmp high usage is because of my mistake. I had tried to move ips.log to /tmp before removing the original file and it caused /tmp to fill up. I removed ips.log from there and now it's usage is back to normal. The original issue still persists.

    XG210_WP03_SFOS 19.5.4 MR-4-Build718# df -h

    Filesystem                Size      Used Available Use% Mounted on

    none                      1.5G      1.2M      1.4G   0% /

    none                      3.8G    512.0K      3.8G   0% /dev

    none                      3.8G     13.1M      3.8G   0% /tmp

    none                      3.8G     14.6M      3.8G   0% /dev/shm

    /dev/boot               126.2M     31.0M     92.5M  25% /boot

    /dev/mapper/mountconf

                            954.9M     88.0M    863.0M   9% /conf

    /dev/content             11.2G    487.0M     10.7G   4% /content

    /dev/var                 87.0G     14.5G     72.5G  17% /var

  • 0 in reply to Farshid

    Can you monitor the IPS log? 

    For example i have 163MB on the IPS.log but it rotates all the time. 

    __________________________________________________________________________________________________________________

  • 0 in reply to LuCar Toni

    I've done it before. It will not rotate and it will grow up until disk space is full. Last time that I left the service in running state, file size reached to 70GB.

  • 0 in reply to Farshid

    Could you create a support case for that? 

    __________________________________________________________________________________________________________________

  • 0 in reply to LuCar Toni

    Unfortunately, we do not have active support subscription for this device.

Reply Children
  • 0 in reply to Farshid

      I am responding only to log rotate part. why is ips log coming continuously will need to be checked by different experts..

    For log rotation not working - you probably have run into situation where log rotation  stopped for some reason.

    Can you check the csc.log if it shows any instances of  "log_rotate:exec Failed".

    If it shows you can restart log rotations by running following  from advanced shell.

    /bin/logrotate "/static/logrotate/logrotate.conf" -s "/tmp/logrotate.status" -l "/log/logrotate.log"

    If the log rotation continues to fail - we will need do more investigations with support access id available. -Shrikant

    BTW - looking at the /log output above log rotation does seems to be working as of mar 1st.

    -rw-r--r--    1 root     0         278.6K Mar  1 23:53 dnsgrabber.log-20240301_235302.gz

    was the device restarted after you have seen 70GB ips.log.

    how long does it take to ips.log to become 70GB in this device?

  • 0 in reply to ShrikantSophos

    I tried "cat /log/csc.log | grep "log_rotate" and I only see logs like following that I only assume it indicates log rotation is working like you have suggested.

    MESSAGE   Mar 01 22:29:07Z  [log_rotate:28371]: opcode 'log_rotate': time taken: 0.075893232 seconds with return status: '200'

    MESSAGE   Mar 01 22:29:57Z  [worker:28461]: {"request":{"method":"nopcode","name":"log_rotate","version":"1.0","type":"text","length":0}}

    Anyway, I manually restarted log rotation with command you have provided and will check for results.

    To answer your other questions:

    This device has not restarted after removing large IPS file.

    I'm not sure how much it takes because I have been notified regarding this issues today but I think it takes about a week. We reimaged this device 26th February and today I've seen 70GB ips.log file.

  • 0 in reply to Farshid

     , Can you please send your access id in private message so we can look at the configuration and logs.

    docs.sophos.com/.../index.html

  • 0 in reply to NileshPatel

    I can't send PM to you, please change your setting to communicate with you. 

    Can you send the `cat /var/tmp/debugp.conf` output to make sure ips is not running in debug mode?

  • 0 in reply to NileshPatel

    Sorry for PM issue, fixed it.

    there was no such file at path that you gave but I confirmed that IPS service is not running in debug mode with following:

    XG210_WP03_SFOS 19.5.4 MR-4-Build718# service -S | grep ips

    ips          RUNNING

    I believe if it was running in debug mode the result would be: "ips         RUNNING,DEBUG"

  • 0 in reply to NileshPatel

    BTW, I left the IPS service in running state since 6 hours ago and till now ips.log size is not increasing to unusual amount and it only takes 11MB but as  suggested, maybe it works fine until some point and then during a short period fills up disk.

Unfiltered HTML