Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Thread Info
  • State Suggested Answer
  • +1 person also asked this people also asked this
  • Locked Locked
  • Replies 27 replies
  • Answers 1 answer
  • Subscribers 45 subscribers
  • Views 7980 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

ips.log filling up disk

Farshid

We have XG210 with SFOS 19.5.4. I've noticed ips.log filling up /var partition till there is no free space on disk and it causes device to boot into fail-safe mode. Stopping IPS service stops log file from growing but when I restart IPS service, this issue occurs again. Switching off IPS Protection button from Intrusion Prevention menu or removing IPS polices from all firewall rules does not resolve this issue and only stopping service stops it. Here is the log that reappears again and again in the log file and causes this issue:

2024-03-03T20:42:03.268400Z [28142]:DAQ:INFO:daq_nmsp.c:2342(daq_nmsp_get_pkt)--> jumbogram read failed. jumbo len: 0 rslot 11 rlen 2048

2024-03-03T20:42:03.268402Z [28142]:DAQ:INFO:daq_nmsp.c:2384(daq_nmsp_get_pkt)--> Exit --> DAQ Error -7

I've tried re-imaging SFOS but it didn't solve the problem. Is there anyway to solve this issue without need to stopping IPS?



This thread was automatically locked due to age.
  • 0 in reply to Farshid

      I am responding only to log rotate part. why is ips log coming continuously will need to be checked by different experts..

    For log rotation not working - you probably have run into situation where log rotation  stopped for some reason.

    Can you check the csc.log if it shows any instances of  "log_rotate:exec Failed".

    If it shows you can restart log rotations by running following  from advanced shell.

    /bin/logrotate "/static/logrotate/logrotate.conf" -s "/tmp/logrotate.status" -l "/log/logrotate.log"

    If the log rotation continues to fail - we will need do more investigations with support access id available. -Shrikant

    BTW - looking at the /log output above log rotation does seems to be working as of mar 1st.

    -rw-r--r--    1 root     0         278.6K Mar  1 23:53 dnsgrabber.log-20240301_235302.gz

    was the device restarted after you have seen 70GB ips.log.

    how long does it take to ips.log to become 70GB in this device?

  • 0 in reply to LuCar Toni

    First Case: 07151744

    Second Case: 07213100

  • 0 in reply to ShrikantSophos

    I tried "cat /log/csc.log | grep "log_rotate" and I only see logs like following that I only assume it indicates log rotation is working like you have suggested.

    MESSAGE   Mar 01 22:29:07Z  [log_rotate:28371]: opcode 'log_rotate': time taken: 0.075893232 seconds with return status: '200'

    MESSAGE   Mar 01 22:29:57Z  [worker:28461]: {"request":{"method":"nopcode","name":"log_rotate","version":"1.0","type":"text","length":0}}

    Anyway, I manually restarted log rotation with command you have provided and will check for results.

    To answer your other questions:

    This device has not restarted after removing large IPS file.

    I'm not sure how much it takes because I have been notified regarding this issues today but I think it takes about a week. We reimaged this device 26th February and today I've seen 70GB ips.log file.

  • 0 in reply to SG001

      is this 19.5MR4 too? do you also see log rotation not happening for ips.log? what is max size of ips.log you are seeing? 

  • 0 in reply to ShrikantSophos

    We have ips.log growing issue in 19.5.4 but I'm not sure root cause for it is log rotation. I'll look for result of command that Shirkant gave and update with results.

  • 0 in reply to ShrikantSophos

    In my oppinion it's not a problem with log rotation. Furthermore it's a problem in the ips /dpi engine.

    As soon as the error occurs the firewall is becoming more and more unstable until all connections are dropped and during this time there are thousands of the messages you mentioned above in the ips.log.

    The ips.log is growing then from 300MB to about 170gb which fills the partition up. All of this happens in just a few minutes.

    The only thing we can do about this is failover to the auxilliary node and clean up the partition on the former primary node.

    We are running into this issue approx. every 2 weeks.

    The cluster is running on SFOS V20.

  • 0 in reply to Farshid

     , Can you please send your access id in private message so we can look at the configuration and logs.

    docs.sophos.com/.../index.html

  • 0 in reply to NileshPatel

    I can't send PM to you, please change your setting to communicate with you. 

    Can you send the `cat /var/tmp/debugp.conf` output to make sure ips is not running in debug mode?

  • 0 in reply to NileshPatel

    Sorry for PM issue, fixed it.

    there was no such file at path that you gave but I confirmed that IPS service is not running in debug mode with following:

    XG210_WP03_SFOS 19.5.4 MR-4-Build718# service -S | grep ips

    ips          RUNNING

    I believe if it was running in debug mode the result would be: "ips         RUNNING,DEBUG"

  • 0 in reply to NileshPatel

    BTW, I left the IPS service in running state since 6 hours ago and till now ips.log size is not increasing to unusual amount and it only takes 11MB but as  suggested, maybe it works fine until some point and then during a short period fills up disk.

Unfiltered HTML