ips.log filling at high rate - normal and good for the SSD lifetime?

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LuCar Toni — Mon, 25 Nov 2024 08:27:48 GMT

We will track this and also reduce the visibility of those debug commands, which are to "harsh" on the system to be only available if Sophos Support is involved.
Thanks for reporting!

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LHerzog — Mon, 25 Nov 2024 08:17:42 GMT

just for reference after the issue is fixed:

this is how a regular ips.log would look like

2024-11-25T08:06:52.172001Z [25865/0x0] [nsg_nse_policy.c:1570:__nsg_error] xxx.xxx.xxx.xxx:55909 to xxx.xxx.xxx.xxx:443: Error from nse: NSE:Stream [0xbe00683e;code:62;sub:104] Flow reset
2024-11-25T08:06:52.172116Z [25865/0x0] [nsg_tcphold.c:1568:tcp_hold_process_control] Failed to get tcp_hold_session ssnptr 0x1dcb7b10 (hold_state: 3).
2024-11-25T08:06:52.189184Z [25863/0x0] [nsg_nse_policy.c:1570:__nsg_error] xxx.xxx.xxx.xxx:55951 to xxx.xxx.xxx.xxx:443: Error from nse: NSE:Stream [0xbe00683e;code:62;sub:104] Flow reset
2024-11-25T08:06:52.354643Z [25862/0x0] [nsg_nse_policy.c:1570:__nsg_error] xxx.xxx.xxx.xxx:55911 to xxx.xxx.xxx.xxx:443: Error from nse: NSE:Stream [0xbe00683e;code:62;sub:104] Flow reset
2024-11-25T08:06:52.354802Z [25862/0x0] [nsg_tcphold.c:1568:tcp_hold_process_control] Failed to get tcp_hold_session ssnptr 0x1de148f0 (hold_state: 3).
2024-11-25T08:06:52.422138Z [25863/0x0] [nsg_nse_policy.c:1570:__nsg_error] xxx.xxx.xxx.xxx:57702 to xxx.xxx.xxx.xxx:443: Error from nse: NSE:Internal [0xb000058f;code:143;sub:5] Flow timeout
UST sessiontbl_get_tuple API returned -1
UST sessiontbl_get_tuple API returned -1
2024-11-25T08:06:52.654064Z [25865/0x0] [nsg_nse_policy.c:1570:__nsg_error] xxx.xxx.xxx.xxx:57704 to xxx.xxx.xxx.xxx:443: Error from nse: NSE:Internal [0xb000058f;code:143;sub:5] Flow timeout
UST sessiontbl_get_tuple API returned -1
UST sessiontbl_get_tuple API returned -1
2024-11-25T08:06:55.510853Z [25865/0x0] [nsg_nse_policy.c:1570:__nsg_error] xxx.xxx.xxx.xxx:57725 to xxx.xxx.xxx.xxx:443: Error from nse: NSE:Internal [0xb000058f;code:143;sub:5] Flow timeout
UST sessiontbl_get_tuple API returned -1
UST sessiontbl_get_tuple API returned -1
2024-11-25T08:06:55.558649Z [25865/0x0] [nsg_nse_policy.c:1570:__nsg_error] xxx.xxx.xxx.xxx:57723 to xxx.xxx.xxx.xxx:443: Error from nse: NSE:Internal [0xb000058f;code:143;sub:5] Flow timeout
UST sessiontbl_get_tuple API returned -1
UST sessiontbl_get_tuple API returned -1
2024-11-25T08:06:55.559531Z [25862/0x0] [nsg_nse_policy.c:1570:__nsg_error] xxx.xxx.xxx.xxx:57722 to xxx.xxx.xxx.xxx:443: Error from nse: NSE:Internal [0xb000058f;code:143;sub:5] Flow timeout
UST sessiontbl_get_tuple API returned -1
UST sessiontbl_get_tuple API returned -1
2024-11-25T08:06:55.618653Z [25862/0x0] [nsg_nse_policy.c:1570:__nsg_error] xxx.xxx.xxx.xxx:57721 to xxx.xxx.xxx.xxx:443: Error from nse: NSE:Internal [0xb000058f;code:143;sub:5] Flow timeout
UST sessiontbl_get_tuple API returned -1
UST sessiontbl_get_tuple API returned -1
2024-11-25T08:06:55.683107Z [25863/0x0] [nsg_nse_policy.c:1570:__nsg_error] xxx.xxx.xxx.xxx:57720 to xxx.xxx.xxx.xxx:443: Error from nse: NSE:Internal [0xb000058f;code:143;sub:5] Flow timeout
UST sessiontbl_get_tuple API returned -1
UST sessiontbl_get_tuple API returned -1

file growth is minimal:
less than 10MB per minute
92.5M 09:14:37 /log/ips.log
92.5M 09:15:37 /log/ips.log

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

Michael Dunn — Fri, 22 Nov 2024 19:30:26 GMT

This has occurred in a few cases recently where support/GES/partner/??? are not deleting debug flag files.

awarrenhttp:
/sdisk/tmp/debug.cfg

ips:
/var/tmp/debugp.conf

Both awarrenhttp and ips have special debug flag files that allow for very finetuned levels of logging to be enabled and disabled (a mask of bits that turn on/off areas of logs) . When awarrenhttp and ips are started they look for the presence of the file. If found, they turn on DEBUG logging. However there is no way of telling csc/service that DEBUG is on. Therefore the output of the "service -S" command is incorrect.

It is critical to do two things:
1) Delete the flag files, ideally immediately after enabling debugging so you do not forget
2) Do not rely on "service -S" to tell you if DEBUG is on. Rely on the whether the log file is filling up with debug lines.

I did not know that we are publishing GES MER publicly.

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LHerzog — Fri, 22 Nov 2024 17:37:42 GMT

Confirmed - Logs back to normal! Great and Thank you very much!

And LuCar Toni was on the right track from the beginning!

I created /var/tmp/debugp.conf for the case mentioned here Webfilter HTTPS decryption breaks ChatGPT: HTTP parsing error encountered

That file contained my own IP address for the ChatGPT debug.

LuCar mentionend https://community.sophos.com/sophos-xg-firewall/f/discussions/145539/ips-log-filling-up-disk where Michael Dunn suggested to search and delete the debugp log file. I missed that.

Debug was enabled/disabled 3 times using service -ds nosync ips:debugp

NileshPatel I communicated 4 times, but found only 6 status code 200 response.

service -ds nosync ips:debugp
200 OK

I expected the file to be ignored when service -ds nosync ips:debugp was used to disable debug again.

Eventually it enables itself again when the firewall is rebooted? That could be checked. Nilesh wrote me, when IPS starts, it loads the debugp file. That was the case here because the firewall was rebooted one week ago for the v21 upgrade.

Sooner or later this wold have happened anyway.

So it's very important to delete the debugp file after the tests or at least have it containing only

mask=0

Maybe that is something for the Doc Team @RR@DocSupport for the article https://support.sophos.com/support/s/article/KBA-000006577?language=en_US

Thanks a lot everyone involved here! It was an exciting experience!

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LHerzog — Fri, 22 Nov 2024 17:22:35 GMT

checking SSH session logs I remembered I enabled a debug my self. more below to keep it readable here.

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

NileshPatel — Fri, 22 Nov 2024 17:21:14 GMT

The debugp mask was configured in /var/tmp/debugp.conf, which enabled debugging. To disable it, you might have run the command service ips:debugp -ds nosync. However, existing connections will still use the mask that was set when they first arrived.

I have updated the mask to 0 in /var/tmp/debugp.conf and restarted the IPS, which has fully disabled the WIS debug mode. The IPS restart also resolved the CSC status issue.

No debug logs are being generated after the IPS restart. Please confirm.

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LHerzog — Fri, 22 Nov 2024 16:09:20 GMT

case 02001662 / Webfilter HTTPS decryption breaks ChatGPT: HTTP parsing error encountered

was about IPS but I got no notification debug or hidden debug was enabled.

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

NileshPatel — Fri, 22 Nov 2024 15:58:40 GMT

service -S |grep -i ips is showing opposite status hence you are seeing less logs when you enable debug (which actually disabled the debug). we are looking into this issue.

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LHerzog — Fri, 22 Nov 2024 15:31:12 GMT

we observed that IPS in Debug mode logs way less than when running in non-Debug mode.

Factor was about 1:4

In non-debug there was all the connection information in the logs.

GES believes there is some kind of low-level default Debug enabled, which is ruled out when the service is put to debug mode.

NON-DEBUG:
2024-11-22T15:03:20.480455Z [27128] daq_metadata session id 5694 rev 6056 appid 0  hbappid 0 idp 18 appfltid 0
2024-11-22T15:03:20.480461Z [27128] verdict 0 pktnum 21011930 flowoff [IPS|WEB|AV] appcat [0:0] dsize 388 [xxx.xxx.xxx.xxx:46620 -> xxx.xxx.xxx.xxx:514]
2024-11-22T15:03:20.480464Z [27128] daq_metadata session id 5694 rev 6056 appid 0  hbappid 0 idp 18 appfltid 0
2024-11-22T15:03:20.480468Z [27128] verdict 0 pktnum 21011931 flowoff [IPS|WEB|AV] appcat [0:0] dsize 401 [xxx.xxx.xxx.xxx:46620 -> xxx.xxx.xxx.xxx:514]
2024-11-22T15:03:20.480470Z [27128] daq_metadata session id 5694 rev 6056 appid 0  hbappid 0 idp 18 appfltid 0
2024-11-22T15:03:20.480475Z [27128] verdict 0 pktnum 21011932 flowoff [IPS|WEB|AV] appcat [0:0] dsize 400 [xxx.xxx.xxx.xxx:46620 -> xxx.xxx.xxx.xxx:514]
2024-11-22T15:03:20.480477Z [27128] daq_metadata session id 5694 rev 6056 appid 0  hbappid 0 idp 18 appfltid 0
2024-11-22T15:03:20.480482Z [27128] verdict 0 pktnum 21011933 flowoff [IPS|WEB|AV] appcat [0:0] dsize 389 [xxx.xxx.xxx.xxx:46620 -> xxx.xxx.xxx.xxx:514]
2024-11-22T15:03:20.480523Z [27128] daq_metadata session id 12985 rev 11175 appid 2712  hbappid 0 idp 5 appfltid 4
2024-11-22T15:03:20.480533Z [27128] verdict 0 pktnum 7752 flowoff [APP|WEB|AV] appcat [2712:5] dsize 25 [xxx.xxx.xxx.xxx:443 -> xxx.xxx.xxx.xxx:42054]
2024-11-22T15:03:20.480604Z [27128] daq_metadata session id 12985 rev 11175 appid 2712  hbappid 0 idp 5 appfltid 4
2024-11-22T15:03:20.480612Z [27128] verdict 0 pktnum 7753 flowoff [APP|WEB|AV] appcat [2712:5] dsize 1250 [xxx.xxx.xxx.xxx:443 -> xxx.xxx.xxx.xxx:42054]
2024-11-22T15:03:20.480650Z [27128] daq_metadata session id 12985 rev 11175 appid 2712  hbappid 0 idp 5 appfltid 4
2024-11-22T15:03:20.480656Z [27128] verdict 0 pktnum 7754 flowoff [APP|WEB|AV] appcat [2712:5] dsize 1250 [xxx.xxx.xxx.xxx:443 -> xxx.xxx.xxx.xxx:42054]


DEBUG:
2024-11-22T15:00:19.086120Z [27127/0x57ea000049e5] [nsg_request_fsm.c:542:request_fsm_body_bytes] Event body_bytes in state RESP_PARSE_BODY_SKIP
2024-11-22T15:00:19.086192Z [27127/0x57ea000049e5] [acl/acl_lookup.c:608:check_access_list] ACL check: USER matched for WEB rule 0 (user-id=24)
2024-11-22T15:00:19.086196Z [27127/0x57ea000049e5] [acl/acl_lookup.c:611:check_access_list] checking policy [000], if matched, allow
2024-11-22T15:00:19.086199Z [27127/0x57ea000049e5] [acl/acl_lookup.c:392:__check_acl] checking 'all type 1'
2024-11-22T15:00:19.086202Z [27127/0x57ea000049e5] [acl/acl_lookup.c:623:check_access_list] All ACLs matched: rule action = allow
2024-11-22T15:00:19.086206Z [27127/0x57ea000049e5] [nsg_request_fsm.c:542:request_fsm_body_bytes] Event body_bytes in state RESP_PARSE_BODY_SKIP
2024-11-22T15:00:19.086209Z [27127/0x57ea000049e5] [acl/acl_lookup.c:608:check_access_list] ACL check: USER matched for WEB rule 0 (user-id=24)
2024-11-22T15:00:19.086211Z [27127/0x57ea000049e5] [acl/acl_lookup.c:611:check_access_list] checking policy [000], if matched, allow
2024-11-22T15:00:19.086213Z [27127/0x57ea000049e5] [acl/acl_lookup.c:392:__check_acl] checking 'all type 1'
2024-11-22T15:00:19.086216Z [27127/0x57ea000049e5] [acl/acl_lookup.c:623:check_access_list] All ACLs matched: rule action = allow
2024-11-22T15:00:19.086218Z [27127/0x57ea000049e5] [nsg_request_fsm.c:542:request_fsm_body_bytes] Event body_bytes in state RESP_PARSE_BODY_SKIP
2024-11-22T15:00:19.086230Z [27127/0x57ea000049e5] [acl/acl_lookup.c:608:check_access_list] ACL check: USER matched for WEB rule 0 (user-id=24)
2024-11-22T15:00:19.086232Z [27127/0x57ea000049e5] [acl/acl_lookup.c:611:check_access_list] checking policy [000], if matched, allow

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LHerzog — Fri, 22 Nov 2024 14:13:35 GMT

no, sorry, the service is not running in debug triple confirmed by Sophos Support and me.

meanwhile even the zipped .gz files getting bigger and bigger now reaching about 2GB while they were at 300MB yesterday

/dev/var                179.3G    112.3G     67.0G 63% /var
ls -lhS --full-time /log/ips.* | awk '{print $5,$7,$9}'
41.2G 14:48:56 /log/ips.log-20241122_143858
28.5G 15:09:24 /log/ips.log
2.3G 12:36:43 /log/ips.log-20241122_123643.gz
1.9G 13:06:46 /log/ips.log-20241122_130646.gz
1.8G 15:09:24 /log/ips.log-20241122_143858.gz
1.7G 14:13:53 /log/ips.log-20241122_141353.gz
1.7G 13:30:50 /log/ips.log-20241122_133050.gz
1.5G 13:52:49 /log/ips.log-20241122_135249.gz
XGS4500_AM02_SFOS 21.0.0 GA-Build169 HA-Primary# service -S |grep -i ips
ips                  RUNNING
ipsec-monitor        RUNNING

Debug would show

service -S |grep -i ips
ips RUNNING,DEBUG
ipsec-monitor RUNNING

GES is working on the issue. Confirmed logging is not normal.

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

NileshPatel — Fri, 22 Nov 2024 14:02:23 GMT

IPS debugging is currently enabled. You can toggle the debugging by using the command below.

service ips:debug -ds nosync

This won't be enabled by default. Was it enabled manually?

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LHerzog — Fri, 22 Nov 2024 09:34:03 GMT

and now I observe, it does not log-rotate correctly at the moment. files growing bigger and bigger.
That must be the same situation as yesterday.

ls -lhS --full-time /log/ips.* | awk '{print $5,$7,$9}'
25.7G 10:32:36 /log/ips.log
17.3G 10:32:36 /log/ips.log-20241122_102838
745.0M 10:16:12 /log/ips.log-20241122_101612.gz
557.8M 09:52:04 /log/ips.log-20241122_095204.gz
479.4M 09:58:58 /log/ips.log-20241122_095858.gz
461.3M 10:04:36 /log/ips.log-20241122_100436.gz
436.6M 10:10:14 /log/ips.log-20241122_101014.gz

an absolutely dangerous situation:

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LHerzog — Fri, 22 Nov 2024 09:19:23 GMT

there is not much of an example in that thread but I think they've had an issue with log lines repeating "jumbogram" string.

Also they were talking about some MB per minute, not GB like here on my XGS:

ls -lhS --full-time /log/ips.log | awk '{print $5,$7,$9}'
9.2G 10:17:13 /log/ips.log
ls -lhS --full-time /log/ips.log | awk '{print $5,$7,$9}'
9.3G 10:17:17 /log/ips.log
ls -lhS --full-time /log/ips.log | awk '{print $5,$7,$9}'
9.4G 10:17:21 /log/ips.log
ls -lhS --full-time /log/ips.log | awk '{print $5,$7,$9}'
9.6G 10:17:25 /log/ips.log
ls -lhS --full-time /log/ips.log | awk '{print $5,$7,$9}'
9.8G 10:17:29 /log/ips.log

-ls -lhS --full-time /log/ips.log | awk '{print $5,$7}'
2.4G 10:20:20
ls -lhS --full-time /log/ips.log | awk '{print $5,$7}'
7.0G 10:22:59
ls -lhS --full-time /log/ips.log | awk '{print $5,$7}'
14.2G 10:26:31

Our log file looks like above - it looks like this even when the IPS scanning is disabled but the IPS service is running. The screenshot from yesterday:

ips.log contains logs about every connection and some other event, too.

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LuCar Toni — Fri, 22 Nov 2024 08:46:14 GMT

The logs do not look like this? RE: ips.log filling up disk

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LHerzog — Fri, 22 Nov 2024 08:34:32 GMT

@Sophos do we have a confirmation that ips.log should be as noisy as it is? How would a ips.log look like on a random mid size firewall?

Those tiny CPU peaks every ode or two minute are only because the firewall is compressing new IPS logs into the .gz files all the time.

lines from our ips.log. look at the timestamps.

2024-11-22T08:32:13.206874Z [27132] verdict 0 pktnum 246732 flowoff [APP|AV] appcat [22245:25] dsize 1460 [xxx.xxx.xxx.xxx:55752 -> xxx.xxx.xxx.xxx:49524]
2024-11-22T08:32:13.206877Z [27132] daq_metadata session id 10238 rev 9840 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.206883Z [27132] verdict 0 pktnum 246733 flowoff [APP|AV] appcat [22245:25] dsize 1460 [xxx.xxx.xxx.xxx:55752 -> xxx.xxx.xxx.xxx:49524]
2024-11-22T08:32:13.206885Z [27132] daq_metadata session id 10238 rev 9840 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.206891Z [27132] verdict 0 pktnum 246734 flowoff [APP|AV] appcat [22245:25] dsize 362 [xxx.xxx.xxx.xxx:55752 -> xxx.xxx.xxx.xxx:49524]
2024-11-22T08:32:13.207020Z [27128] daq_metadata session id 16708 rev 44417 appid 100  hbappid 0 idp 6 appfltid 3
2024-11-22T08:32:13.207031Z [27128] verdict 3 pktnum 40 flowoff [APP|WEB|AV] appcat [100:5] dsize 0 [xxx.xxx.xxx.xxx:57041 -> xxx.xxx.xxx.xxx:443]
2024-11-22T08:32:13.207187Z [27132] daq_metadata session id 186 rev 30620 appid 0  hbappid 0 idp 18 appfltid 0
2024-11-22T08:32:13.207194Z [27132] verdict 0 pktnum 1904446 flowoff [IPS|WEB|AV] appcat [0:0] dsize 186 [xxx.xxx.xxx.xxx:43153 -> xxx.xxx.xxx.xxx:514]
2024-11-22T08:32:13.207387Z [27128] daq_metadata session id 19369 rev 24179 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.207403Z [27128] verdict 0 pktnum 42303 flowoff [APP|AV] appcat [22245:25] dsize 76 [xxx.xxx.xxx.xxx:55752 -> xxx.xxx.xxx.xxx:62628]
2024-11-22T08:32:13.207418Z [27132] daq_metadata session id 11766 rev 21901 appid 100  hbappid 0 idp 5 appfltid 4
2024-11-22T08:32:13.207426Z [27132] verdict 3 pktnum 677 flowoff [APP|WEB|AV] appcat [100:5] dsize 0 [xxx.xxx.xxx.xxx:40222 -> xxx.xxx.xxx.xxx:443]
2024-11-22T08:32:13.207460Z [27128] daq_metadata session id 13230 rev 4211 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.207490Z [27128] verdict 0 pktnum 590486 flowoff [APP|AV] appcat [22245:25] dsize 92 [xxx.xxx.xxx.xxx:55752 -> xxx.xxx.xxx.xxx:57352]
2024-11-22T08:32:13.207505Z [27128] daq_metadata session id 1827 rev 52159 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.207522Z [27132] daq_metadata session id 15394 rev 58396 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.207522Z [27127] daq_metadata session id 19396 rev 48441 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.207527Z [27128] verdict 0 pktnum 48110 flowoff [APP|AV] appcat [22245:25] dsize 140 [xxx.xxx.xxx.xxx:55752 -> xxx.xxx.xxx.xxx:62620]
2024-11-22T08:32:13.207538Z [27127] verdict 0 pktnum 48051 flowoff [APP|AV] appcat [22245:25] dsize 140 [xxx.xxx.xxx.xxx:55752 -> xxx.xxx.xxx.xxx:61487]
2024-11-22T08:32:13.207543Z [27132] verdict 0 pktnum 48182 flowoff [APP|AV] appcat [22245:25] dsize 140 [xxx.xxx.xxx.xxx:55752 -> xxx.xxx.xxx.xxx:49507]
2024-11-22T08:32:13.207589Z [27127] daq_metadata session id 17528 rev 26228 appid 20019  hbappid 20019 idp 14 appfltid 0
2024-11-22T08:32:13.207600Z [27127] verdict 0 pktnum 49464 flowoff [APP|WEB|AV] appcat [20019:25] dsize 1460 [xxx.xxx.xxx.xxx:60979 -> xxx.xxx.xxx.xxx:3268]
2024-11-22T08:32:13.207603Z [27127] daq_metadata session id 17528 rev 26228 appid 20019  hbappid 20019 idp 14 appfltid 0
2024-11-22T08:32:13.207612Z [27127] verdict 0 pktnum 49465 flowoff [APP|WEB|AV] appcat [20019:25] dsize 1460 [xxx.xxx.xxx.xxx:60979 -> xxx.xxx.xxx.xxx:3268]
2024-11-22T08:32:13.207614Z [27132] daq_metadata session id 11766 rev 21901 appid 100  hbappid 0 idp 5 appfltid 4
2024-11-22T08:32:13.207615Z [27127] daq_metadata session id 17528 rev 26228 appid 20019  hbappid 20019 idp 14 appfltid 0
2024-11-22T08:32:13.207625Z [27127] verdict 0 pktnum 49466 flowoff [APP|WEB|AV] appcat [20019:25] dsize 1460 [xxx.xxx.xxx.xxx:60979 -> xxx.xxx.xxx.xxx:3268]
2024-11-22T08:32:13.207629Z [27127] daq_metadata session id 17528 rev 26228 appid 20019  hbappid 20019 idp 14 appfltid 0
2024-11-22T08:32:13.207633Z [27132] verdict 3 pktnum 678 flowoff [APP|WEB|AV] appcat [100:5] dsize 0 [xxx.xxx.xxx.xxx:40222 -> xxx.xxx.xxx.xxx:443]
2024-11-22T08:32:13.207639Z [27127] verdict 0 pktnum 49467 flowoff [APP|WEB|AV] appcat [20019:25] dsize 1460 [xxx.xxx.xxx.xxx:60979 -> xxx.xxx.xxx.xxx:3268]
2024-11-22T08:32:13.207642Z [27127] daq_metadata session id 17528 rev 26228 appid 20019  hbappid 20019 idp 14 appfltid 0
2024-11-22T08:32:13.207646Z [27132] daq_metadata session id 7348 rev 8853 appid 2792  hbappid 0 idp 5 appfltid 4
2024-11-22T08:32:13.207647Z [27128] daq_metadata session id 1875 rev 27197 appid 0  hbappid 0 idp 6 appfltid 0
2024-11-22T08:32:13.207653Z [27127] verdict 0 pktnum 49468 flowoff [APP|WEB|AV] appcat [20019:25] dsize 1460 [xxx.xxx.xxx.xxx:60979 -> xxx.xxx.xxx.xxx:3268]
2024-11-22T08:32:13.207656Z [27127] daq_metadata session id 17528 rev 26228 appid 20019  hbappid 20019 idp 14 appfltid 0
2024-11-22T08:32:13.207655Z [27132] verdict 3 pktnum 94 flowoff [APP|WEB|AV] appcat [2792:12] dsize 0 [xxx.xxx.xxx.xxx:33666 -> xxx.xxx.xxx.xxx:443]
2024-11-22T08:32:13.207660Z [27132] daq_metadata session id 11766 rev 21901 appid 100  hbappid 0 idp 5 appfltid 4
2024-11-22T08:32:13.207665Z [27127] verdict 0 pktnum 49469 flowoff [APP|WEB|AV] appcat [20019:25] dsize 1460 [xxx.xxx.xxx.xxx:60979 -> xxx.xxx.xxx.xxx:3268]
2024-11-22T08:32:13.207658Z [27128] verdict 0 pktnum 62572 flowoff [IPS|APP] appcat [0:0] dsize 168 [xxx.xxx.xxx.xxx:42078 -> xxx.xxx.xxx.xxx:9997]
2024-11-22T08:32:13.207668Z [27127] daq_metadata session id 17528 rev 26228 appid 20019  hbappid 20019 idp 14 appfltid 0
2024-11-22T08:32:13.207669Z [27132] verdict 3 pktnum 679 flowoff [APP|WEB|AV] appcat [100:5] dsize 0 [xxx.xxx.xxx.xxx:40222 -> xxx.xxx.xxx.xxx:443]
2024-11-22T08:32:13.207677Z [27127] verdict 0 pktnum 49470 flowoff [APP|WEB|AV] appcat [20019:25] dsize 729 [xxx.xxx.xxx.xxx:60979 -> xxx.xxx.xxx.xxx:3268]
2024-11-22T08:32:13.207682Z [27132] daq_metadata session id 11766 rev 21901 appid 100  hbappid 0 idp 5 appfltid 4
2024-11-22T08:32:13.207689Z [27132] verdict 3 pktnum 680 flowoff [APP|WEB|AV] appcat [100:5] dsize 0 [xxx.xxx.xxx.xxx:40222 -> xxx.xxx.xxx.xxx:443]
2024-11-22T08:32:13.207788Z [27128] daq_metadata session id 1875 rev 27197 appid 0  hbappid 0 idp 6 appfltid 0
2024-11-22T08:32:13.207793Z [27127] daq_metadata session id 19396 rev 48441 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.207798Z [27128] verdict 0 pktnum 62519 flowoff [IPS|APP] appcat [0:0] dsize 0 [xxx.xxx.xxx.xxx:9997 -> xxx.xxx.xxx.xxx:42078]
2024-11-22T08:32:13.207805Z [27127] verdict 0 pktnum 35805 flowoff [APP|AV] appcat [22245:25] dsize 0 [xxx.xxx.xxx.xxx:61487 -> xxx.xxx.xxx.xxx:55752]
2024-11-22T08:32:13.207816Z [27132] daq_metadata session id 11766 rev 21901 appid 100  hbappid 0 idp 5 appfltid 4
2024-11-22T08:32:13.207824Z [27132] verdict 3 pktnum 681 flowoff [APP|WEB|AV] appcat [100:5] dsize 0 [xxx.xxx.xxx.xxx:40222 -> xxx.xxx.xxx.xxx:443]
2024-11-22T08:32:13.207843Z [27128] daq_metadata session id 13230 rev 4211 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.207853Z [27128] verdict 0 pktnum 545486 flowoff [APP|AV] appcat [22245:25] dsize 0 [xxx.xxx.xxx.xxx:57352 -> xxx.xxx.xxx.xxx:55752]
2024-11-22T08:32:13.207865Z [27128] daq_metadata session id 1827 rev 52159 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.207874Z [27128] verdict 0 pktnum 33047 flowoff [APP|AV] appcat [22245:25] dsize 0 [xxx.xxx.xxx.xxx:62620 -> xxx.xxx.xxx.xxx:55752]
2024-11-22T08:32:13.207877Z [27128] daq_metadata session id 19369 rev 24179 appid 22245  hbappid 22245 idp 6 appfltid 3
2024-11-22T08:32:13.207878Z [27127] daq_metadata session id 17528 rev 26228 appid 20019  hbappid 20019 idp 14 appfltid 0
2024-11-22T08:32:13.207885Z [27128] verdict 0 pktnum 22436 flowoff [APP|AV] appcat [22245:25] dsize 0 [xxx.xxx.xxx.xxx:62628 -> xxx.xxx.xxx.xxx:55752]
2024-11-22T08:32:13.207887Z [27127] verdict 0 pktnum 32526 flowoff [APP|WEB|AV] appcat [20019:25] dsize 0 [xxx.xxx.xxx.xxx:3268 -> xxx.xxx.xxx.xxx:60979]
2024-11-22T08:32:13.207908Z [27127] daq_metadata session id 17528 rev 26228 appid 20019  hbappid 20019 idp 14 appfltid 0
2024-11-22T08:32:13.207915Z [27127] verdict 0 pktnum 32527 flowoff [APP|WEB|AV] appcat [20019:25] dsize 0 [xxx.xxx.xxx.xxx:3268 -> xxx.xxx.xxx.xxx:60979]
2024-11-22T08:32:13.207926Z [27127] daq_metadata session id 6610 rev 8910 appid 19433  hbappid 19433 idp 6 appfltid 0
2024-11-22T08:32:13.207936Z [27127] verdict 0 pktnum 93598 flowoff [APP|WEB|AV] appcat [19433:25] dsize 10 [xxx.xxx.xxx.xxx:49761 -> xxx.xxx.xxx.xxx:3389]
2024-11-22T08:32:13.207942Z [27132] daq_metadata session id 7348 rev 8853 appid 2792  hbappid 0 idp 5 appfltid 4
2024-11-22T08:32:13.207950Z [27132] verdict 3 pktnum 95 flowoff [APP|WEB|AV] appcat [2792:12] dsize 0 [xxx.xxx.xxx.xxx:33666 -> xxx.xxx.xxx.xxx:443]
2024-11-22T08:32:13.207991Z [27128] daq_metadata session id 1875 rev 27197 appid 0  hbappid 0 idp 6 appfltid 0
2024-11-22T08:32:13.207999Z [27128] verdict 0 pktnum 62573 flowoff [IPS|APP] appcat [0:0] dsize 43 [xxx.xxx.xxx.xxx:42078 -> xxx.xxx.xxx.xxx:9997]
2024-11-22T08:32:13.208085Z [27128] daq_metadata session id 1875 rev 27197 appid 0  hbappid 0 idp 6 appfltid 0
2024-11-22T08:32:13.208092Z [27128] verdict 0 pktnum 62520 flowoff [IPS|APP] appcat [0:0] dsize 0 [xxx.xxx.xxx.xxx:9997 -> xxx.xxx.xxx.xxx:42078]

RE: ips.log filling at high rate - normal and good for the SSD lifetime?

LHerzog — Thu, 21 Nov 2024 14:56:31 GMT

the 24GB ips stale log file has been archived and then deleted manually.
and two other screenshots

RE: ips.log filling hat high rate - normal and good for the SSD lifetime?

LHerzog — Thu, 21 Nov 2024 13:21:11 GMT

for the IPS Drops we're seeing massively today it seems to be related to this URL

2024-11-21 10:07:26Web filtermessageid="16001" log_type="Content Filtering" log_component="HTTP" log_subtype="Allowed" fw_rule_id="2" fw_rule_name="User-2-WAN" fw_rule_section="Local rule" user="user@domain" user_group="OU=OU,DC=domain" web_policy_id="4" web_policy="OUR_DefaultPolicy" category="Advertisements" category_type="Unproductive" url="https://ih.adscale.de/adscale-ih/tpui?tpid=48&tpuid=-###truncated###-ayA4IB" content_type="" override_token="" src_ip="172.xxx.xxx.82" dst_ip="3.120.35.187" protocol="TCP" src_port="60035" dst_port="443" bytes_sent="1489" bytes_received="2603" domain="ih.adscale.de" exception="" activity_name="" reason="" user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0" status_code="302" transaction_id="feeedb56-f36f-46cf-a9f7-be58abb1c5e0" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="500959744" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

2024-11-21 10:07:26IPSmessageid="07002" log_type="IDP" log_component="Signatures" log_subtype="Drop" ips_policy="" ips_policy_id="16" fw_rule_id="2" fw_rule_name="User-2-WAN" fw_rule_section="Local rule" user="user@domain" sig_id="20583" message="BROWSER-FIREFOX Mozilla multiple location headers malicious redirect attempt" classification="Web Application Attack" rule_priority="3" src_ip="3.120.35.187" src_country="DEU" dst_ip="172.xxx.xxx.82" dst_country="R1" protocol="TCP" src_port="443" dst_port="60035" OS="Windows" category="browser-firefox" victim="Client"

also for other IPs

we'll block this FQDN

RE: ips.log filling hat high rate - normal and good for the SSD lifetime?

LHerzog — Thu, 21 Nov 2024 13:14:58 GMT

sure

no results.

cat /log/applog.log* | grep -i "log_rotate:exec Failed"

ls applog.log* -l
-rw-r--r--    1 root     root      28427822 Nov 21 14:12 applog.log
-rw-r--r--    1 root     root       2933453 Nov 14 09:14 applog.log-20241114_091429.gz
-rw-r--r--    1 root     root       3938860 Nov 14 15:15 applog.log-20241114_151521.gz
-rw-r--r--    1 root     root       2939131 Nov 15 06:48 applog.log-20241115_064840.gz
-rw-r--r--    1 root     root       3690161 Nov 15 14:11 applog.log-20241115_141123.gz
-rw-r--r--    1 root     root       3148974 Nov 16 04:24 applog.log-20241116_042414.gz

zcat /log/applog.*.gz | grep -i "log_rotate:exec Failed"

cat /log/applog.log | grep -i "failed"

...
Nov 21 04:10:00Z SendAppFeedback: app-feedback command failed. Details can be found in /log/app-feedback.log
Nov 21 06:30:20Z getModuleInformation opcode failed
Nov 21 06:30:20Z getModuleInformation opcode failed
Nov 21 09:04:20Z Get random adminpassword state failed
Nov 21 09:19:06Z Get random adminpassword state failed
next logs are after the reboot.

RE: ips.log filling hat high rate - normal and good for the SSD lifetime?

Raphael Alganes — Thu, 21 Nov 2024 13:14:57 GMT

Hello,

Regret to hear about your issue. Thank you for sharing support case, we shall be tracking progress on our end.

Thank you for your patience

Regards,

RE: ips.log filling hat high rate - normal and good for the SSD lifetime?

Mayur Makvana — Thu, 21 Nov 2024 13:09:30 GMT

Hello,

Can you share the output of below?

cat /log/applog.log | grep -i "log_rotate:exec Failed"

RE: ips.log filling hat high rate - normal and good for the SSD lifetime?

LHerzog — Thu, 21 Nov 2024 13:04:29 GMT

for heartbeat, yes, but not for IPS

RE: ips.log filling hat high rate - normal and good for the SSD lifetime?

LuCar Toni — Thu, 21 Nov 2024 13:01:31 GMT

Did you create for this XGS Appliance an case before? Because it looks like, there was special debugging enabled.

RE: ips.log filling hat high rate - normal and good for the SSD lifetime?

LHerzog — Thu, 21 Nov 2024 12:56:59 GMT

no, IPS was not in debug.

When we enabled IPS debug today the fill rate was not much changed to without debug.

RE: ips.log filling hat high rate - normal and good for the SSD lifetime?

LuCar Toni — Thu, 21 Nov 2024 12:54:14 GMT

Maybe someone in a older support ID enabled IPS Debugging?

RE: ips.log filling hat high rate - normal and good for the SSD lifetime?

LHerzog — Thu, 21 Nov 2024 12:21:28 GMT

showing the secondary heartbeat issue:

XGS4500_AM02_SFOS 21.0.0 GA-Build169 HA-Primary# tail /log/heartbeatd.log
gr_io: Broken pipe, Offset => 0
gr_io: Resource temporarily unavailable, after retrying 5 times
gr_io: Resource temporarily unavailable, after retrying 5 times
gr_io: Resource temporarily unavailable, after retrying 5 times
gr_io: Resource temporarily unavailable, after retrying 5 times

I think this is dangerous. Why do do these huge unzipped files remain in /log dir?

Having a few of them and the disk will be full again

XGS4500_AM02_SFOS 21.0.0 GA-Build169 HA-Primary# ls -lhS ips.*
-rw-r--r--    1 root     root       24.4G Nov 21 11:22 ips.log-20241121_111359
-rw-r--r--    1 root     root        4.0G Nov 21 13:47 ips.log
-rw-r--r--    1 root     root        3.3G Nov 21 13:46 ips.log-20241121_134627
-rw-r--r--    1 root     root      185.0M Nov 21 13:40 ips.log-20241121_134007.gz
-rw-r--r--    1 root     root      169.3M Nov 21 13:43 ips.log-20241121_134303.gz
-rw-r--r--    1 root     root      157.1M Nov 21 13:32 ips.log-20241121_133215.gz
-rw-r--r--    1 root     root      145.8M Nov 21 13:37 ips.log-20241121_133715.gz
-rw-r--r--    1 root     root      142.5M Nov 21 13:34 ips.log-20241121_133445.gz
#logrotate#
XGS4500_AM02_SFOS 21.0.0 GA-Build169 HA-Primary# ls -lhS ips.*
-rw-r--r--    1 root     root       24.4G Nov 21 11:22 ips.log-20241121_111359
-rw-r--r--    1 root     root        3.3G Nov 21 13:46 ips.log-20241121_134627
-rw-r--r--    1 root     root      306.3M Nov 21 13:47 ips.log
-rw-r--r--    1 root     root      185.0M Nov 21 13:40 ips.log-20241121_134007.gz
-rw-r--r--    1 root     root      169.3M Nov 21 13:43 ips.log-20241121_134303.gz
-rw-r--r--    1 root     root      157.1M Nov 21 13:32 ips.log-20241121_133215.gz
-rw-r--r--    1 root     root      145.8M Nov 21 13:37 ips.log-20241121_133715.gz
-rw-r--r--    1 root     root      142.5M Nov 21 13:34 ips.log-20241121_133445.gz
-rw-r--r--    1 root     root       34.1M Nov 21 13:47 ips.log-20241121_134627.gz
XGS4500_AM02_SFOS 21.0.0 GA-Build169 HA-Primary# date
Thu Nov 21 13:47:33 CET 2024