[NC-52120] Daily report - Reports are received every day at different time - bug

As you were told earlier, the reports will start at the time you wish, with the criteria you wish. The fact that the reports arrive at different times will be a factor of how busy the unit is at the time it is preparing the report.

Control plane should never have priority over data plane.

Stuart

Stuart,

a suggestion:

since the daily report correspond to the previous day, XG can create the report at midnight when no one is connected of before the scheduled time to send the report and at the schedule time send the report. This was one of the things that many customers experienced and it is the "business card" of XG for trial customers too.

Many times, why the daily report arrives at different time each day?

You is how it works today but it is not a nice way to present Sophos to new customers.

Regards

I think you are missing the point a little.

The report is produced at midnight, and takes as long as resources allow. Are you telling us that the Customer is complaining because each report doesn't arrive at the same time? If there is more data to process today than yesterday, would you expect that to take the same time to process?  This to my mind is a somewhat bogus complaint and by any definition is not a bug.

As long as the daily report is available to review NBD I dont see the issue.

12 hours to produce a report?

If the delivery time is 9:00 but XG sends when it can, it is unprofessional! In the daily report settings, put that the report will be sent starting from this hours. Some delays can occur due to the XG resource usage or something similar.

I am a consultant now, but when I was working as a System Admin, I read daily reports always at the same time.

Luk, can you send some screenshots of the Daily Report configuration please.

 I made an assumption regarding the report being produced at midnight, given it was a daily report. I certainly havent seen the issue of delayed reporting that you are talking of here.

Stuart

Also, the email should be sent at 11 and not sometime in 12:xx range

Thanks, and these reports are being mailed to local recipients or through intermediate MTA's?

Thanks, and these reports are being mailed to local recipients or through intermediate MTA's?

External SMTP server.

Luk, I have kicked off an investigation, but going back to your original screenshot it shows reports being delivered to a mailbox at 1-2 hrs after the 'send at' time. Could it be external delivery delays that are also additive??

I do not think so, as failed logins, appliance restart, backup configuration are not delayed at all.

I tried with local XG smtp settings. Same thing!

Hi Luk,

Circling back to this issue.

This is an example of terminological inexactitude :) In other words "Send e-Mail at" doesn't mean what you (we) think it means.

The report scheduling system looks for reports being scheduled every hour and if it finds a report has been scheduled it starts to produce that report. The report can take variable amounts of time depending upon amount of data, system load, other resource constraints like CPU and memory.

So "Send e-Mail at" actually means "Begin producing report at"

I am going to work with engineering to understand how we can best resolve this UI language issue - I doubt we will resolve this before GA as I don't think its as easy as simply changing the words. Unless you think a change of wording and documentation would suffice?

I know this has been something you've reported for a while now, so thank you for your patience, we will resolve the confusion. However, as it stands the reports will still be received at variable times depending on the circumstances above, so my best advice at this time is if you want to see a report at a specific time, schedule it 2 hours earlier than you need it.

Unknown said:

I am going to work with engineering to understand how we can best resolve this UI language issue - I doubt we will resolve this before GA as I don't think its as easy as simply changing the words. Unless you think a change of wording and documentation would suffice?

I know this has been something you've reported for a while now, so thank you for your patience, we will resolve the confusion. However, as it stands the reports will still be received at variable times depending on the circumstances above, so my best advice at this time is if you want to see a report at a specific time, schedule it 2 hours earlier than you need it.

PMStuart this is an example of XG bad design. I think that you should investigate to re-engineer the process and make sure we receive the report at scheduled time and changing the work is not a nice workaround. Time is time. So, all the time, we need to explain to customers: "yes, it is by design. As you can read, the report starts to be calculated at that time and you will receive the report each day at different time." This is already being explained to them.

Now: think about how to fix the issue properly (no workaround) but as I showed, the report is from the day before, so you can:

• really generate the report at the time of the scheduled time and send it within 1/3 minutes OR
• generate the report at midnight and save it somewhere in the /var/log or /var/reports and send it at the scheduled time

Up to you. In my case, I am receiving reports even after more than 1 hour, there are more than something that you need to re-engineer as I have 3/5 IPs at home and few rules. If extracting logs from the DB requires so much time...I am sure you need to go back to design phase and understand how to built a new reporting module. This is not a bad idea, as logging and reporting (at the moment) are the worst topic on XG.

With UTM 9 SG650 with 4000 users, daily report arrives within 3 minutes. Just to let you know some differences.

Thanks

I have to agree with Luk. There is something seriously wrong with the database if it can't generate simple reports within a few minutes. A correctly sized system shouldn't be so overwhelmed that it can't generate a report promptly. If there is a cron job that starts a report every hour on the scheduled time, maybe you guys can change the cron timing and start the report 15-20 minutes before the gui calls for it for daily reports and then queue it for mail at the scheduled time. Just throwing ideas here since a KB article saying a 9am report doesn't mean 9am, it could be 9 or 10 or whatever depending on system load is not something an end user wants to hear.

Thanks for your efforts on this by the way.

To be honest, I prefer that they fix the problem at source. As you said, if XG cannot generate reports in few minutes, something must be revisited or built from scratch.

I was going to post about central reporting but alas; they just need to fix it on box.

Something so simple yet nobody seems to see the point of a scheduled report?  If it's this big of an issue, maybe rip out on-box reports all together and use that space/horsepower for log archives, verbose logging, etc.  Force everyone to central for advanced metrics and reporting.  I know this isn't a solution...just my frustrated opinion.

Hello Stuart Hatto,

I apologize, but I will repeat again: could your developers use what do you have at home?

I mean reporting module in UTM v9. In the past, this system (I guess sometimes v8.3 or v8.4) also had problems with fast enough report generation, and the developers solved this problem quite neatly - and this system is still in use for report generation in v9!

The principle is trivially simple, at regular intervals cron deamon assigns a reporting module to continuously pre-process report data in small portions during the day so that when the report is to be processed for the whole period (day or week) the system is not extremely busy and generate reports in the required time. That shouldn't be a problem to apply in v18 EAP1?!?

And again my advice is free ....

Regards

alda

Unknown said:

Hi Luk,

Circling back to this issue.

This is an example of terminological inexactitude :) In other words "Send e-Mail at" doesn't mean what you (we) think it means.

The report scheduling system looks for reports being scheduled every hour and if it finds a report has been scheduled it starts to produce that report. The report can take variable amounts of time depending upon amount of data, system load, other resource constraints like CPU and memory.

 

So "Send e-Mail at" actually means "Begin producing report at"

 

I am going to work with engineering to understand how we can best resolve this UI language issue - I doubt we will resolve this before GA as I don't think its as easy as simply changing the words. Unless you think a change of wording and documentation would suffice?

I know this has been something you've reported for a while now, so thank you for your patience, we will resolve the confusion. However, as it stands the reports will still be received at variable times depending on the circumstances above, so my best advice at this time is if you want to see a report at a specific time, schedule it 2 hours earlier than you need it.