Auto-refresh and log file rotation

Notes, tips, and other usefull things on how to use LogMX

Moderator: admin

Post Reply
joost
Posts: 12
Joined: Tue Apr 26, 2016 7:53 am

Auto-refresh and log file rotation

Post by joost »

With auto-refresh on (sftp file manager), when the opened log file is rotated, emitter show/hide settings are cleared, and all log entries are removed from the table (only the entries since last rotation are shown). Expectation is that when a log file is rotated, the already read/displayed log entries are unaffected, as well as any filter/emitter/marked settings.

I cannot simply use SFTP directory manager to additionally load the rotated files, because that results in too much log files (>1GB). LogMX can't seem to handle that, even with 2G max memory setting.
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Auto-refresh and log file rotation

Post by admin »

Hello,

Do you mean that if the current file containing the set of entries "A" is rotated to a new set of entries "B", LogMX should display A and B after the rotation? If yes, I think it could induce some wrong behavior when the log file is flushed manually on purpose to start with a new empty log file (e.g. the app that just crashed with a lot of logs is restarted with a new empty log file).

Concerning the memory needed to handle big file, you are right, this is a known fact: since LogMX is written in Java, all strings are internally stored in UTF-16, so you need 2 bytes to store each character (even for simple ASCII characters that use 1 byte in the log file). Moreover, LogMX needs to store some internal model data for log entries (links between emitters tree and entries, entries outline info at the right of log entries table, ...). So in general, LogMX needs a quantity of memory around 2.3 times the size of the log file. But for very large file, we suggest only using AutoRefresh in Tail mode, with alerts if needed.

John
joost
Posts: 12
Joined: Tue Apr 26, 2016 7:53 am

Re: Auto-refresh and log file rotation

Post by joost »

Hi John,

Let me explain. The logs I have to work with look like this:

Code: Select all

-rw-r----- 1 loguser loggroup 10485654 Apr 28 10:00 server-diagnostic-8537.log
-rw-r----- 1 loguser loggroup 10483336 Apr 28 10:06 server-diagnostic-8538.log
-rw-r----- 1 loguser loggroup 10485587 Apr 28 10:11 server-diagnostic-8539.log
-rw-r----- 1 loguser loggroup 10465138 Apr 28 10:19 server-diagnostic-8540.log
-rw-r----- 1 loguser loggroup 10482281 Apr 28 10:33 server-diagnostic-8541.log
-rw-r----- 1 loguser loggroup 10478825 Apr 28 10:48 server-diagnostic-8542.log
-rw-r----- 1 loguser loggroup 10478338 Apr 28 10:57 server-diagnostic-8543.log
-rw-r----- 1 loguser loggroup 10466710 Apr 28 11:14 server-diagnostic-8544.log
-rw-r----- 1 loguser loggroup 10482711 Apr 28 11:24 server-diagnostic-8545.log
-rw-r----- 1 loguser loggroup 10485303 Apr 28 11:34 server-diagnostic-8546.log
-rw-r----- 1 loguser loggroup   848975 Apr 28 11:37 server-diagnostic.log
Log entries are written to "server-diagnostic.log", and whenever that file reaches a certain size (10MB in this case), the log file is rotated, and the oldest log file is purged (deleted). So, after the "server-diagnostic.log" has passed 10MB, the log directory contents will look like this:

Code: Select all

-rw-r----- 1 loguser loggroup 10483336 Apr 28 10:06 server-diagnostic-8538.log
-rw-r----- 1 loguser loggroup 10485587 Apr 28 10:11 server-diagnostic-8539.log
-rw-r----- 1 loguser loggroup 10465138 Apr 28 10:19 server-diagnostic-8540.log
-rw-r----- 1 loguser loggroup 10482281 Apr 28 10:33 server-diagnostic-8541.log
-rw-r----- 1 loguser loggroup 10478825 Apr 28 10:48 server-diagnostic-8542.log
-rw-r----- 1 loguser loggroup 10478338 Apr 28 10:57 server-diagnostic-8543.log
-rw-r----- 1 loguser loggroup 10466710 Apr 28 11:14 server-diagnostic-8544.log
-rw-r----- 1 loguser loggroup 10482711 Apr 28 11:24 server-diagnostic-8545.log
-rw-r----- 1 loguser loggroup 10485303 Apr 28 11:34 server-diagnostic-8546.log
-rw-r----- 1 loguser loggroup 10465138 Apr 28 11:44 server-diagnostic-8547.log
-rw-r----- 1 loguser loggroup   18501 Apr 28 11:45 server-diagnostic.log
Note three things:

1: The oldest log file has been physically deleted
2: What WAS "server-diagnostic.log" has been moved to "server-diagnostic-8547.log"
3: File "server-diagnostic.log" is truncated

Tail mode does come close working with this.I tried:

Image

But I notice the following issues whenever rotation takes place:

1. I lose any marked entries set on entries in server-diagnostic.log file
2. I start seeing "file not found" error icons in the refresh bar (right below entries table)

How can I correctly monitor the log files as explained above?
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Auto-refresh and log file rotation

Post by admin »

Hello Joost,

Sorry, you were right. There was a bug in SFTP-dir when a monitored file was removed. We have now fixed it in an internal build, and now marked entries are still marked after a file has been removed, and (as John said) only the log entries that came from removed files are removed from the main log entries list. Sorry for the inconvenience.

This bug is now fixed in v5.0.0 that will be released soon (one or two weeks), I will let you know when it's available by posting a new message here.

Xavier
joost
Posts: 12
Joined: Tue Apr 26, 2016 7:53 am

Re: Auto-refresh and log file rotation

Post by joost »

admin wrote:only the log entries that came from removed files are removed from the main log entries list
Could you elaborate on this? In the scenario I gave earlier, which log entries would be removed from the main log entries list?
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Auto-refresh and log file rotation

Post by admin »

Hello Joost,

In the scenario you gave earlier, when the oldest log file "server-diagnostic-8537.log" is deleted (your step 1), the log entries that were included in this file are removed from LogMX log entries table. In other words, LogMX removed entries from its display only when they were included in a file that doesn't exist anymore.

I hope that this behavior matches your needs. If it doesn't, maybe what you may want is to add LogMX Alerts to archive entries that are of some interest for you (and also to be alerted). This way these entries won't be lost. Like John said, not removing these entries could lead to inconsistent state and confusing situations for the user. We could technically add an option to avoid the removal of such entries but I'm not sure it would be a good thing. Please let us know what you think.

Xavier
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Auto-refresh and log file rotation

Post by admin »

Hello Joost,

LogMX v5.5.0 is now released, and it fixes this issue :)
Sorry for the delay.

Xavier
Post Reply