Using LogMX with large logfiles

Notes, tips, and other usefull things on how to use LogMX

Moderator: admin

Post Reply
ibenjes
Posts: 2
Joined: Thu Nov 27, 2014 5:18 pm

Using LogMX with large logfiles

Post by ibenjes »

Hi,

Environment: MacBookPrp running Yosemite
I am currently trying LogMX and there are a lot of features I do like however it doesn't seem to work well with very large logfiles or has a Memory leak somewhere.
I had LogMX running for one or two days, monitoring a log from my dev system running JBoss. I've restarted the server a few times for updates while LogMX was still monitoring it.

Today I've noticed that my machine was running low on memory and I checked who the offender was: LogMX with 5.9 GB.
My dev system is generating a lot of log output (level is set to debug). It seems LogMX loads everything into memory.
Is there a way to tell LogMX to only have a 'window' of the last few hundret MB or e.g. 10 000 lines?
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Using LogMX with large logfiles

Post by admin »

Hello,

If LogMX uses more than 50 MB of memory while no file is opened and that you have already clicked on the "Garbage collection" icon (Image) in the memory status panel, then it is indeed a memory leak (bug). If it's your case, please let us know.

If it's not your case (i.e. > 50 MB while files are opened or garbage collection not forced manually), you have two ways to limit the amount of memory LogMX can use :
  1. For each monitored file (i.e. in Auto Refresh mode), you can set the maximum number of log entries to keep in memory using these controls in the AutoRefresh toolbar:
    Image
    Only the N most recent log entries will be kept in memory (1,000,000 in this screenshot), and oldest entries couldn't be displayed anymore.
  2. A more global and brutal way is to change the maximum amount of memory that LogMX can use: in the file "startup.conf" in the LogMX directory, you can set it on the line "MAX_MEMORY=" (see comments just before this line). Note that if you set this parameter to 1 GB, it will mean 1 GB for all opened files in LogMX, plus LogMX internal data. By default, this setting is left empty, which means 600 MB max (or 400 MB max in LogMX v5.2.0 and older). So I guess someone has already modified this parameter on your LogMX install since you managed to use 5.9 GB.
Please let me know if you still have troubles.

Xavier
ibenjes
Posts: 2
Joined: Thu Nov 27, 2014 5:18 pm

Re: Using LogMX with large logfiles

Post by ibenjes »

This is the memory usage reported by LogMX
Memory usage reported by LogMX
Memory usage reported by LogMX
Screen Shot 2014-11-28 at 08.55.21.png (8.31 KiB) Viewed 10399 times
These are some stats reported by LogMX
Screen Shot 2014-11-28 at 08.55.11.png
Screen Shot 2014-11-28 at 08.55.11.png (45.44 KiB) Viewed 10399 times
And this is what the OS X Activity Monitor reports:
Activity Monitor
Activity Monitor
Screen Shot 2014-11-28 at 08.54.57.png (21.74 KiB) Viewed 10399 times
So yes I think there is a big memory leak here.
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Using LogMX with large logfiles

Post by admin »

What bothers me is (for what I can see in your screenshot) that the JVM (Java) reports that LogMX currently uses only 70 MB, but 612 MB are still allocated to LogMX. It means that at some point, you had plenty of loaded log entries (so you used 600 MB at this point), then you flushed or closed some entries and/or files, to get to 70 MB used, but the JVM still keeps 600 MB. This situation can occur, but should not persist over time. So it's more a garbage collection issue than a memory leak in my opinion (but I agree it must be fixed anyway). This is problem number 1. I can see 3 answers to this problem:
  1. You didn't wait enough time before taking the screenshot (very likely)
  2. You didn't force a manual garbage collection before taking the screenshot (click on trash can icon, several clicks may also force the VM to release some more memory)
  3. The policy of your automatic garbage collection process of your VM doesn't want to release this memory yet. Maybe this policy should be enforced by LogMX (currently, LogMX uses the default garbage collection policy)
Problem number 2 is that the OS X Activity Monitor reports more than 1 GB allocated to LogMX, instead of something like 650 MB (612 for LogMX + ~ 30 for VM internal stuff).

Could you please tell me which version of Java and LogMX you are using ("Help" > "About" > blue "i" icon > "java.version") ? Also, could you please tell me if the items 1. and 2. in the above list are right so that we can fix this issue?

Xavier
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Using LogMX with large logfiles

Post by admin »

Hello,

After some further research, it seems that problem 1 is due to Java default behavior when using a 64 bits JDK/JRE on a multi-processors host (=> using a parallel GC that doesn't release memory back to the OS, see http://stackoverflow.com/questions/1191 ... at-runtime and http://stackoverflow.com/questions/6755 ... mory-to-os).
Indeed, now 64-bits JDKs/JREs are by default only supporting "server" VM (instead of 32-bits JDKs/JREs which provides "client" and "server" VMs, and using the "client" VM by default). And this "server" VM is tuned for high availability applications: using more RAM, for enhanced performances, whereas the "client" VM tries to use a smaller memory footprint instead. The good news is that the behavior of "server" VM can be tuned to request it to give back memory to the OS when possible: in the file "logmx.command" that you use to start LogMX, update the last line to use something like:

Code: Select all

java -XX:+UseSerialGC -Xminf0.5 -Xmaxf0.5 -Xmx$MAX_MEMORY .........
(add "-XX:+UseSerialGC -Xminf0.5 -Xmaxf0.5" just before "-Xmx$MAX_MEMORY")

Please note that:
  1. this works for Oracle and IBM Java VMs, please tell me it doesn't work, you may use another VM (see output of "java -version")
  2. you will still need to force (maybe several times) a garbage collection using the LogMX memory panel to release the memory (next LogMX version will automatically force a GC when closing a file containing a great number of log entries)
The next LogMX version will allow you to apply this patch directly in its startup configuration file "startup.conf", and this workaround will be documented (cannot use this fix by default in LogMX distribution because it is VM dependent).

As for problem 2 (OS X Activity Monitor report), I just don't get it. I found something that may be related: https://forum.xojo.com/16483-yosemite-u ... ry-fixed/0. Maybe you can look at this Activity Monitor just after LogMX starts, without any opened file in LogMX, just to see if it matches LogMX memory status. Or maybe applying the above workaround will fix this displayed value...

Please let me know.

Xavier
Post Reply