Log4J Parser for Multi-line message

Here developers can talk about how to write a Parser for LogMX

Moderator: admin

Post Reply
CraigB
Posts: 1
Joined: Thu Mar 15, 2018 11:06 am

Log4J Parser for Multi-line message

Post by CraigB »

Hello

I'm using your tool to analyse log files for an app using Log4J. The parser I'm using is

Code: Select all

%d{ISO8601} SAST %level   %c %mx{version::int}:%mx{subversion::int} [%t] %m%n
It works for almost all the the log files that are created. There are 2 exceptions. The one is a log file that has a message over multiple lines e.g.:

Code: Select all

2018-03-15 00:02:00,542 SAST INFO   log.request 1067:0009 [ebx-logUpdateScheduler] **** Logs Updated (logVersion=1.4) ****
2018-03-15 10:08:19,769 SAST DEBUG  log.request 1067:0009 [ebx-scheduler-worker-1] Request computation took 0 ms: 
  Request[table=[/root/definition/pnode in dataset ResolveParty in snapshot 2018-03-01T12:58:48.771], specificFilter=[./firstInProcess='true']]@64138678
  OptimizationInfo[Using index {pk} /idInProcess; sliceEnd: MAX; requiringTableAccess: true; requiringAdditionalSort: false]@1155ab7c
2018-03-15 10:08:29,895 SAST DEBUG  log.request 1067:0009 [ebx-scheduler-worker-1] Request computation took 0 ms: 
  Request[table=[/root/definition/pnode in dataset ResolveParty in snapshot 2018-03-01T12:58:48.771], specificFilter=[./firstInProcess='true']]@26679d22
  OptimizationInfo[Using index {pk} /idInProcess; sliceEnd: MAX; requiringTableAccess: true; requiringAdditionalSort: false]@3f1abf14
and the other is where the file has a combination of EOL characters, e.g. some lines end with \n and others end with \r\n. In this case, the \r\n lines seem to be ignored and included in the message of the first log message.

Do you have any advice for these 2 scenarios?

Kind regards
Craig
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Log4J Parser for Multi-line message

Post by admin »

Hello Craig,

It seems that the Pattern you're using for your Parser may not be quite right: from the tests I did with your log format, it seems that the Pattern to use should be me more something like this:

Code: Select all

%d{ISO8601} SAST %-6level %c %mx{version::int}:%mx{subversion::int} [%t] %m%n
The difference is I used "%-6level" instead of "%level " (with 2 extra white-space characters at the end). That's because your logs seem to use a left-aligned padded Log level (you can see that in your logs where you have "INFO " with 2 extra white-spaces and "DEBUG " with only 1 white-space). So with the Parser you used, LogMX didn't recognize the DEBUG entries because DEBUG has only 1 extra white-space, so it thought the whole line was the end of the message of the previous log entry.

I then tried to reproduce the issue about \n and \r\n, but it seems fine: I used some \n and some \r\n in the same file but the parsing was correct. Maybe you were confused because of the first issue?

Please let me know if you still have issues (especially concerning \r\n).

Xavier
Post Reply