Regex, OR, multiline and mixed logs

Here developers can talk about how to write a Parser for LogMX

Moderator: admin

Post Reply
Chemo
Posts: 2
Joined: Thu Apr 16, 2020 8:51 am

Regex, OR, multiline and mixed logs

Post by Chemo »

Hi.

I really, really appreciate this LogViewer!
I have one problem concerning multiline log messages, like:

Code: Select all

10:52:15,123 FATAL Exception 1 thrown
caused by Exception 2
caused by Exception 3
with a multiline message block.

RegEx Parser can read these multiline blocks (is this a new feature?), like:

Code: Select all

(\S:\SS:\S,\S) (\S) (.*)
for (Timestamp) (Level) (Message)

So everything great!

However, I have am mixed log with different log messages, like:

Code: Select all

10:52:15,123 FATAL Exception 1 thrown
caused by Exception 2
caused by Exception 3
DEBUG 10-52-16-456 This is another 
multiline message
in different format
Which can be fetched with an OR-RegEx like

Code: Select all

(\S:\SS:\S,\S) (\S) (.*) | (\S) (\S-\S-\S-\S) (\S)
for (Timestamp) (Level) (Message) | (Level) (Timestamp) (Message)

(Awesome by the way)

But NOW suddnely the expression does not catch multiline blocks anymore. :(

For the Example ebove it would now only catch the first lines:
-> 10:52:15,123 FATAL Exception 1 thrown
-> DEBUG 10-52-16-456 This is another

Is this a bug or am I missing something?

Thank you!
Chemo.
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Regex, OR, multiline and mixed logs

Post by admin »

Hello Chemo,

Happy to hear you like LogMX ;-)

In order to parse the given text

Code: Select all

10:52:15,123 FATAL Exception 1 thrown
caused by Exception 2
caused by Exception 3
using a Regex parser, I would use this Regex instead:

Code: Select all

(\S+?:\S+?:\S+?,\S+?)\s+?(\S+)\s+?(.*)
Basically, when you want to match multiple characters, \S+ should be used instead of \S (and adding the extra ? at the end is just for better performances, so in the end \S+? will match "10" for example). For more information about Regular expressions, you can check https://logmx.com/docs/regex-parsers.html.

Concerning the mixed logs, you cannot do it using a RegexParser because the order of the matched fields is different: "Date,Level" and "Level,Date". You can indeed use the | character to match different patterns, but the fields order must be the same.

That being said, LogMX can still parse such mixed logs, but not through a Regex Parser: you have to use a Java Class parser in order to interpret different fields orders. You can read more about such parsers here: https://logmx.com/parser-dev. If you need help writing your Java Class Parser, I can also write it for you, let me know ;-)

Xavier
Chemo
Posts: 2
Joined: Thu Apr 16, 2020 8:51 am

Re: Regex, OR, multiline and mixed logs

Post by Chemo »

Thank you for the quick answer.

Right, I forgot the +. (It was just an example, my actual case is a ittle bit more complex.) Also thanks for the performance advice.

However, the |-operator does seem to work very well with different orders of fields in different patterns. At least the log maps the blocks correctly to the fields. Only problem was that it stopped reading multiple lines per log entry. The problem also occures with same order patterns.

I played around a little more and realized it's only the third line and following that gets lost. So the first line break is captured.

A more simple example, with same order fields:

For

Code: Select all

INFO This is a 
multiline
message
[INFO] This is another
multiline
message
the RegEx parser (INFO) (.*) with (Level) (Message) will correctly capture one message:

Code: Select all

This is a 
multiline
message
[INFO] This is another
multiline
message
the RegEx parser \[(INFO)\] (.*) with (Level) (Message) will correctly capture one message:

Code: Select all

This is another
multiline
message
but the RegEx parser (INFO) (.*)|\[(INFO)\] (.*) finds two messages but only the first two lines of each:

Code: Select all

This is a 
multiline

Code: Select all

This is another 
multiline
However, I will try out the Java Parsers.

Thank you very much!
Chemo.
Post Reply