Adding a new column to the parser

Here developers can talk about how to write a Parser for LogMX

Moderator: admin

Post Reply
itayrose
Posts: 4
Joined: Wed Jun 03, 2015 12:10 pm

Adding a new column to the parser

Post by itayrose »

Hi,
I'm using the latest version of LogMX and I have a problem.
I'm trying to parse logs sent by log4j2 using syslog appender.
LogMx application parses the incoming info correctly, but doesn't display a pid column (or any other user-defined column).
Note: the logs are correct - I verified that the log sent on the socket contains the pid info. In addition, when I read it from a file (using rollingfileAppender) the LogMX succesfully displaying the pid column.

I also tried to use socketAppender, but the results were the same.

Can you please help me and tell me what I should do to fix it?
Thanks in advance,
Itay
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Adding a new column to the parser

Post by admin »

Hello,

Could you please post here your log4j2 configuration file (especially your "<Appenders><Syslog></Appenders>" part), LogMX parsing greatly depends on how this appender is configured (see https://logging.apache.org/log4j/2.x/ma ... ogAppender for more information).

Also, have you tried to enable/disable the option in "Tools" > "Options" > "Parsers" > "Parsers option" which is called "Syslog: handle PID as part of emitter"?

Xavier
itayrose
Posts: 4
Joined: Wed Jun 03, 2015 12:10 pm

Re: Adding a new column to the parser

Post by itayrose »

Hey,
Thanks for your quick reply,
I have already tried to enable/disable the option "Syslog: handle PID as part of emitter" and it didn't help.
Before I asked the question in the forum, I have looked in the site that you gave me the link to (actually I am dealing with this problem for over a week and I have checked everything)

My log4j2 configuration is:
<Syslog name="Syslog" format="RFC5242" host="localhost" port="4501" protocol="udp" mdcId="mdc">
<loggerFields>
<KeyvaluePair key="l" value="%p"/>
<KeyvaluePair key="t" value="%t"/>
<KeyvaluePair key="m" value="%m"/>
</loggerFields>
</Syslog>

As I have already said, the log contains the pid information. The parsing problem is in LogMX application (only in the socket/syslog parser)

Thanks,
Itay
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Adding a new column to the parser

Post by admin »

Hello,

It seems that there is a typo in your Log4j configuration file: I think you meant "format="RFC5424"" instead of "format="RFC5242"". This typo is causing Log4j to use BSD format (seems to be RFC3164) instead of RFC5424.

Also, it seems that current Syslog Parser of LogMX doesn't support syslogs that doesn't use a colon character (:) to separate the header and the message, like in "<13>Jan 9 1:33:0 10.0.0.99 mount: My message". So I could not test your configuration (with RFC 5424 LogMX badly parsed these logs since the file name contains a ":", and with RFC3164 LogMX did not parse at all). This will be fixed in next LogMX version v5.4.0 that will be released soon.

Also, I did not see any PID in the raw logs received by LogMX from the socket before parsing (with both RFCs). (note: the next version will also display the received raw text if parsing failed)

Knowing that, I propose you to post here the raw logs you received in LogMX (you told me you saw a PID so I guess you found a way to get it, like Wireshark or something else...). I will write a Parser for you (with a PID column) for the format you chose (RFC 3164 or 5424). Or if you prefer, I can contact you in private to send you a patched version of LogMX.

Xavier
itayrose
Posts: 4
Joined: Wed Jun 03, 2015 12:10 pm

Re: Adding a new column to the parser

Post by itayrose »

Hello,
Thanks again for your help,
The typo error which you have find, was not in my log4j2 configuration and it occurred only when I typed in this forum (the configuration is fine).

My logs seem like this log:

<111>1 2015-06-10T09:09:47:744+02:00 MyComputerName c:\mydir\log4j2.xml -- [mdc@1111 c="my logger" d="2015-06-10 09:09:47,744" l="current level" m="myMessage" pid="myProcessName" t="mythread"] myMessage

In addition, as I wrote before, I have already built a parser which successfully handle with this logs when they are written to a file but doesn't handle with them correctly (not adding a pid column) when they are sent through the socket.

The regex parser is:
(?:[^=]*)="([^"]*)"(?:[^=]*)="([^"]*)"(?:[^=]*)="([^"]*)"(?:[^=]*)="([^"]*)"(?:[^=]*)="([^"]*)"(?:[^=]*)="([^"]*)(?:.*)

Thanks a lot,
Itay
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Adding a new column to the parser

Post by admin »

Hello,

Thank you for your feedback. You are right, the column is not displayed for this SyslogAppender, and it is for LocalFileManager. This is now fixed in LogMX v5.4.0 that will be released in July (with many other great new features and improvements). In fact, this SyslogAppender has never been tested on our side with LogMX before you posted your message here: only raw socket connections and Log4j/Logback SocketAppenders were supported and tested at that time. Now LogMX v5.4.0 fully supports SyslogAppender, using RFC 5424 and RFC 3164 (some issues with RFC 5424 format parsing had also been fixed).

There is no simple workaround for you at this time for v5.3.4: the only solution consists in writing a Java LogMX Parser that will insert this PID in any other "native" field. If you need something like this, please let me know, I will provide you such a Parser.

I will let you know when LogMX v5.4.0 is released by posting a message here.

Thank you.
Xavier
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Adding a new column to the parser

Post by admin »

Another user just discovered this bug (viewtopic.php?f=1&t=1694&p=5616#p5615), and proposed a good workaround: adding the missing fields in the "Entry details" panel at the bottom of the main window: using the "Settings" button on this panel's toolbar, you can add all the missing fields in this panel. You can also use the "Export" feature to copy all log entries to a new tab, and all the columns will be displayed in this new tab.
itayrose
Posts: 4
Joined: Wed Jun 03, 2015 12:10 pm

Re: Adding a new column to the parser

Post by itayrose »

Thanks again for your response,
I am looking forward to the next release...
The workaround using the entry details won't help me because I need to filter some of the logs according to my user defined data.
I am already using your second solution (putting it in the emitter "native" field).

Thank you,
Itay
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Adding a new column to the parser

Post by admin »

Hello Itay,

LogMX v5.4.0 is now released and includes (among many other great things), a fix for these issues :)

Please let me know if you have any other trouble or question.

Xavier
Post Reply