Help with JSON parser

Here developers can talk about how to write a Parser for LogMX

Moderator: admin

Post Reply
rodb
Posts: 9
Joined: Tue Mar 10, 2020 12:46 am

Help with JSON parser

Post by rodb » Fri Mar 13, 2020 7:19 pm

Hello

I have tried using the standard JSON parser for the below log file, but it doesn't work. I also tried to create a regex parser, but the problem I think is that the file lacks any newline characters. Logmx picks up the first set of tags, but then logs the rest of the file as the message as if it is all just one entry.

Code: Select all

[{"d":"Thu, 12 Mar 2020 16:29:49 GMT","t":"WARN","m":"[VidyoMediaManager] The sococo camera limit is 4 "},{"d":"Thu, 12 Mar 2020 16:29:49 GMT","t":"WARN","m":"[Vidyo] service initialized "},{"d":"Thu, 12 Mar 2020 16:29:50 GMT","t":"WARN","m":"[vidyoplugin] presence connected: firstload: true, gated: false, failed: false "},{"d":"Thu, 12 Mar 2020 16:31:43 GMT","t":"WARN","m":"$health(Media Connection) is -> Failed ✖"},{"d":"Thu, 12 Mar 2020 16:31:43 GMT","t":"WARN","m":"$health(Media Connection) is repaired ✓"},{"d":"Thu, 12 Mar 2020 16:32:36 GMT","t":"WARN","m":"$health(Media Connection) is -> Failed ✖"}]
This is the regex that sort of works, but only creates one log line:

Code: Select all

.*?\"(\w{3}, \d{2} \w{3} \d{4} \d{2}:\d{2}:\d{2}).*?:\"(\w+?)\".*?:\"(\[.*?\])?(.*)\},?
Any help most appreciated.

thank you, Rod

admin
Site Admin
Posts: 529
Joined: Sun Dec 17, 2006 10:30 pm

Re: Help with JSON parser

Post by admin » Sat Mar 14, 2020 3:07 am

Hello,

Yes, the built-in JSON parser is only compatible with Log4j JSON files, so it is not compatible with this format.
But to be honest, I find this log format really weird, I have never seen a log file composed of a single line containing all the log entries... What about when there are a lot of log entries? Usually, applications parse text/log files line by line (which is the case of LogMX), with a buffer for each line, which makes it really challenging when a single line can be huge (mainly because the size of the line is not known in advance).

Also, It doesn't look efficient to maintain a valid JSON structure because of the high-level array "[]" containing all the entries : how does the logger know that it's the last entry? Even if the final ']' character is written when the application terminates, what about sudden application crash? What about when the application restarts: does the logger remove this final ']' to add more entries? Does it start a new line?

Are you able to change this log format? (just adding a new-line character after each entry should make things way easier). If you cannot change it, we can try using a small JSON parser to parse the giant line and create many log entries from a single line, but I'm not sure if it will properly work, with descent performances.

Please let me know!
Xavier

rodb
Posts: 9
Joined: Tue Mar 10, 2020 12:46 am

Re: Help with JSON parser

Post by rodb » Sun Mar 15, 2020 10:42 pm

Hi Xavier

Ths log file is from the Sococo application and unfortunately, I can't change it. I could pre-format the log file by formatting it with jsonlint.com but that adds an extra step that I was hoping to avoid, especially if the file is larger.

We could also remove the opening and closing square brackets by manually editing the log file, but I'm not sure if that would help.

If you are able to help with creating a small JSON parser to cope with the lack of line endings that would be great.

thank you, Rod

admin
Site Admin
Posts: 529
Joined: Sun Dec 17, 2006 10:30 pm

Re: Help with JSON parser

Post by admin » Mon Mar 16, 2020 1:01 am

Hello Rod,

OK so I gave it a try, and a 2 MB log file consisting of only 1 giant line worked, but I already felt that the parsing was not immediate (~half a second) on a pretty good machine (due to this giant line). Some optimizations may be required.

You will first have to add this JSON parser library in your LogMX "lib" directory:
https://repo1.maven.org/maven2/com/goog ... -2.8.6.jar

Then, import this Parser:
sococo-parser.export
(3.99 KiB) Downloaded 85 times
If you get an error about some missing class "com.google.gson...", please add "lib/gson-2.8.6.jar" on the line "ADDITIONAL_CLASSPATH=" of the file "startup.conf" in the LogMX directory (this error may happen depending on which OS/Java version you're using, and depending on how you start LogMX).

Let me know if you have any issues.
Xavier

rodb
Posts: 9
Joined: Tue Mar 10, 2020 12:46 am

Re: Help with JSON parser

Post by rodb » Mon Mar 16, 2020 8:47 am

Hey Xavier

This actually works pretty well, but is missing capturing the emitter.

This is optional, depending on if it's in the message field or not, and always looks like:

Code: Select all

"m":"[VidyoMediaManager] 
So in this case, the emitter is VidyoMediaManager.

Note, sometimes the message contains [object Object] near the end but we don't want to include that as an emitter.

Many thanks, Rod

admin
Site Admin
Posts: 529
Joined: Sun Dec 17, 2006 10:30 pm

Re: Help with JSON parser

Post by admin » Mon Mar 16, 2020 11:27 pm

Hi Rod,

Got it, here is a new version (you can recognize it in LogMX options, "Parsers" tab, because its version is "v2"):
sococo-parser.export
(4.62 KiB) Downloaded 289 times
Let me know!
Xavier

rodb
Posts: 9
Joined: Tue Mar 10, 2020 12:46 am

Re: Help with JSON parser

Post by rodb » Tue Mar 17, 2020 6:19 pm

Hey Xavier

That works a treat - thank you.

Rod

Post Reply