Regex parser, timestamp with milliseconds and timezone, time is parsed wrong

Here developers can talk about how to write a Parser for LogMX

Moderator: admin

Post Reply
Alonzzo2
Posts: 3
Joined: Wed Sep 02, 2020 9:40 am

Regex parser, timestamp with milliseconds and timezone, time is parsed wrong

Post by Alonzzo2 »

Hi,
This is my log example:
{"Subject":"Unclassified","Time":"2020-09-02T04:25:31.1924648+03:00","Level":"INFO","ClassName":"blabla","Thread":"21","Message":"yadayday"}

This is my parser:
\{"Subject":"(.*?)","Time":"(.*?)","Level":"(.*?)","ClassName":"(.*?)","Thread":"(.*?)","Message":"(.*?)"\}

This is my date format (I tried countless variations after the 'ss' part):
yyyy-MM-dd'T'HH:mm:ss.SSSSSSSXXX

The parser indicates that the date and time were parsed successfully, but the result is:
02/09/2020 04:57:35
Notice that the parsed time isn't related to the original log's time

I know the result format comes from the Display/Display entries date with custom format configuration, so this is what I configured there:
dd/MM/yyyy HH:mm:ss

Can someone assist me?
Thanks!
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Regex parser, timestamp with milliseconds and timezone, time is parsed wrong

Post by admin »

Hello,

I believe the dates in these logs don't use milliseconds, but probably nanoseconds (because there are at least 7 digits after the seconds). And as mentioned in the LogMX documentation, only milliseconds are handled. So LogMX interprets this as 1,924,648 milliseconds, which is around 32 minutes, hence the time difference in the parsed date.
Sorry then, but for now, you will have to use milliseconds in the logs to use such parser (in this case your date format will end with "SSSX"), or use a "Java Class Parser" instead of this "Regex Parser" so that you can parse the date as nanoseconds.
We plan to support nanoseconds for non-Java class Parsers later on, but we don't know yet when it will be available. We will post a message in this thread as soon as we have some updates.

Xavier
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Regex parser, timestamp with milliseconds and timezone, time is parsed wrong

Post by admin »

Actually, let me help you even further: here is a Java Class Parser I wrote for you to handle nanoseconds dates:
logmx.parser.export
(4.01 KiB) Downloaded 343 times
To import it, go to Options > Parsers, and click on the import button at the bottom right corner, and select the file you downloaded from the above link. You can now disable or remove the Parser you already created, and open your logs normally.
The attached parser comes with the source code and compiled class (so that you can modify it). Once imported, you will find both in the LogMX "parsers" directory (see https://logmx.com/parser-dev for more help about Java Class Parsers).

Let me know if you have any question!

Xavier
Alonzzo2
Posts: 3
Joined: Wed Sep 02, 2020 9:40 am

Re: Regex parser, timestamp with milliseconds and timezone, time is parsed wrong

Post by Alonzzo2 »

Wow.
1. Thanks for the intention of letting me know once you'll have any news regarding support for nano seconds
2. Thanks for noticing these are probably nano seconds :)
3. MANY THANKS for compiling a parser for me!! I really appreciate it!!

I had no problems using it, but, at the 'display entries date with custom format', I can't display milliseconds, since I guess the date pattern you used ignores it?

I'll do some reading about how to create a custom parser and check if I can modify the parser you wrote for me to convert the nano seconds field to milliseconds and use it.
I'll also ask my team to try using a different pattern for out logs. It's weird we're even using nano seconds...

Thank you very much!!
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Regex parser, timestamp with milliseconds and timezone, time is parsed wrong

Post by admin »

Hello,

No worries, happy to help you use LogMX :)
When you say you "can't display milliseconds at the 'display entries date with custom format'", do you mean the timestamps in the main logs view doesn't display milliseconds as indicated by this Display option? or maybe your intent was to display nanoseconds? On my end, I'm able to display milliseconds for these logs+parser: (nanoseconds are not supported when using this "Display" option, but if you turn off this option, the original date string coming from the logs will be displayed, including nanoseconds if they include nanoseconds)
Image1.png
Image1.png (5.07 KiB) Viewed 8356 times
Image2.png
Image2.png (2.24 KiB) Viewed 8356 times
Xavier
Alonzzo2
Posts: 3
Joined: Wed Sep 02, 2020 9:40 am

Re: Regex parser, timestamp with milliseconds and timezone, time is parsed wrong

Post by Alonzzo2 »

My bad, you're right, it works.
Must have confused with all the parsers and datetime formats I played with.
It works perfectly, thanks!
Post Reply