Writing a regex parser to Parse multiple lines

Here developers can talk about how to write a Parser for LogMX

Moderator: admin

Post Reply
logmxuser
Posts: 4
Joined: Tue Jan 31, 2017 8:11 pm

Writing a regex parser to Parse multiple lines

Post by logmxuser »

Hi,

I have a following block which I want to parse

2017-01-25 01:55:43.883 [ERROR]: transport error:
System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at LibTransport.NetworkStreamTransport.Send(IMessage message, ResponseDataHandler responsehandler, Object obj, Int32 timeout)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at LibTransport.NetworkStreamTransport.Send(IMessage message, ResponseDataHandler responsehandler, Object obj, Int32 timeout)

2017-01-25 01:55:43.883 [DEBUG]: TransportError(): Transport error: 'BindFailure', message: 'transport error (System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at LibTransport.NetworkStreamTransport.Send(IMessage message, ResponseDataHandler responsehandler, Object obj, Int32 timeout))
: at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at LibTransport.NetworkStreamTransport.Send(IMessage message, ResponseDataHandler responsehandler, Object obj, Int32 timeout)'


How do I configure the regex parser to parse it so that it appears correctly? The one in bold is the complete message

<Timestamp> <Level> <Message>

Below is my string but it does not work well

(\S+ \S+) (.*?) (.*?)\n

Thanks,

G
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Writing a regex parser to Parse multiple lines

Post by admin »

Hello,

In fact, you don't need the final \n, and you may want to avoid capturing the [] around the log level. I tried with this regex and it worked with the logs you gave in your message:

Code: Select all

(\S+ \S+) \[(\S+?)\]: (.*)
For your convenience, I've exported my Parser in a file you can download here:
logmx.parsers.export
(404 Bytes) Downloaded 554 times
(to import it in LogMX, go to Options > Parsers > then click on the Import button at the bottom right-hand corner)

Xavier
logmxuser
Posts: 4
Joined: Tue Jan 31, 2017 8:11 pm

Re: Writing a regex parser to Parse multiple lines

Post by logmxuser »

Thanks Xavier. Will this be able to handle following also?

2017-01-25 01:55:43.883 [ERROR]: transport error:

***********
Here is a banner
**********

2017-01-25 01:56:43.883 [DEBUG]: Syslog logs
logmxuser
Posts: 4
Joined: Tue Jan 31, 2017 8:11 pm

Re: Writing a regex parser to Parse multiple lines

Post by logmxuser »

SO I verified that also works. Thanks a lot.

One more thing for now. Here is another log. I would like to merge this view with the first log I provided.

The parser you mentioned would not work for this

(\S+ \S+) \[(\S+?)\]: (.*)

Is there a parser which can work on both of these simultaneously so that I can see the merged view?

2017-01-24 17:07:24.271 : [in] Type 'help' or '?' for a list of commands...
2017-01-24 17:07:24.271 : [in]
2017-01-24 17:07:24.275 : [in] INFO System Ready
2017-01-24 17:07:26.735 : [in] INFO :Connected to controller at 192.168.2.2/51903
2017-01-24 17:07:27.327 : [in] Received Request

Thank you again
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Writing a regex parser to Parse multiple lines

Post by admin »

Hello,

You could parse these two different log formats with a single Parser, but not with a "Regex" Parser. You would have to use a "Java Class" Parser: by writing a single Java class, you could allow the two formats for a single Parser. But using a single Parser to handle these two formats makes sense only if you have these two different formats within the same log file (LogMX can use only one Parser for each log file). If it's not the case (i.e. each format is used in a separate log file), then you should create 2 different Parsers: one for each format.

Please let me know if you need some help to setup a Java Class Parser, or another Regex Parser (in order to help you I would need to know what "[in]" means in the other log).

Xavier
logmxuser
Posts: 4
Joined: Tue Jan 31, 2017 8:11 pm

Re: Writing a regex parser to Parse multiple lines

Post by logmxuser »

Thanks Xavier. I created two different "regex parsers" for now to handle these

[in] is the in or out messages sent by the system. They can be part of the message. This is the parser I wrote for [in] messages

(\S+ \S+) (.*?) (.*?)
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Writing a regex parser to Parse multiple lines

Post by admin »

Hello,

While this Regex may work, I would have used:

Code: Select all

(\S+ \S+) : (.*)
in order to ignore ':' ([in] will be included in message)
Post Reply