Writing a regex parser to Parse multiple lines

Here developers can talk about how to write a Parser for LogMX

Moderator: admin

logmxuser
Posts: 4
Joined: Tue Jan 31, 2017 8:11 pm

Writing a regex parser to Parse multiple lines

Postby logmxuser » Tue Jan 31, 2017 8:17 pm

Hi,

I have a following block which I want to parse

2017-01-25 01:55:43.883 [ERROR]: transport error:
System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at LibTransport.NetworkStreamTransport.Send(IMessage message, ResponseDataHandler responsehandler, Object obj, Int32 timeout)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at LibTransport.NetworkStreamTransport.Send(IMessage message, ResponseDataHandler responsehandler, Object obj, Int32 timeout)

2017-01-25 01:55:43.883 [DEBUG]: TransportError(): Transport error: 'BindFailure', message: 'transport error (System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at LibTransport.NetworkStreamTransport.Send(IMessage message, ResponseDataHandler responsehandler, Object obj, Int32 timeout))
: at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at LibTransport.NetworkStreamTransport.Send(IMessage message, ResponseDataHandler responsehandler, Object obj, Int32 timeout)'


How do I configure the regex parser to parse it so that it appears correctly? The one in bold is the complete message

<Timestamp> <Level> <Message>

Below is my string but it does not work well

(\S+ \S+) (.*?) (.*?)\n

Thanks,

G

admin
Site Admin
Posts: 418
Joined: Sun Dec 17, 2006 10:30 pm

Re: Writing a regex parser to Parse multiple lines

Postby admin » Tue Jan 31, 2017 11:13 pm

Hello,

In fact, you don't need the final \n, and you may want to avoid capturing the [] around the log level. I tried with this regex and it worked with the logs you gave in your message:

Code: Select all

(\S+ \S+) \[(\S+?)\]: (.*)


For your convenience, I've exported my Parser in a file you can download here:
logmx.parsers.export
(404 Bytes) Downloaded 28 times
(to import it in LogMX, go to Options > Parsers > then click on the Import button at the bottom right-hand corner)

Xavier

logmxuser
Posts: 4
Joined: Tue Jan 31, 2017 8:11 pm

Re: Writing a regex parser to Parse multiple lines

Postby logmxuser » Wed Feb 01, 2017 2:27 pm

Thanks Xavier. Will this be able to handle following also?

2017-01-25 01:55:43.883 [ERROR]: transport error:

***********
Here is a banner
**********

2017-01-25 01:56:43.883 [DEBUG]: Syslog logs

logmxuser
Posts: 4
Joined: Tue Jan 31, 2017 8:11 pm

Re: Writing a regex parser to Parse multiple lines

Postby logmxuser » Wed Feb 01, 2017 3:16 pm

SO I verified that also works. Thanks a lot.

One more thing for now. Here is another log. I would like to merge this view with the first log I provided.

The parser you mentioned would not work for this

(\S+ \S+) \[(\S+?)\]: (.*)

Is there a parser which can work on both of these simultaneously so that I can see the merged view?

2017-01-24 17:07:24.271 : [in] Type 'help' or '?' for a list of commands...
2017-01-24 17:07:24.271 : [in]
2017-01-24 17:07:24.275 : [in] INFO System Ready
2017-01-24 17:07:26.735 : [in] INFO :Connected to controller at 192.168.2.2/51903
2017-01-24 17:07:27.327 : [in] Received Request

Thank you again

admin
Site Admin
Posts: 418
Joined: Sun Dec 17, 2006 10:30 pm

Re: Writing a regex parser to Parse multiple lines

Postby admin » Wed Feb 01, 2017 11:10 pm

Hello,

You could parse these two different log formats with a single Parser, but not with a "Regex" Parser. You would have to use a "Java Class" Parser: by writing a single Java class, you could allow the two formats for a single Parser. But using a single Parser to handle these two formats makes sense only if you have these two different formats within the same log file (LogMX can use only one Parser for each log file). If it's not the case (i.e. each format is used in a separate log file), then you should create 2 different Parsers: one for each format.

Please let me know if you need some help to setup a Java Class Parser, or another Regex Parser (in order to help you I would need to know what "[in]" means in the other log).

Xavier

logmxuser
Posts: 4
Joined: Tue Jan 31, 2017 8:11 pm

Re: Writing a regex parser to Parse multiple lines

Postby logmxuser » Thu Feb 02, 2017 4:02 pm

Thanks Xavier. I created two different "regex parsers" for now to handle these

[in] is the in or out messages sent by the system. They can be part of the message. This is the parser I wrote for [in] messages

(\S+ \S+) (.*?) (.*?)

admin
Site Admin
Posts: 418
Joined: Sun Dec 17, 2006 10:30 pm

Re: Writing a regex parser to Parse multiple lines

Postby admin » Thu Feb 02, 2017 11:15 pm

Hello,

While this Regex may work, I would have used:

Code: Select all

(\S+ \S+) : (.*)

in order to ignore ':' ([in] will be included in message)


Return to “Writing a Parser”

Who is online

Users browsing this forum: No registered users and 1 guest