Writing a regex parser

Here developers can talk about how to write a Parser for LogMX

Moderator: admin

Post Reply
Camelia
Posts: 5
Joined: Thu Oct 06, 2016 6:24 am

Writing a regex parser

Post by Camelia » Thu Oct 06, 2016 6:52 am

Hello,

I am trying to use this tool, seems to be what I need, but I need some help. I have an error log file, with multiple kind of errors, is it possible to create a parse to do this things?
- search only ERROR and count them after message (there are multiple kind of errors, so I need to know the repeat number for each kind of them)
- to color with different colors each kind of error

Code: Select all

08:35:36,112 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (ajp-/127.0.0.1:8010-3) ORA-02292: integrity constraint (PMTOOLS.SYS_C0023760) violated - child record found

ORA-02292: integrity constraint (PMTOOLS.SYS_C0023760) violated - child record found
: java.sql.SQLIntegrityConstraintViolationException: ORA-02292: integrity constraint (PMTOOLS.SYS_C0023760) violated - child record found
10:09:39,387 ERROR [stderr] (ajp-/127.0.0.1:8010-47) ClientAbortException:  java.net.SocketException: Broken pipe
10:09:39,388 ERROR [stderr] (ajp-/127.0.0.1:8010-47) 	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
10:09:39,388 ERROR [stderr] (ajp-/127.0.0.1:8010-47) 	at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450)
10:09:39,388 ERROR [stderr] (ajp-/127.0.0.1:8010-47) 	at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351)
10:09:39,388 ERROR [stderr] (ajp-/127.0.0.1:8010-47) 	at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:426)
10:09:39,389 ERROR [stderr] (ajp-/127.0.0.1:8010-47) 	at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:415)
10:09:39,389 ERROR [stderr] (ajp-/127.0.0.1:8010-47) 	at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
10:26:32,750 ERROR [com.siemens.ic.pmtools.services.pdf.internal.MilestoneMeetingReportPdfGenerator] (ajp-/127.0.0.1:8010-41) QuestionType is null in Question 134384099 in QuestionPool 124931799
10:26:32,781 ERROR [com.siemens.ic.pmtools.services.pdf.internal.MilestoneMeetingReportPdfGenerator] (ajp-/127.0.0.1:8010-41) QuestionType is null in Question 134384149 in QuestionPool 124931849
11:03:23,968 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/pmtools].[Faces Servlet]] (ajp-/127.0.0.1:8010-51) JBWEB000236: Servlet.service() for servlet Faces Servlet threw exception: java.io.IOException: Processing of multipart/form-data request failed. null
	at org.primefaces.webapp.MultipartRequest.parseRequest(MultipartRequest.java:70) [primefaces-3.5.25.jar:3.5.25]

I am looking forward to get your answer.
Thank you.

admin
Site Admin
Posts: 419
Joined: Sun Dec 17, 2006 10:30 pm

Re: Writing a regex parser

Post by admin » Thu Oct 06, 2016 11:22 am

Hello,

I'm not sure to understand "search only ERROR and count them after message".
Concerning the color, you can set a color for each different log level (e.g. "ERROR"). All log entries with "ERROR" level will use the same color. What do you call "error" here?

Xavier

Camelia
Posts: 5
Joined: Thu Oct 06, 2016 6:24 am

Re: Writing a regex parser

Post by Camelia » Thu Oct 06, 2016 11:59 am

Hello,

Before, I copied only a small part from my log file, but my log file can have also info, warning logs, this one should be missed.

Code: Select all

00:00:02,888 INFO  [stdout] (SimpleThreadPoolWorker-3) true
 00:00:46,019 WARN  [org.hibernate.hql.internal.ast.QueryTranslatorImpl] (ajp-/127.0.0.1:8010-70) HHH000104: firstResult/maxResults specified with collection fetch; applying in memory!
 
I created a parser, which searches after ERROR word, but now I need to count them, all which are the same (to not have duplicates) and also make a difference between them and I need to color with different colors even if all of them are ERROR.
This is may parser: (\S+ ERROR) (.*?) (.*?) (.*?), but I don't know how to obtain what I need.
Thank you very much for your replay.

admin
Site Admin
Posts: 419
Joined: Sun Dec 17, 2006 10:30 pm

Re: Writing a regex parser

Post by admin » Thu Oct 06, 2016 9:42 pm

Hello,

Sorry but I'm not sure to fully understand what you'd like to achieve. If you want to know the number of log entries that have the same log message (I assume this is what you meant by "are the same"), you can right-click on this entry, go into the "Filter entries..." popup menu item, and then "Display entries having same Message". Then you can check the number of entries at the bottom right-hand corner (number of displayed entries, number of selected entries). If you want to automate this and get the number of different error log message for all your log file at once, you can easily do it using a LogMX Java Class Parser. You could for example get something like: "Error X happens 150 times, Error Y happens 61 times, ..." You can read more about Java Parsers here: http://www.logmx.com/parser-dev. Please let me know if you need help to set up your Java class Parser.

Concerning the color, I'm afraid there is no way, for now, to have several ERROR entries with different colors. Yet, this will be possible with a feature (Highlighters) that is on our road-map (no precise release date for now).

Xavier

Camelia
Posts: 5
Joined: Thu Oct 06, 2016 6:24 am

Re: Writing a regex parser

Post by Camelia » Mon Oct 10, 2016 12:57 pm

Hello Xavier,

First of all I want to thank you for your quick replay. :)
Last week when I tried to set different colors (before to ask you) I did a lot of configurations, so, now I see logs but are not colored at all (before I saw them colored with red). How can I set the color again?

I assume that I am not able to see all my log errors, my log files are big (around 200.000 lines), after I use my parser, there are visible only 36 errors, but I counted more than this value. Is there maximum number of errors?

My last question, how can I ignore errors which contains different messages? (e.g "JBWEB000236: Servlet.service()", "receivers left so skip this message: Betreff", "WELD-000215 Attempt to call end() on a transient conversation"....). Is it possible?

Thank you in advance.

admin
Site Admin
Posts: 419
Joined: Sun Dec 17, 2006 10:30 pm

Re: Writing a regex parser

Post by admin » Tue Oct 11, 2016 2:59 am

Hello,

Concerning the lost log levels colors, to go back to the default log levels configuration, you can import the file "logmx.levels.export" attached here:
logmx.levels.export
(466 Bytes) Downloaded 27 times
To do this, download this file by clicking on the link above, then in LogMX Options, tab "Levels", click on the button at the bottom right-hand corner to import this file. If you still have trouble, remove all your levels before importing this file.

Concerning the missing errors, there is no real maximum number of errors to display. You are only limited by the quantity of RAM you have. And in my opinion, 200.000 lines is not a lot, so RAM should not be your issue here. Are you using AutoRefresh? If yes, then you may want to check if you have set a limit on the number of entries to display, as shown in the image below:
AutoRefresh.png
AutoRefresh.png (16.95 KiB) Viewed 2989 times
If it still doesn't work, check if you have set any filter, hidden Emitters, or if your log level slider (bottom left-hand corner of main window) doesn't hide any level.

And finally, to hide log entries that contain some specific messages, you can right-click on this entry, then option "Filter entries", then "Display entries having different Message".

Xavier

Camelia
Posts: 5
Joined: Thu Oct 06, 2016 6:24 am

Re: Writing a regex parser

Post by Camelia » Tue Oct 11, 2016 7:35 am

Hello Xavier,

I have trouble with my log parser, when I load my log file I got messages "...unknown level". I got a table with errors (166) not colored with red, even if I imported file you recommended me). I am not very sure if what I got it's correct.
Can I send you my entire log file, maybe you can help me to construct parser correctly. Can you give me an e-mail address?

Thank you very much.

Camelia
Posts: 5
Joined: Thu Oct 06, 2016 6:24 am

Re: Writing a regex parser

Post by Camelia » Tue Oct 11, 2016 8:11 am

Xavier,

When I use filter "display..." I got error message "Invalid regular expression syntax on filter #1:".
What I am doing wrong?

admin
Site Admin
Posts: 419
Joined: Sun Dec 17, 2006 10:30 pm

Re: Writing a regex parser

Post by admin » Tue Oct 11, 2016 11:14 am

Hello,

Please send me your log file and your current regex Parser you are using to support@logmx.com (with "Forum regex parser" as e-mail title). To export your Parser to an e-mail, simply select it in the Options>Parsers window, then click on the Export button (at the right).

Concerning the error you have when Filtering, could you please copy/paste (here or in your mail) the log entry you are clicking on to create the filter, then take a screen shot of the error message, including the filter created for you above log entries.

Thank you.
Xavier

Post Reply