Colorization of regexp within log messages

Notes, tips, and other usefull things on how to use LogMX

Moderator: admin

Post Reply
bedge
Posts: 10
Joined: Sun Aug 24, 2014 2:11 am

Colorization of regexp within log messages

Post by bedge »

While logMx is the most fully featured log viewer I've seen, I want all the things I've come to like in every method of viewing logs. One of those is the ability to pass the logs through a colorizer of sorts to highlight specific text.

Note: the preview of this message showed the attachents incorrectly, the first was repeated twice so the text may not match up exactly.

Here's a sample of tail -f | ccze:
Screen Shot 2014-08-29 at 10.27.52 AM.png
Screen Shot 2014-08-29 at 10.27.52 AM.png (119.1 KiB) Viewed 6495 times
Here are the same logs seen with logmx.
Screen Shot 2014-08-29 at 10.27.52 AM.png
Screen Shot 2014-08-29 at 10.27.52 AM.png (119.1 KiB) Viewed 6495 times
So while logmx is superior to tail -f | ccze in every other way, the former is easier to glance at and see data without resorting to filtering the data.

Is there any way to add regexp colorization to the message text?
Attachments
Screen Shot 2014-08-29 at 10.32.37 AM.png
Screen Shot 2014-08-29 at 10.32.37 AM.png (59.14 KiB) Viewed 6495 times
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Colorization of regexp within log messages

Post by admin »

Hello,

It would be a good idea. Could you please explain a bit further what you would like to have?:
  1. how would the user specify the text to highlight? using something like the current filter? like a search?
  2. what if the matching (highlighted) text would be hidden in its main table cell (because of too long text before this match)
Thank you for your ideas!
bedge
Posts: 10
Joined: Sun Aug 24, 2014 2:11 am

Re: Colorization of regexp within log messages

Post by bedge »

Here's an attempt at aswering the previous questions
1)
As I'm basically lazy, I'd like to have some default log type profiles, similar to what both ccze and multitail have where you specify log4j, apache, syslog, kern.log etc and they basically "do the right thing" to make the log text as readable as possible by highlighting key phrases, quoted strings, java class names, IP/MAC addrs etc.

Would it be possible to specify an external log colorizer so that logmx filtered the raw log data through an external utility, like ccze and used the ansi color codes as-is? This would immediately provide all of the colorization options provided by the existing utility and not require an extensive regexp matching editor where users would specify the color conversions.
Adding the option of an external filter may be useful for a variety of other features too. It could be used to leverage the logmx remote log aggregation capacity to pipe data from many sources into user defined scripts.

In lieu of an external colorizer, I suppose one would need a regexp pattern matcher for colorization. The issue there is that if you're adding a bunch or regexp pattern matching at runtime in java, does it adversely affect the app performance? That may be another point in favor of a native app doing the conversion.

2)
Not sure what the problem is here. Is it that only the displayed length would be passed to the matcher? So a wider window may expose more of a string to the colorizer?
I would prefer that the colorization take place before the window size is evaluated so the entire string for each message is processed.
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Colorization of regexp within log messages

Post by admin »

  1. I think I understand what you mean. The main difference between ccze/multitail and LogMX is that LogMX parses the logs to organize log entries fields in a structured table. At this point, no need to display each different field with a different color in LogMX (date, log level, emitter, ...) since each field is displayed in a separate table cell. Then I think the only field that would need to contain highlighted text would be the "Message" field (and some user-defined fields). A customizable colorizer is a good idea for these fields. Yet, I don't think that using a native application to colorize it would be better for performances, think LogMX would first have to read logs, then send it to this native code, then retrieve the colorized text to display it. In Autorefresh mode it would mean a lot of small exchanges between these two parts. Java is slow at loading the required classes the first time, but once loaded (i.e. repetitive tasks), performances are almost the same as native apps. In Java or in native code, what is sure is that it would slow down a bit the whole process of reading and displaying logs...
    To implement it, we could maybe use something like a customizable Java Colorizer Class to write and to use in LogMX, just like current LogMX Parsers & Managers can be written in Java. Indeed, I guess that a set of regexes could do the job for this Colorizer, or let the Java Colorizer Class do the job, why not.
  2. The limitation to the first part above is this second part: what I was trying to say is that even if your text is highlighted to see it more easily, you wouldn't be able to, if it's outside its table cell. It's not that "only the displayed length would be passed to the matcher", but it's that a lot of work would be done to colorize text that would not be displayed. Contrary to console apps like tail/ccze/multitail, LogMX will never display the whole stream of logs, entries after entries. If an entry field is to wide to fit in its container (main table cell or entry details below the table), a scrollable view is used (in entry details view) to scroll this entry field, but it will not be followed by next entry field... So I think it would be less powerful than an highlighted unique stream of text displayed in a console (I'm not sure I'm very clear to explain this, it's maybe purely subjective to what I personally prefer)
As a conclusion, I personally think that such a customizable colorizer is a good idea, even if I think that it would be less powerful than console mode highlighting. So we will add this to our "feature requests" list, I will let you know when it will be included in a release.

Thank you for this idea!
Xavier
bedge
Posts: 10
Joined: Sun Aug 24, 2014 2:11 am

Re: Colorization of regexp within log messages

Post by bedge »

Your points make perfect sense. Thank you for considering this.
Post Reply