Custom Parser

Here developers can talk about how to write a Parser for LogMX

Moderator: admin

Post Reply
megapixel
Posts: 4
Joined: Fri Oct 30, 2009 4:42 pm

Custom Parser

Post by megapixel »

Hi,

I recently purchased LogMX and I was wondering if someone would be kind enough to assist me in writing my first parser. I have been given the task to use LogMX to manage about 15 different logs, and will most likely have to create a custom one for each.

Here are 3 sample lines of a log file that I am working with right now:

Code: Select all

[Tue Oct 27 11:49:39 2009] [notice] Digest: done

[Wed Oct 28 16:31:57 2009] [notice] Apache/2.2.11 (Unix) DAV/2 configured -- resuming normal operations

[Wed Oct 28 17:42:39 2009] [error] [client 192.168.10.100] File does not exist: /low/low
megapixel
Posts: 4
Joined: Fri Oct 30, 2009 4:42 pm

Post by megapixel »

Maybe I could even get away with doing a Log4jPattern Parser with this.

I have tried the following with no luck:

Code: Select all

 %d{EEE MMM dd HH:mm:ss yyyy} %p %m

megapixel
Posts: 4
Joined: Fri Oct 30, 2009 4:42 pm

Post by megapixel »

forgot my brackets, got it working with log4jpattern parser

Code: Select all

[%d{EEE MMM dd HH:mm:ss yyyy}] [%p] %m
Looks like the next step is to create a java class parser to fine tune it more
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Post by admin »

Hello,

You're right, your last post shows the good Pattern to use.
I don't think you have to write a Java Class Parser for this simple log format. The only thing is that since this format doesn't include any "Emitter" (%c) information (like class, package, process, ...), all your log entries will be considered as emitted by "<AnonymousClass>".

If you really want to write a Java Parser Class, you can get help here:
https://logmx.com/parser-dev
https://logmx.com/dev/api

Note: In your first post, I saw you have to write about 15 parsers: be careful on your parsers ordering: LogMX tries to parse a file with each Parser, in the order given in Tools>Options>Parsers, so if you have a log format which is not strict at all like
INFO Hello world...
WARN Hello world...
you may want to move this parser at the end of the list; or else, LogMX will first try to recognize this format for each log file to load, which may be valid but not what you expected for other log formats.

Please let me know if you have any other problem/question
Xavier.
megapixel
Posts: 4
Joined: Fri Oct 30, 2009 4:42 pm

Post by megapixel »

Thanks for the information Xavier. Could you help me with the Java Parser Class below? I copied the structure from an earlier post and tried to modify it to fit my needs as best as possible, but don't know where I went wrong.

Code: Select all

package high_audit.apache;

import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.regex.Matcher;
import java.util.regex.Pattern; 

import com.lightysoft.logmx.business.ParsedEntry;
import com.lightysoft.logmx.mgr.LogFileParser;

public class ApacheErrorLogParser extends LogFileParser {

	//e.g. [Thu Oct 22 19:00:42 2009] 
	   private static final DateFormat APACHE_TIMESTAMP = new SimpleDateFormat("EEE MMM dd HH:mm:ss yyyy");
	   
	   //[Sat Oct 31 00:33:07 2009] [error] [client 192.168.20.100] client denied by server
	   private static final Pattern ERROR = Pattern.compile("^[(.*)] [error] [(.*)] (.*)$");
	   
	   //[Thu Oct 22 19:00:42 2009] [notice] Digest: done
	   private static final Pattern NOTICE = Pattern.compile("^[(.*)] [notice] (.*)");
	   
	   private ParsedEntry entry;
	   private StringBuilder message = new StringBuilder();
	   
	   @Override
	   public Date getAbsoluteEntryDate(ParsedEntry entry) throws Exception {
	      return (Date)entry.getExtraInfo();
	   }

	   @Override
	   public String getParserName() {
	      return "Apache Error Log Parser";
	   }

	   @Override
	   public Date getRelativeEntryDate(ParsedEntry entry) throws Exception {
	      return getAbsoluteEntryDate(entry);
	   }

	   @Override
	   public String getSupportedFileType() {
	      return "Apache Error logs";
	   }

	   @Override
	   protected void parseLine(String line) throws Exception {
	   
	      //end of file
	      if (line == null) {
	         recordPreviousEntryIfExists();
	         return;
	      }
	   
	      Matcher matcher;
	      
	      //check for ERROR
	      matcher = ERROR.matcher(line);
	      if (matcher.matches()) {
	         recordPreviousEntryIfExists();
	         entry = createNewEntry();
	         entry.setEmitter("errors");
	         entry.setDate(matcher.group(1));
	         entry.setExtraInfo(APACHE_TIMESTAMP.parse(matcher.group(1)));
	         entry.setLevel("ERROR");
	         message.setLength(0); //reset message buffer
	         return;
	      }
	      
	      //check for NOTICE
	      matcher = NOTICE.matcher(line);
	      if (matcher.matches()) {
	         recordPreviousEntryIfExists();
	         entry = createNewEntry();
	         entry.setEmitter("notices");
	         entry.setDate(matcher.group(1));
	         entry.setExtraInfo(APACHE_TIMESTAMP.parse(matcher.group(1)));
	         entry.setLevel("INFO");
	         message.setLength(0); //reset message buffer
	         return;
	      }
	      
	      //accumulate message
	      message.append(line).append("\n");
	      
	   }

	    /**
	     * Send to LogMX the current parsed log entry
	     * @throws Exception
	     */
	    private void recordPreviousEntryIfExists() throws Exception {
	        if (entry != null) {
	             entry.setMessage(message.toString());
	            addEntry(entry);
	        }
	    }


	} 
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Post by admin »

Hello,

Here is a fixed version:

Code: Select all

  //e.g. [Thu Oct 22 19:00:42 2009]
    private static final DateFormat APACHE_TIMESTAMP = new SimpleDateFormat(
        "EEE MMM dd HH:mm:ss yyyy", Locale.US);

    //[Sat Oct 31 00:33:07 2009] [error] [client 192.168.20.100] client denied by server
    private static final Pattern ERROR = Pattern
        .compile("^\\[(.*?)\\] \\[error\\] \\[(.*?)\\] (.*)$");

    //[Thu Oct 22 19:00:42 2009] [notice] Digest: done
    private static final Pattern NOTICE = Pattern.compile("^\\[(.*?)\\] \\[notice\\] (.*?)");

    private ParsedEntry entry;

    private StringBuilder message = new StringBuilder(80);


    @Override
    public Date getAbsoluteEntryDate(ParsedEntry entry) throws Exception {
        return (Date) entry.getExtraInfo();
    }

    @Override
    public String getParserName() {
        return "Apache Error Log Parser";
    }

    @Override
    public Date getRelativeEntryDate(ParsedEntry entry) throws Exception {
        return getAbsoluteEntryDate(entry);
    }

    @Override
    public String getSupportedFileType() {
        return "Apache Error logs";
    }

    @Override
    protected void parseLine(String line) throws Exception {

        //end of file
        if (line == null) {
            recordPreviousEntryIfExists();
            return;
        }

        Matcher matcher;

        //check for ERROR
        matcher = ERROR.matcher(line);
        if (matcher.matches()) {
            prepareNewEntry();
            entry.setEmitter("errors");
            entry.setDate(matcher.group(1));
            entry.setExtraInfo(APACHE_TIMESTAMP.parse(matcher.group(1)));
            entry.setLevel("ERROR");
            message.append(matcher.group(3));
            return;
        } else {
            //check for NOTICE
            matcher = NOTICE.matcher(line);
            if (matcher.matches()) {
                prepareNewEntry();
                entry.setEmitter("notices");
                entry.setDate(matcher.group(1));
                entry.setExtraInfo(APACHE_TIMESTAMP.parse(matcher.group(1)));
                entry.setLevel("INFO");
                message.append(matcher.group(2));
                return;
            } else {
                message.append('\n').append(line);
            }
        }
    }

    /**
     * Send to LogMX the current parsed log entry
     * @throws Exception
     */
    private void recordPreviousEntryIfExists() throws Exception {
        if (entry != null) {
            entry.setMessage(message.toString());
            addEntry(entry);
        }
    }

    /**
     * Send to LogMX the current parsed log entry, then create a new one
     * @throws Exception
     */
    private void prepareNewEntry() throws Exception {
        recordPreviousEntryIfExists();
        entry = createNewEntry();
        message = new StringBuilder(80);
    }
  • The main problem was the non-escaped "[" characters in regular expressions.
  • You also truncated entry message, and didn't handle multiple-lines entries (occurs when a log message contains "\n")
  • You don't seem to be interested in displaying the text between [] just after [error]... As I said, for this simple log format, I don't think you need to use a Java class parser. You only have to use the Log4J pattern explained above, and add "notice" and "error" levels (Tools>Options>Levels). In this case, using a L4j pattern will produce the same effect.
Post Reply