Public parser repository

Here developers can talk about how to write a Parser for LogMX

Moderator: admin

Post Reply
roy@smartbear.com
Posts: 2
Joined: Wed Oct 07, 2009 4:51 pm
Location: Austin, TX
Contact:

Public parser repository

Post by roy@smartbear.com »

I need a parser for the Eclipse log file format. Is there a public parser repository where I can find a parser that supports this format?

If there is no such parser available, I'll write my own. I would be more than happy to donate my parser to a public repository so other people can use it.

Thanks,
Roy
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Post by admin »

Hello,

There's no Eclipse log file parser at this time, but you can try this Log4jPattern Parser:

Code: Select all

!ENTRY %c %x %X %d{yyyy-MM-dd HH:mm:ss.SSS}%n!MESSAGE %m
(tested with Eclipse 3.4.1, workspace log file ".metadata/.log")

It handles emitter (i.e. plug-in), message, date, but not the log level, which seems to be an integer just after the emitter:
1 => info
2 => warning
4 => error

To handle these log levels, you can use this Log4jPattern Parser:

Code: Select all

!ENTRY %c %p %X %d{yyyy-MM-dd HH:mm:ss.SSS}%n!MESSAGE %m
But you will have to add "1", "2", and "4" levels mapped to Info, Warning, Error to your LogMX configuration (see Tools > Options > Level)

But the best solution would be to write a Java Class Parser (http://www.logmx.com/p_parser_dev.php) to convert 1, 2, 4 to info, warning, error, and to handle "!SESSION" and "!SUBENTRY" blocks.

Let me know if you need more help to handle this format.

Xavier
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Post by admin »

For the public parser repository, we are currently collecting some common log parsers, and we will open such a repository on our website soon :wink:
roy@smartbear.com
Posts: 2
Joined: Wed Oct 07, 2009 4:51 pm
Location: Austin, TX
Contact:

Post by roy@smartbear.com »

Yeah, I already went and wrote my own parser. Feel free to add it to your public repo:

Code: Select all


import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import com.lightysoft.logmx.business.ParsedEntry;
import com.lightysoft.logmx.mgr.LogFileParser;

/**
 * @author Roy Paterson
 */
public class EclipseLogParser extends LogFileParser {

	//e.g. 2009-09-15 09:34:16.000
	private static final DateFormat ECLIPSE_TIMESTAMP = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
	
	//!SESSION 2009-09-16 09:07:32.751 -----------------------------------------------
	private static final Pattern SESSION = Pattern.compile("^!SESSION (.*) -----------------------------------------------$");
	
	//!ENTRY com.smartbear.collaborator.ui.standalone 4 0 2009-09-16 09:07:41.440
	private static final Pattern ENTRY = Pattern.compile("^!ENTRY (.*) (\\d) \\d (.*)$");
	
	private ParsedEntry entry;
	private StringBuilder message = new StringBuilder();
	
	@Override
	public Date getAbsoluteEntryDate(ParsedEntry entry) throws Exception {
		return (Date)entry.getExtraInfo();
	}

	@Override
	public String getParserName() {
		return "Eclipse log parser";
	}

	@Override
	public Date getRelativeEntryDate(ParsedEntry entry) throws Exception {
		return getAbsoluteEntryDate(entry);
	}

	@Override
	public String getSupportedFileType() {
		return "Eclipse logs";
	}

	@Override
	protected void parseLine(String line) throws Exception {
	
		//end of file
		if (line == null) {
			recordPreviousEntryIfExists();
			return;
		}
	
		Matcher matcher;
		
		//check for SESSION
		matcher = SESSION.matcher(line);
		if (matcher.matches()) {
			recordPreviousEntryIfExists();
			entry = createNewEntry();
			entry.setEmitter("SESSION");
			entry.setDate(matcher.group(1));
			entry.setExtraInfo(ECLIPSE_TIMESTAMP.parse(matcher.group(1)));
			entry.setLevel("INFO");
			message.setLength(0); //reset message buffer
			return;
		}
		
		//check for ENTRY
		matcher = ENTRY.matcher(line);
		if (matcher.matches()) {
			recordPreviousEntryIfExists();
			entry = createNewEntry();
			entry.setEmitter(matcher.group(1));
			entry.setLevel(toLogMxLevel(Integer.parseInt(matcher.group(2))));
			entry.setDate(matcher.group(3));
			entry.setExtraInfo(ECLIPSE_TIMESTAMP.parse(matcher.group(3)));
			message.setLength(0); //reset message buffer
			return;
		}
		
		//accumulate message
		message.append(line).append("\n");
		
	}

    /**
     * Send to LogMX the current parsed log entry
     * @throws Exception
     */
    private void recordPreviousEntryIfExists() throws Exception {
        if (entry != null) {
       		entry.setMessage(message.toString());
            addEntry(entry);
        }
    }

    private static String toLogMxLevel(int statusCode) {
    	switch(statusCode) {
    		case 0: //IStatus.OK
    			return "INFO";
    		case 1: //IStatus.INFO
    			return "INFO";
    		case 2: //IStatus.WARNING
    			return "WARNING";
    		case 4: //IStatus.ERROR
    			return "ERROR";
    		case 8: //IStatus.CANCEL
    			return "WARNING";
    		default:
    			return "ERROR";
    	}
    }

}
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Post by admin »

Thank you very much, well done :D

May I suggest an optimization: using reluctant quantifier ".*?" instead of greedy quantifier ".*" will greatly improve regular expression processing. I think it can be used here without problem (reluctant performs a "minimal matching" compared to greedy that will try to match all characters in string)
alexk195
Posts: 6
Joined: Tue May 03, 2011 7:25 am

Re: Public parser repository

Post by alexk195 »

Here is my simple parser to parse fixed length log files genereted by CAN Analyzer CanKing:

Code: Select all

package sample.parser;

import java.text.DecimalFormatSymbols;
import java.text.SimpleDateFormat;
import java.util.Arrays;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.text.DecimalFormat;

import com.lightysoft.logmx.business.ParsedEntry;
import com.lightysoft.logmx.mgr.LogFileParser;

/**
 * 
 * @author kramera
 * 
 * Parser for CanKing log files of following format (fixed positions)



 0    0C49020D X       6  30  02  00  00  00  02          0.....      34769.328890 R
 0    0C490D02 X       8  20  63  75  24  38  20  36  35   cu$8 65    34769.329720 R
 0    0C490D02 X       8  21  35  33  35  20  2D  31  04  !535 -1.    34769.330360 R
 0    0C490D02 X       2  22  FA                          ".          34769.330810 R
 0    0C49020D X       6  30  00  00  02  00  02          0.....      34769.333690 R
 0    00000342         8  4A  56  00  00  00  00  00  00  JV......    34769.646650 R
 0    000002C2         8  4A  56  00  00  00  00  00  00  JV......    34769.647550 R
 0    00000342         8  42  47  00  40  00  00  00  00  BG.@....    34769.748600 R
 0    000002C2         8  42  47  00  00  00  00  00  00  BG......    34769.748990 R
 0    0C44020D X       8  10  11  52  65  61  64  41  63  ..ReadAc    34770.010490 R
 0    0C440D02 X       6  30  01  00  00  00  01          0.....      34770.011130 R
 0    0C44020D X       8  20  63  75  42  79  74  65  24   cuByte$    34770.011770 R
 0    0C44020D X       5  21  31  33  05  14              !13..       34770.012600 R
 0    0C45020D X       8  10  0D  52  65  61  64  41  63  ..ReadAc    34770.013880 R
 0    0C46020D X       8  10  0D  49  73  41  63  63  75  ..IsAccu    34770.014590 R
 0    0C440D02 X       6  30  00  00  02  00  01          0.....      34770.017850 R
 
 
 */
public class CanKingParser extends LogFileParser {
    private DecimalFormatSymbols dc;
    private DecimalFormat nf;// = new DecimalFormat("#.#",);
    
    /** Key of user-defined field "timestamp" */
    private static final String EXTRA_FIELD_KEY_CAN_ID = "Object Id";
    private static final String EXTRA_FIELD_KEY_CAN_DATA_HEX = "Data Hex";
    private static final String EXTRA_FIELD_KEY_CAN_DATA_LENGTH = "Data Length";
    private static final String EXTRA_FIELD_KEY_CAN_DESTINATION = "Destination";
    private static final String EXTRA_FIELD_KEY_CAN_EXTENDED = "Extended";
    
    
    
    public CanKingParser()
    {
    	 dc = new DecimalFormatSymbols();
    	 dc.setDecimalSeparator('.');
    	 nf = new DecimalFormat("#.#",dc); 
    }

    /** User-defined fields names (here, only one) */
    private static final List<String> EXTRA_FIELDS_KEYS = Arrays
        .asList(EXTRA_FIELD_KEY_CAN_ID,
        		EXTRA_FIELD_KEY_CAN_DATA_HEX,
        		EXTRA_FIELD_KEY_CAN_DATA_LENGTH,
        		EXTRA_FIELD_KEY_CAN_DESTINATION,
        		EXTRA_FIELD_KEY_CAN_EXTENDED);  
    
    private ParsedEntry entry = null;
    
	@Override
	public Date getAbsoluteEntryDate(ParsedEntry arg0) throws Exception {
	       //return DATE_FORMAT.parse(arg0.getDate()); // (the right-part "T0+..." will be ignored by the formatter)
		   
		   double n = nf.parse(arg0.getDate()).doubleValue();
	       return new Date((long)(n*1000.0));
	}

	@Override
	public String getParserName() {
		 return "CanKing Log Parser V2";
	}

	@Override
	public Date getRelativeEntryDate(ParsedEntry arg0) throws Exception {
	    
		    return getAbsoluteEntryDate(arg0);
	}

	@Override
	public String getSupportedFileType() {
		   return "CanKing log Files";
	}
	
	/** 
     * Returns the ordered list of user-defined fields to display (given by their key), for each entry.
     * @see com.lightysoft.logmx.mgr.LogFileParser#getUserDefinedFields()
     */
    @Override
    public List<String> getUserDefinedFields() {
        return EXTRA_FIELDS_KEYS;
    }

    
	@Override
	protected void parseLine(String arg0) throws Exception {

			if (arg0 == null)
				return;
			
			if (arg0.length() != 84)
				return;
			
            entry = createNewEntry();

            assert(entry!=null);

            if (entry != null)
            {
            	String extended = arg0.substring(15, 16).trim();
            	boolean extSyntax = extended.compareTo("X")==0;
            	
            	String canId = arg0.substring(6, 14);
            	
            	String timeStamp=arg0.substring(70, 82); 
	            entry.setDate(timeStamp);
	            
	            /*
	            double n = nf.parse(timeStamp).doubleValue();
	            Double db = new Double(n);
	            String tmp = db.toString();
	            entry.setThread(tmp);
	            */
	            
	            if (extSyntax)
	            	entry.setLevel("CAN_EXT");
	            else
	            	entry.setLevel("CAN_NORM");
	            
	            if (extSyntax)
	            	entry.setEmitter(canId.substring(6,8));
	            else
	            	entry.setEmitter(canId);
	            
	            entry.setMessage(arg0.substring(58, 66));
	         
	            entry.setUserDefinedFields(new HashMap<String, Object>(3)); 
	            entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_CAN_ID,canId );
	            entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_CAN_DATA_HEX,  arg0.substring(26, 56));
	            entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_CAN_DATA_LENGTH, arg0.substring(23, 24));
	            if (extSyntax)
	            	entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_CAN_DESTINATION, canId.substring(4,6));
	            
	            entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_CAN_EXTENDED, extended);
	            
	            addEntry(entry);
            }
	}

}

admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Public parser repository

Post by admin »

Hello alexk195,

Thank you for your contribution :wink: we have added your parser to our internal repository.
Our public parser repository will be on-line as soon as we have collected a few more parsers.

Xavier.
alexk195
Posts: 6
Joined: Tue May 03, 2011 7:25 am

Re: Public parser repository

Post by alexk195 »

Here my another custom Parser for parsing linux kernel logs, dmesg output etc. It searches for keywords like "error","panic", "warning" for appropriate level. Still needs proper time stamp parsing but works in general.

@admin/developer: Some feature to disable columns in custom parser would be nice! E.g. Don't show Emitter column, if every entry has null Emitter.

Code: Select all

package sample.parser;

import java.text.SimpleDateFormat;
import java.util.Arrays;
import java.util.Date;
import java.util.HashMap;
import java.util.List;

import com.lightysoft.logmx.business.ParsedEntry;
import com.lightysoft.logmx.mgr.LogFileParser;

/**
 * 
 * @author Alexander Kramer
 * 
 * Parser for Kernel logs, detects words "error","panic","warning" in messages to set the level.
 * TODO: Parsing timestamp doesn't work.

Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2346369
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2373697
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2401089
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2428577
 
 
 */
public class KernelLogParser extends LogFileParser {
    
	  
	protected SimpleDateFormat dateFormat = new SimpleDateFormat("MMM dd HH:mm:ss");
	
    public KernelLogParser()
    {
    }
	
    
    /**
     * Custom columns as String constants
     */
    private static final String EXTRA_FIELD_KEY_HOST_NAME = "Host";

    /**
     * Put all custom fields in an const array
     */
     private static final List<String> EXTRA_FIELDS_KEYS = Arrays
        .asList(EXTRA_FIELD_KEY_HOST_NAME
        		);  
    
    private ParsedEntry entry = null;
    
	@Override
	public Date getAbsoluteEntryDate(ParsedEntry arg0) throws Exception {
		   
		Date date = new Date();
		parseDateTime(arg0.getDate(), date);
	    return date;
	}

	@Override
	public String getParserName() {
		 return "Kernel Log Parser V1";
	}

	@Override
	public Date getRelativeEntryDate(ParsedEntry arg0) throws Exception {
	    
		    return getAbsoluteEntryDate(arg0);
	}

	@Override
	public String getSupportedFileType() {
		   return "Kernel logs";
	}
	
	/** 
     * Returns the ordered list of user-defined fields to display (given by their key), for each entry.
     * @see com.lightysoft.logmx.mgr.LogFileParser#getUserDefinedFields()
     */
    @Override
    public List<String> getUserDefinedFields() {
        return EXTRA_FIELDS_KEYS;
    }

    /// returns true if timestamp is in proper format
    protected boolean parseDateTime(String dateTime, Date date)
    {
    	String[] fields = dateTime.split(" ");
    	if (fields.length != 3)
    		return false;
    	String[] timeFields = fields[2].split(":");
    	if (timeFields.length != 3)
    		return false;
    	try
    	{
    		date = dateFormat.parse(dateTime); 
    	}catch(Exception e)
    	{
    		// SimpleDateFormat cannot parse it, TODO: define format or parse by itself
    		// return false;
    	}
    	
    	return true;
    }
    
	@Override
	protected void parseLine(String arg0) throws Exception {

			if (arg0 == null)
				return;
			
		    entry = createNewEntry();

            if (entry != null)
            {
            	String dateTimeString = arg0.substring(0, 16).trim();
            	
            	Date date = new Date();
            	
            	// quick check of format
            	if (!parseDateTime(dateTimeString, date))
            		return;
            	
            	entry.setDate(dateTimeString);
            	
            	String exclTimeStr = arg0.substring(16).trim();
	            
            	int spaceIndex = exclTimeStr.indexOf(' ');

            	//additional check for format 
            	if (spaceIndex<0) return;
            	
            	String hostName = exclTimeStr.substring(0,spaceIndex).trim();
            	
            	entry.setUserDefinedFields(new HashMap<String, Object>()); 
            	 
            	entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_HOST_NAME, hostName);
            	  
            	String message = exclTimeStr.substring(spaceIndex).trim();
            	entry.setMessage(message);
            	
            	// defining levels, search for keywords 
            	
            	String messageLowerCase = message.toLowerCase();
            	
	            if ((messageLowerCase.indexOf("error")>=0)|| messageLowerCase.indexOf("panic")>=0)
	            	entry.setLevel("ERROR");
	            else
	            if (messageLowerCase.indexOf("warning")>=0)
	            	entry.setLevel("WARNING");
	            else
	            	entry.setLevel("TRACE");
	            
	            // We don't use them:
	            entry.setEmitter(null);
	            entry.setThread(null);
	            
	            addEntry(entry);
	           
            }
	}

}

admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Public parser repository

Post by admin »

Hello,

Thank you!
Here is a version with Timestamp parsing.
In your version, Timestamp parsing didn't work because in the "parseDateTime()" method, the new value for the "date" parameter was lost once this method exited (in Java, all parameters are passed by value and not by reference).

Note that the timestamp year is missing in this log file format, so LogMX considers it to be 1970.
Also note that the timestamp parsing depends on the Locale specified for this Parser (set in "Tools" > "Options" > "Parsers" > "Kernel logs"). For example, to parse "Aug" as the 8th month of the year, your must specify an English Locale for this Parser.

Code: Select all

package sample.parser;

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Arrays;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import com.lightysoft.logmx.business.ParsedEntry;
import com.lightysoft.logmx.mgr.LogFileParser;

/**
*
* @author Alexander Kramer
*
* Parser for Kernel logs, detects words "error","panic","warning" in messages to set the level.

Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2346369
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2373697
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2401089
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2428577


*/
public class KernelLogParser extends LogFileParser {

    protected SimpleDateFormat dateFormat = null;

    /** Pattern for entry begin */
    private final static Pattern ENTRY_BEGIN_PATTERN = Pattern
        .compile("^(\\S+?\\s+?\\d{1,2}\\s+?\\d{2}:\\d{2}:\\d{2})\\s+?(\\S+?)\\s+(.*)$");


    public KernelLogParser() {
    }


    /**
     * Custom columns as String constants
     */
    private static final String EXTRA_FIELD_KEY_HOST_NAME = "Host";

    /**
     * Put all custom fields in an const array
     */
    private static final List<String> EXTRA_FIELDS_KEYS = Arrays
        .asList(EXTRA_FIELD_KEY_HOST_NAME);

    private ParsedEntry entry = null;


    @Override
    public Date getAbsoluteEntryDate(ParsedEntry arg0) throws Exception {
        return parseDateTime(arg0.getDate());
    }

    @Override
    public Date getRelativeEntryDate(ParsedEntry arg0) throws Exception {
        return getAbsoluteEntryDate(arg0);
    }

    @Override
    public String getParserName() {
        return "Kernel Log Parser V2";
    }

    @Override
    public String getSupportedFileType() {
        return "Kernel logs";
    }

    /**
      * Returns the ordered list of user-defined fields to display (given by their key), for each entry.
      * @see com.lightysoft.logmx.mgr.LogFileParser#getUserDefinedFields()
      */
    @Override
    public List<String> getUserDefinedFields() {
        return EXTRA_FIELDS_KEYS;
    }

    /// returns a non-null Date if timestamp is in proper format
    protected Date parseDateTime(String dateTime) {
        if (dateFormat == null) {
            // Now create the date formatter using the right Locale
            // (method "getLocale()" can't be called from the constructor) 
            dateFormat = new SimpleDateFormat("MMM d HH:mm:ss", getLocale());
        }
        try {
            return dateFormat.parse(dateTime);
        } catch (ParseException e) {
            // Invalid format 
            return null;
        }
    }

    @Override
    protected void parseLine(String line) throws Exception {
        if (line == null)
            return;

        // Check line format
        Matcher matcher = ENTRY_BEGIN_PATTERN.matcher(line);
        if (!matcher.matches()) {
            return;
        }

        entry = createNewEntry();

        entry.setDate(matcher.group(1));

        String hostName = matcher.group(2);
        entry.setUserDefinedFields(new HashMap<String, Object>());
        entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_HOST_NAME, hostName);

        String message = matcher.group(3).trim();
        entry.setMessage(message);

        // defining levels, search for keywords
        String messageLowerCase = message.toLowerCase();

        if ((messageLowerCase.indexOf("error") >= 0) || messageLowerCase.indexOf("panic") >= 0)
            entry.setLevel("ERROR");
        else if (messageLowerCase.indexOf("warning") >= 0)
            entry.setLevel("WARNING");
        else
            entry.setLevel("TRACE");

        // We don't use them:
        entry.setEmitter(null);
        entry.setThread(null);

        addEntry(entry);
    }

}
mdi
Posts: 4
Joined: Thu Dec 06, 2012 1:23 pm

Re: Public parser repository

Post by mdi »

Hello

Is there a public parser repository now?

Is there an example somewhere, how to read Websphere SystemOut.log and SystemErr.log, the websphere appserver logfiles?

Regards
Matthias
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Public parser repository

Post by admin »

Hello,

The public parser repository is not available yet, but here is a parser for Websphere logs:
http://www.logmx.com/downloads/forum/We ... Parser.zip

This ZIP file contains the LogMX parser, its source file, a log file example, and an info file about this log format (this package comes from our "Private parser repository", that should become "Public" in the future :) ).

To test it, simply copy the class file in LogMX directory "parsers\classes\sample\parser" (see http://www.logmx.com/p_parser_dev.php for more details on Parsers development)

Xavier.
mdi
Posts: 4
Joined: Thu Dec 06, 2012 1:23 pm

Re: Public parser repository

Post by mdi »

Hi Xavier

Thank you very much for the fast answer. I will try your parser.

Regards
Matthias
Post Reply