Problems with selfmade Manager

Here developers can talk about how to write a Manager for LogMX

Moderator: admin

Post Reply
KeyKon
Posts: 2
Joined: Thu Jan 19, 2012 10:39 am

Problems with selfmade Manager

Post by KeyKon »

I wrote a manager for receiving data from the serial port, that saves them in a file and displays them live in LogMX.
But i figuread out a Problem, if there is heavy input (about 500KB in less than a minute) the heap usage of LogMX hits the maximum and it gets uncontrollable.

Do you have any ideas where the problem might be? Is there anything I might have done wrong in the Manager or might it be a problem in the LogMX-Garbagecollector?

Regards,
KeyKon
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Problems with selfmade Manager

Post by admin »

Hello,

I personally don't have any idea where the problem is, you should not experience memory issues writing your own Log Manager, since it is not supposed to keep in memory a lot of information. It should only read one line of text from the underlying resource only when it is called (readLine()), plus in your case detecting any file change through file size/date change (since i understood it handles AutoRefresh).
For your information, LogMX doesn't tune its Java GarbageCollector, the default GC settings are used.

Just a few ideas:
  1. Do you use a BufferedReader? if yes, check its buffer size.
  2. 500Kb / minute doesn't seem "heavy" to me for LogMX, it has been tested with much more. Did you try increasing the amount of memory LogMX can use? (-Xmx option in your launch script in LogMX directory: "launcher.cfg" when using LogMX.exe, or "LogMX.bat", or "logmx.sh" for Linux/Mac). Just to know how much memory it tries to allocate.
  3. Could you try to put the content received from your serial port to a local file, then parse this local file, to check if the same problem occurs? (if yes, maybe a Parser issue, or maybe LogMX itself)
Just in case you didn't already found them, here are a few links about Manager implementation: If you didn't resolve your problem, feel free to send me your Manager (by Private Message on this forum or at support@logmx.com).

Xavier.
KeyKon
Posts: 2
Joined: Thu Jan 19, 2012 10:39 am

Re: Problems with selfmade Manager

Post by KeyKon »

You supposed right, it handles AutoRefresh, and I think thats where I generate the problem.
I tested reading the logfile with the builtin localfilemanager, this worked without problems, it also was much faster than my filereadmanager, so I did some testing how my manager interacts with LogMX.
I'm sure its not normal, that LogMX reads the whole file each AutoRefreshCycle, but I don't know what method I implemented wrong...
Maybe you got any idea, here a little log, how LogMX calls the methods each cycle:

Code: Select all

readFileHeader arg0: 4096
readLineAtOffset arg0: 0
getCurrentOffset -> 104
readLineAtOffset arg0: 104
getCurrentOffset -> 219
readLineAtOffset arg0: 219
getCurrentOffset -> 334
readLineAtOffset arg0: 334
getCurrentOffset -> 385
[...]
readLineAtOffset arg0: 1575
getCurrentOffset -> 1603
readLineAtOffset arg0: 1603
getCurrentOffset -> 1605
It really seems to read the whole file each time! Which of my methods might work wrong?
For your information:
The manager has a part that waits for data from the serialport and writes them to a local file, and an other part that is reading from this file (With RandomAccessFile())

KeyKon
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Problems with selfmade Manager

Post by admin »

Hello,

The sequence of calls between LogMX and your Manager seems correct to me, from what I see: each cycle, LogMX asks the Manager its current position in file (getCurrentOffset()), then it asks the Manager to read the next line of text starting at this position (readLineAtOffset()). See http://www.logmx.com/dev/api/com/lighty ... nager.html for more details about these methods.

But could you please let us know how you implement getCurrentFileLastModifDate(), getCurrentFileSize(), and/or getCurrentFileInfo()? These methods are used by LogMX to detect changes needing to be read by the Manager.

I don't know if this is possible, but I think it would be more reliable to avoid using a file to exchange data between the part of your Manager that reads the serial port, and the part that reads this file to send it back to LogMX. Is it possible in your case to use a buffer in memory instead of a file, via a producer/consumer strategy? (http://en.wikipedia.org/wiki/Producer-consumer_problem). Anyway, maybe the part of your Manager that reads this file buffer never releases the bytes it has read? (which would explain why your Manager uses so much memory).

Another idea: you may want to look at "Memory Analyzer Tool" (http://www.eclipse.org/mat), an open-source tool that will show you the class that holds too much memory in a Java program: easy to use, no instrumentation or other tool needed, just run LogMX normally with your Manager, and let the tool analyze LogMX process to scrutinize your Manager memory allocation (a JDK is required, a JRE is not enough).

Please let me know when you have some news.
Xavier
akk
Posts: 2
Joined: Wed Dec 12, 2012 10:50 am

Re: Problems with selfmade Manager

Post by akk »

KeyKon wrote:I wrote a manager for receiving data from the serial port, that saves them in a file and displays them live in LogMX.
But i figuread out a Problem, if there is heavy input (about 500KB in less than a minute) the heap usage of LogMX hits the maximum and it gets uncontrollable.

Do you have any ideas where the problem might be? Is there anything I might have done wrong in the Manager or might it be a problem in the LogMX-Garbagecollector?

Regards,
KeyKon


Hi Keykon,

I'm new to writing logmx manager. Need your help in writing a manager for "receiving data from the serial port, that saves them in a file and displays them live in LogMX" :)
i would appreciate if you could share the manager you have made, so that i could make the necessary changes depending on my requirement and use it.

Thanks,
akk
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Problems with selfmade Manager

Post by admin »

Hello akk,

I don't think that LogMX should both write an output file, and display its logs. LogMX is just a viewer, and making it responsible for writing a file from a serial port seems a bit risky to me: what will happen if LogMX is closed accidentally? (or if it crashes, what I don't expect though :wink:). Plus, you don't really control when your Manager is created, started, stopped... If you can make another small application write serial port data to a file, you can use LogMX to monitor this file content in real-time.

If you have a problem to implement your Manager, feel free to post on this forum or mail us (support@logmx.com)

Xavier
akk
Posts: 2
Joined: Wed Dec 12, 2012 10:50 am

Re: Problems with selfmade Manager

Post by akk »

Thanks Xavier for immediate reply.
We are actually looking at using a single tool either LogMx or PCTool(To create logfile and parse).

Can you guide me how to make manager for "receiving data from the serial port, that saves them in a file and displays them live in LogMX".

For serial read and logging to a file, i have attached the code below.

Code: Select all

import java.io.*;
import java.util.*;
import javax.comm.*;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

public class SerialRead implements Runnable, SerialPortEventListener   {
    static CommPortIdentifier portId;
    static Enumeration portList;
 
    InputStream inputStream;
    SerialPort serialPort;
    Thread readThread;
 
    public static void main(String[] args) {
        portList = CommPortIdentifier.getPortIdentifiers();
 
        while (portList.hasMoreElements()) {
            portId = (CommPortIdentifier) portList.nextElement();
            if (portId.getPortType() == CommPortIdentifier.PORT_SERIAL) {
                if (portId.getName().equals("COM1")) {
                	SerialRead reader = new SerialRead();
                }
            }
        }
    }
 
    public void SerialRead() {
        try {
            serialPort = (SerialPort) portId.open("SerialReadApp", 2000);
        } catch (PortInUseException e) {System.out.println(e);}
        try {
            inputStream = serialPort.getInputStream();
        } catch (IOException e) {System.out.println(e);}
        try {
            serialPort.addEventListener(this);
        } catch (TooManyListenersException e) {System.out.println(e);}
        serialPort.notifyOnDataAvailable(true);
        try {
            serialPort.setSerialPortParams(9600,
                SerialPort.DATABITS_8,
                SerialPort.STOPBITS_1,
                SerialPort.PARITY_NONE);
        } catch (UnsupportedCommOperationException e) {
            System.out.println(e);
        }
        readThread = new Thread(this);
        readThread.start();
    }
 
    public void run() {
        try {
            Thread.sleep(20000);
        } catch (InterruptedException e) {
            System.out.println(e);
        }
    }
 
    public void serialEvent(SerialPortEvent event) {
        switch(event.getEventType()) {
            case SerialPortEvent.BI:
            case SerialPortEvent.OE:
            case SerialPortEvent.FE:
            case SerialPortEvent.PE:
            case SerialPortEvent.CD:
            case SerialPortEvent.CTS:
            case SerialPortEvent.DSR:
            case SerialPortEvent.RI:
            case SerialPortEvent.OUTPUT_BUFFER_EMPTY:
                break;
            case SerialPortEvent.DATA_AVAILABLE:
                byte[] readBuffer = new byte[255];
                try {
                    while (inputStream.available() > 0) {
                        int numBytes = inputStream.read(readBuffer);
                    	
                        String str = new String(readBuffer);
                        
                    	 FileOutputStream fos = new FileOutputStream("c:\\akk\\logfile.txt", true);
                    	 OutputStreamWriter out = new OutputStreamWriter(fos, "UTF-8"); 
                    	 
                    	 out.write(str);
                    	 
                    	 fos.close();
                    	
                		       }
                    //System.out.print(new String(readBuffer));
                } catch (IOException e) {
                    System.out.println(e);
                }
                break;
        }
    }
}


If possible, kindly provide me the email id of KeyKon, so that i can get in touch with him.

Regards,
akk
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Problems with selfmade Manager

Post by admin »

Hello akk,

Well, as I explained in my last post i REALLY don't think you should let LogMX write a file with data received from a serial port :evil:. LogMX was not designed to do this, nor its LogManagers. It doesn't matter if it's your client or even your boss who asked you to do this, implementing this may cause you great pain later in debugging... and you should explain this to the one who ask you this, or show him this post.

If you still want to play with the devil, here are a few hints:
  1. Carefully read http://www.logmx.com/dev/api/com/lighty ... nager.html and http://www.logmx.com/p_manager_dev.php
  2. Look at the HTTP sample Manager from your LogMX "managers" directory
  3. Do not put any "main()" method in your Manager (like in your code sample). If you use it for your tests, do not expect LogMX to call it, because it won't
  4. Keep in mind that LogMX can instantiate your Manager any time using the default constructor (even if no file is to be read) and without calling "releaseResources()" after that. Wait for a call to "prepareForReading()" to prepare yourself for reading and allocating some stuff
  5. You may want to start a Thread that reads from your serial port in method "prepareForReading()" and stop it in "releaseResources()"
  6. You will have to choose if you want to keep the read-data in cache to send it back to LogMX, or to only write this read-data to the file, to read this file later when LogMX will ask you to. In my own opinion, using a cache will be a bit more complicated and may lead to large memory allocation if something goes wrong: what if LogMX user switches off "AutoRefresh" feature while your thread reads from serial port to put data to a file PLUS a cache? if LogMX doesn't call "readLine()" anymore, your cache may never be cleared and continue to grow as new data arrives on serial port... If you don't use a cache and write directly to the file, you can keep in cache the current date&time, and current file size, to give it back to LogMX when implementing "getCurrentFileInfo()", "getCurrentFileLastModifDate()", or "getCurrentFileSize()".
We can't help you more since your code has some unresolved dependencies and we can't test serial port reading on our side.

As for KeyKon email, sorry but for privacy issues, I can't give you more information that the one he left on his profile. Have you tried to look at his profile on this forum or simply write him a private message using this forum? Maybe he didn't enable the email notification to post here, so he still don't know we are having this discussion here...

Xavier
Post Reply