Improving Speed?

Here developers can talk about how to write a Manager for LogMX

Moderator: admin

Post Reply
rmitola
Posts: 17
Joined: Mon Jun 08, 2015 7:08 pm

Improving Speed?

Post by rmitola »

Hello. I have written a custom manager and parser and am attempting to log a large amount (around 4000 logs). This is taking more than a minute, and I would like it to be faster. While trying to pinpoint any bottlenecks in my code, I noticed from console logging certain key points in my manager (e.g. every time readLine() is called) and parser (e.g. parseLine()) as well as looking at the GUI's status bar (which will read +101 entries added) that LogMX will process 101 of my logs in the blink of an eye, wait a moment, process 101 more, wait again, ect. This led me to wonder if perhaps the speed issue is out of my control, and has to do with how LogMX handles logs internally? Do you have any insights on this?

Perhaps important as well: I am receiving logs over a socket, and all 4000 are received very quickly. It is the processing of the logs that is taking a while.

Thanks!
jobemakar
Posts: 27
Joined: Thu Jun 11, 2015 9:49 pm

Re: Improving Speed?

Post by jobemakar »

For a little bit of context - we are using LogMX to view logs in realtime from our Facebook app. So timing is important. Our current logging solution (that we're trying to move away from) allows us to view the logs in realtime with no noticeable delay.

That solution is SOSMax and uses TCP:
http://www.sos.powerflasher.com/develop ... /download/
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Improving Speed?

Post by admin »

Hello,

You are right, in LogMX v5.3.4, all Managers except LocalFileManager use a 2000 ms AutoRefresh delay. So it means that the embedded SocketManager or your user-defined Manager will be triggered every 2000 ms, not faster. Unfortunately, this delay cannot be configured in v5.3.4. The good news is that we have already fixed this in LogMX v5.4.0 while rewriting our SocketManager: There is now 3 configurable settings for LocalManager, SocketManager, and other Managers:

Image

We are also thinking of allowing each user-defined Manager to force a specific delay for itself using a new method on class "LogFileManager". Also, the whole SocketManager has been re-written to improve its performances and to allow multiple TCP/UDP connections in parallel (as you already know :wink:). A synchronized queue is now used to store data coming from the different sockets, and socket readers can freely put some data in the queue without any delay (except for when the queue reaches a -configurable- size, then the socket reader is blocked as long as the queue size is greater than the configured size). This queue is read each X ms, and X is configurable like described above.

LogMX v5.4.0 is still planned for July, there is currently no major delay for its release.

Xavier
rmitola
Posts: 17
Joined: Mon Jun 08, 2015 7:08 pm

Re: Improving Speed?

Post by rmitola »

Excellent; it's good to know it's not my code and that there will also be an added feature.

Thanks!
jobemakar
Posts: 27
Joined: Thu Jun 11, 2015 9:49 pm

Re: Improving Speed?

Post by jobemakar »

Thanks Xavier. Two questions:
[*]Since you're exposing the delay as configurable, would it also make sense to expose the number of entries per parse pass to allow?
[*]July is a a big month :) Can you share when you expect this release to occur?
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Improving Speed?

Post by admin »

jobemakar wrote:Thanks Xavier. Two questions:
  1. Since you're exposing the delay as configurable, would it also make sense to expose the number of entries per parse pass to allow?
  2. July is a a big month :) Can you share when you expect this release to occur?
Hello jobemakar,
  1. OK, we will add this.
  2. It is currently planned for July 17th
To answer you previous post (I saw it after I posted mine), I'm pretty confident that you will be able to monitor your logs fast enough using LogMX v5.4.0. If you want, we can deliver you a pre-release version in a few days so that you can do your own tests ASAP (I already have your e-mail address and also rmitola's address from our previous conversations).

Xavier
jobemakar
Posts: 27
Joined: Thu Jun 11, 2015 9:49 pm

Re: Improving Speed?

Post by jobemakar »

Absolutely we'd love to evaluate the pre-release, thanks! :)
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Improving Speed?

Post by admin »

All these new options are now implemented in v5.4.0, I've just sent you a Beta version by e-mail :wink:
Post Reply