Date filter for SMB directory manager

If you think LogMX needs a new useful feature!
Post Reply
chamm
Posts: 2
Joined: Wed Dec 14, 2016 2:36 pm

Date filter for SMB directory manager

Post by chamm »

I'm still learning this so far excellent application, but I think it would be incredibly useful if the SMB directory manager could include a date filter.

Both filtering only files between dates x and y, as well as within the last x days would be helpful.

Use case: I am using LogMX for logfiles in excess of 500MB per day over a remote SMB connection, across 8 different servers. The folders in question have 60 days of log files, so in order for me to set up a workspace, I would need to load an impossible amount of data. (500MB x 8 x 60 = ~240GB) Instead, I would set up the bookmark/workspace to only load files from the last day, and I could set those to only load the last 500 kb using the Tail settings.

There are also situations where I'll need to interlace files from those 8 different servers, but it's to look for an event that happened last Tuesday. Being able to filter the specific date(s) would make this possible, whereas now, I need to load and parse each logfile individually.

Overall, I'm quite impressed with the product so far. This humble suggestion would make it even better.

Thank you!
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Date filter for SMB directory manager

Post by admin »

Hello,

Thank you for your feedback and for this good idea :)
How would you like LogMX to filter the files by date?:
  1. Automatically parsing all the files to get the timestamp of its log entries?
  2. Using file meta-data: last-modification-date of each file?
  3. Using file name? (e.g. "app-logs-2016-12-14.log")
Solution 1 seems difficult to implement and could take a long time.
Solution 2 seems a bit tricky because the log file may be modified only a few seconds after the end of the day when the logger finished rolling it (not sure, it's just an idea, but using this date may not be reliable for other reasons)
Solution 3 could be easily implemented using the current "File filter" parameter (in "Open logs" > "SMB directory manager"): on top of the * character, we could also add \d \m \y \H \M \S special tokens so that the log file that will be opened every day could match a specific date relative to the current date, e.g. "app-logs-\y-\m-\d.log". Of course, selecting a time frame could be tricky, but we could also allow a suffix to specify an arithmetic operation, e.g. to get the last 3 days: "app-logs-{\y-\m-\d}{-3d}.log".

Please let me know what you think, and feel free to suggest other ideas if you have some, your thoughts are very useful to us!

Thanks,
Xavier
chamm
Posts: 2
Joined: Wed Dec 14, 2016 2:36 pm

Re: Date filter for SMB directory manager

Post by chamm »

Just to clarify the problem statement: I'd like to be able to open and parse single files, or even parts of single files, from a remote SMB directory, without needing to transfer the entire file.

Idea #1 - This would be awesome, but I can't think of a way it could be reasonably implemented. Theoretically, the application could selectively grab bits and pieces of timestamps from throughout the files without reading the entire file, but I don't even know if the SMB protocol allows an application to "reach in" and grab a piece of a file without reading the whole thing. If it could, it could grab timestamps, perform a smart search, and piece together an index to pull only bits of these huge logfiles. This would be a powerful long-term solution, but incredibly complex to write, and probably not really practical.

Idea #2 - This is probably the best "universal" way to implement this in a way that could be done fairly quickly. Files modified within the last x time would be a pretty powerful way to pull only selected logfiles. It would still require transferring some of the large files, but it won't try to read all of the, I believe, be useful to everyone, not just me. :D

Idea #3 - This idea would actually work quite well in my situation. All of my logfiles are named with a date format, and I had no idea that date variables could be used in a file filter. I'm pretty sure if I can figure out the syntax and variables of the parts of the date, this would meet my immediate need. If you can direct me to a reference for those variables and arithmetic operations and variables, I think I could probably figure out what I need to do.

Idea #4 - This isn't one that you suggested, but I also thought it would be super-cool if we could use regular expressions in the "smb-dir" manager. For example:

Code: Select all

smb-dir://DOMAIN\user@server.domain.local/folder/Logs/?regex=/^app-log-\d{4}-\d{2}-15\.log/
would only parse files from the 15th of the month. It's similar to idea #3, and maybe it's already possible, but I haven't figured out how to do it.

I'm going to stay engaged in this thread, and I appreciate the reply. I like the product so far, and I'm excited about the possibilities.
admin
Site Admin
Posts: 555
Joined: Sun Dec 17, 2006 10:30 pm

Re: Date filter for SMB directory manager

Post by admin »

Thanks for sharing your thoughts!
In fact, maybe I was not clear, but the idea #3 (date format tokens) is not possible at this time, I was just trying to image how we could use such a feature once it would be implemented. But since you think it would be a useful feature, I've just added this idea to our roadmap :) I' will let you know when it will be released by posting a message here (no date yet).

Concerning Idea #4, in fact you can already to that, but it's not as precise as a Java Regex: for now you can only use * (equivalent to .* in regexp syntax). So you can type this URL in the main LogMX toolbar:
smb-dir://DOMAIN\user@Host/ShareName//my/log/folder/?f=app-log-*-*-15.log?i=0?a=1
  • f= means file Filter
  • i= means Interlace merge (0/1) (i.e. sort log entries by timestamp)
  • a= means AutoRefresh (0/1)
You can have more details on this syntax in the "Open logs" dialog, at the top of the "SMB Directory Manager" page. To have an example, you can also use this dialog form to open the desired logs, then once the logs are opened, just copy the URL displayed in the main toolbar.
Post Reply