SmartLog not so smart – stops logging

Hello,

recently I talked about SmartLog in: Dr. StrangeLog or: How I Learned to Stop Worrying and Love the SmartLog

Provoking the Gods by writing about SmartLog being buggy sometimes, they punished us with SmartLog stopping to show/indexing fresh log entries.

Good indicators that something is wrong with SmartLog are:

  • The window stays empty after start. It should always show you the most recent indexed logs!
  • A generic search string like “dns” returns only results from Yesterday or older (nothing from today)
  • The Index Rate in the lower right corner of SmartLog Windows shows 0 Logs being indexed per second (hard to miss as you dont always look down there)

For us luckily the solution was quite easy this time. As I already mentioned in the last post, the the Config file ($SMARTLOGDIR/conf/smartlog_settings.txt) contains the specific setting:

:min_disk_space (100240) – represents the minimal  disk space SmartLog should keep free thus shrinking the Index when this limit is reached which results in logs being dropped from the SmartLog index and you will not be able to search for them anymore. However it will leave your original Logfiles untouched!

So I looked at our management Station’s disk space (df -h) and noticed we only had 95gigs left. After lowering this value and keeping a close eye on all the different Disk Space tresholds (I talked about this in Smart Event (Intro) Database self destruction) SmartLog started to index again after issuing a “smartlogstop;smartlogstart”

So if you base all of your firewall log queries on SmartLog nowadays (as we do) keep an eye open on the indicators I mentioned above so that you don’t accidentally miss something, tell a colleague you don’t see his traffic because SmartLog is just not indexing fresh logs anymore!

I personally would much more prefer SmartLog purging old log entries (fifo) than stopping indexing fresh logs alltogether!

Or is there already a way/hidden config parameter to make it do just this? Feel free to enlighten me with a comment!

btw: Up next will be an interesting ClusterXL/VSX/Cisco problem we are currently troubleshooting with Checkpoint. I will write about it as soon as we have a working solution.

Regards
Sebastian

Advertisements

About SebastianB

read it in my blog
This entry was posted in Checkpoint and tagged , , . Bookmark the permalink.

2 Responses to SmartLog not so smart – stops logging

  1. Dudi Hazan says:

    SmartLog won’t stop indexing when there is no more disk space. It will delete old indexes instead or function according to the disk maintenance policy defined (in the GUI from R77 and conf file before that).
    The stop indexing problem was related to a log switch that wasn’t handled properly. The fact that you did smartlogstop;smartlogstart brought you the resolution.

    Best regards,
    Dudi.

    • SebastianB says:

      Hello,

      you are probably right. Sad thing is we have this issue constantly since we upgraded the management station to R76 (upgrade export, upgrade import).

      A “fw logswitch” followed by a “smartlogstop;smartlogsstart” resolves it but this happens like nearly every night.

      Regards
      Sebastian

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s