recently I talked about SmartLog in: Dr. StrangeLog or: How I Learned to Stop Worrying and Love the SmartLog
Provoking the Gods by writing about SmartLog being buggy sometimes, they punished us with SmartLog stopping to show/indexing fresh log entries.
Good indicators that something is wrong with SmartLog are:
- The window stays empty after start. It should always show you the most recent indexed logs!
- A generic search string like “dns” returns only results from Yesterday or older (nothing from today)
- The Index Rate in the lower right corner of SmartLog Windows shows 0 Logs being indexed per second (hard to miss as you dont always look down there)
For us luckily the solution was quite easy this time. As I already mentioned in the last post, the the Config file ($SMARTLOGDIR/conf/smartlog_settings.txt) contains the specific setting:
:min_disk_space (100240) – represents the minimal disk space SmartLog should keep free thus shrinking the Index when this limit is reached which results in logs being dropped from the SmartLog index and you will not be able to search for them anymore. However it will leave your original Logfiles untouched!
So I looked at our management Station’s disk space (df -h) and noticed we only had 95gigs left. After lowering this value and keeping a close eye on all the different Disk Space tresholds (I talked about this in Smart Event (Intro) Database self destruction) SmartLog started to index again after issuing a “smartlogstop;smartlogstart”
So if you base all of your firewall log queries on SmartLog nowadays (as we do) keep an eye open on the indicators I mentioned above so that you don’t accidentally miss something, tell a colleague you don’t see his traffic because SmartLog is just not indexing fresh logs anymore!
I personally would much more prefer SmartLog purging old log entries (fifo) than stopping indexing fresh logs alltogether!
Or is there already a way/hidden config parameter to make it do just this? Feel free to enlighten me with a comment!
btw: Up next will be an interesting ClusterXL/VSX/Cisco problem we are currently troubleshooting with Checkpoint. I will write about it as soon as we have a working solution.