A Next Generation Entropy Based Framework for Alert Detection in System Logs

Recent research efforts have highlighted capability of entropy based approaches in the automatic discovery of alerts in system logs. We refer to messages maybe of interest to an administrator as alerts. In the best case, they have been shown to detect all alerts at a false positive rate of 0%.

In this work, we extend the recent research to present the detailed evaluations of three entropy based approaches on new datasets not utilized in previous papers. We also extend the approach with the introduction of a Cluster Membership Anomaly score. This extension of the approach is intended to reduce the false positive rates required to detect all alerts. Previous work has shown that false positive rates required for the detection of all alerts for an entropy based approach could be very high. The results show that the Cluster Membership Anomaly has value for the reduction of false positive rates.