Lognroll: Discovering Accurate Log Templates by Iterative Filtering

Abstract

Modern IT systems rely heavily on log analytics for critical operational tasks. Since the volume of logs produced from numerous distributed components is overwhelming, it requires us to employ automated processing. The first step of automated log processing is to convert streams of log lines into the sequence of log format IDs, called log templates. A log template serves as a base string with unfilled parts from which logs are generated during runtime by substitution of contextual information. The problem of log template discovery from the volume of collected logs poses a great challenge due to the semi-structured nature of the logs and the computational overheads. Our investigation reveals that existing techniques show various limitations. We approach the log template discovery problem as search-based learning by applying the ILP (Inductive Logic Programming) framework. The algorithm core consists of narrowing down the logs into smaller sets by analyzing value compositions on selected log column positions. Our evaluation shows that it produces accurate log templates from diverse application logs with small computational costs compared to existing methods. With the quality metric we defined, we obtained about 21%-51% improvements of log template quality.