The mtail project focuses on extracting monitoring data from application logs into a timeseries database like prometheus. Mtail does this by using a set of user-defined extraction programms reading from specified log files, named pipes or unix sockets. It’s intended to run one mtail per machine serving multiple applicatons.

Kubernetes has a documentation page describing it’s logging architecture. As a side note: All system components (kubelet, containerd) not running in containers write their logs to journald if systemd is present else they write a log file under /var/log. The documentation mentions two solutions a logging-agent like mtail can be executed.

  • As a side-car to the application, where logs are being shared through a emptyDir volume.

  • As a node-level logging agent using a daemonset and access to the nodes log directories using a hostPath volume.

Note that kubernetes exposes container logs at /var/log/containers these logs might be symlinks to /var/lib/docker/containers. Thus this directory must be made available through a volume mount to the logging-agent as well.

An example of mtail being configured as a logging-agent using a daemonset can be found on this github gist.