The mtail project focuses on extracting monitoring data from application logs into a timeseries database like prometheus. Mtail does this by using a set of user-defined extraction programms reading from specified log files, named pipes or unix sockets. It’s intended to run one mtail per machine serving multiple applicatons.
Kubernetes has a documentation page describing it’s logging architecture.
As a side note: All system components (kubelet, containerd) not running in
containers write their logs to journald if systemd is present else they write a
log file under
/var/log. The documentation mentions two solutions a
logging-agent like mtail can be executed.
As a side-car to the application, where logs are being shared through a emptyDir volume.
As a node-level logging agent using a daemonset and access to the nodes log directories using a hostPath volume.
Note that kubernetes exposes container logs at
/var/log/containers these logs
might be symlinks to
/var/lib/docker/containers. Thus this directory must be
made available through a volume mount to the logging-agent as well.
An example of mtail being configured as a logging-agent using a daemonset can be found on this github gist.