Deep Dive into Kubernetes logs Structure or its Location

Comprehensive Tutorial: Understanding Kubernetes Logs and Their Locations

Kubernetes generates various logs for troubleshooting, monitoring, and maintaining the health of the cluster. Each log type provides unique insights into different components of the Kubernetes ecosystem. This tutorial explains the directory structure and the purpose of the logs based on the provided image.

1. Log Directory Structure Overview

In Kubernetes, logs are typically stored in the /var/log/ directory. These logs are categorized by components such as containers, kubelet, API server, etc. Here’s the detailed breakdown:

a. `/var/log/containers/`

Structure: <pod-name>_<namespace>_<container-name>-<container-id>.log
Purpose: Stores logs for each container running in Kubernetes pods.
Use Case:
- Debugging container-specific issues.
- Viewing stdout and stderr output for individual containers.

b. `/var/log/pods/`

Structure: <namespace>_<pod-name>_<uid>/<container-name>-stdout.log and <container-name>-stderr.log
Purpose:
- stdout.log: Logs standard output of the container.
- stderr.log: Logs standard error output of the container.
Use Case:
- Isolating logs for specific pods and their containers for debugging.

c. `/var/log/kubelet/`

Log Files:
- kubelet.log: Main log file for the kubelet service.
- audit.log: Logs audit events for kubelet actions.
- error.log: Records error logs for kubelet.
Purpose:
- Tracks node-level events handled by kubelet.
- Useful for troubleshooting node-to-cluster communication.
Use Case:
- Diagnosing issues related to pod scheduling and resource allocation.

d. `/var/log/kube-apiserver/`

Log Files:
- apiserver.log: Main API server event logs.
- audit.log: Logs API request audit trails.
- error.log: Captures errors for the API server.
Purpose:
- Tracks all requests and responses managed by the Kubernetes API server.
Use Case:
- Identifying unauthorized API access or request failures.

e. `/var/log/kube-scheduler/`

Log Files:
- scheduler.log: Logs scheduler activity for pod placement.
- error.log: Captures scheduler-related errors.
Purpose:
- Monitors pod placement decisions and scheduling failures.
Use Case:
- Debugging issues with pod scheduling and resource allocation.

f. `/var/log/kube-controller-manager/`

Log Files:
- controller-manager.log: Logs activities of controllers in the cluster.
- error.log: Records errors related to the controller manager.
Purpose:
- Provides insights into resource controllers, like ReplicaSet and Deployment controllers.
Use Case:
- Troubleshooting issues with resource scaling, replication, and garbage collection.

g. `/var/log/etcd/`

Log Files:
- etcd.log: Main log file for the etcd key-value store.
- snapshot.log: Tracks etcd snapshots for backup and recovery.
- error.log: Records etcd-related errors.
Purpose:
- Tracks events in the etcd datastore, which stores cluster state information.
Use Case:
- Debugging issues with cluster state synchronization or data corruption.

h. `/var/log/containerd/`

Log Files:
- containerd.log: Logs container runtime activities.
- error.log: Captures container runtime errors.
Purpose:
- Monitors container runtime operations like pulling images and managing containers.
Use Case:
- Debugging container startup and runtime issues.

i. `/var/log/network/`

Log Files:
- cni.log: Logs from Container Network Interface (CNI) plugins.
- flannel.log, calico.log: Logs specific to network providers.
- error.log: Captures network-related errors.
Purpose:
- Tracks network configuration and connectivity for pods and nodes.
Use Case:
- Troubleshooting pod network issues and overlay network configurations.

j. `/var/log/node/`

Log Files:
- syslog: General system logs (Ubuntu/Debian).
- messages: System logs for RHEL/CentOS.
- dmesg.log: Logs hardware and boot events.
- auth.log: Tracks SSH and sudo actions.
- error.log: Node-level error logs.
Purpose:
- Provides a detailed view of node operations and system-level activities.
Use Case:
- Debugging hardware, boot, or authentication issues on nodes.

2. How to Access Kubernetes Logs

You can access Kubernetes logs using different tools and methods:

a. Accessing Logs via CLI

Pod Logs: kubectl logs <pod-name> -n <namespace> Use -c <container-name> if the pod has multiple containers.
Node Logs: Access logs directly from the node via SSH: ssh <node-ip> cd /var/log/

b. Using Kubernetes Dashboards

Tools like Lens or Kubernetes Dashboard provide a GUI to view logs.

c. Aggregated Logs with Log Management Tools

Use tools like ELK Stack, Fluentd, or Promtail to aggregate logs.

3. Best Practices for Managing Kubernetes Logs

Enable Log Rotation:
- Prevent disk space exhaustion by configuring log rotation: vi /etc/logrotate.d/kubelet
Use Centralized Logging:
- Aggregate logs with tools like Fluentd, Elastic Stack, or Cloud Logging.
Define Retention Policies:
- Set up policies to retain logs based on business or compliance needs.
Monitor Critical Logs:
- Focus on key components like kubelet, API server, and scheduler.
Secure Log Access:
- Restrict log access to authorized personnel only.

Here is a detailed table summarizing the log file locations, purposes, and use cases for Kubernetes logs:

Comprehensive Table of Kubernetes Logs

Log Directory/Location	Log File(s)	Purpose	Use Case
/var/log/containers/	`<pod-name>_<namespace>_<container-name>-<container-id>.log`	Logs for each container in the pods.	Debugging container-specific issues.
/var/log/pods/	`<namespace>_<pod-name>_<uid>/<container-name>-stdout.log<namespace>_<pod-name>_<uid>/<container-name>-stderr.log`	`stdout.log`: Logs standard output.`stderr.log`: Logs standard error.	Debugging container logs specific to pods.
/var/log/kubelet/	`kubelet.logaudit.logerror.log`	`kubelet.log`: Node-level kubelet events.`audit.log`: Logs kubelet actions.`error.log`: Kubelet error logs.	Diagnosing issues with pod scheduling and node management.
/var/log/kube-apiserver/	`apiserver.logaudit.logerror.log`	`apiserver.log`: API server events.`audit.log`: API request trails.`error.log`: API server errors.	Monitoring API requests, failures, and unauthorized access.
/var/log/kube-scheduler/	`scheduler.logerror.log`	`scheduler.log`: Logs pod placement decisions.`error.log`: Scheduler-related errors.	Debugging pod scheduling failures and resource allocation.
/var/log/kube-controller-manager/	`controller-manager.logerror.log`	`controller-manager.log`: Logs controller activities.`error.log`: Errors in controller manager.	Troubleshooting replication, scaling, and resource management.
/var/log/etcd/	`etcd.logsnapshot.logerror.log`	`etcd.log`: Main etcd datastore logs.`snapshot.log`: Logs for etcd snapshots.`error.log`: Logs etcd errors.	Debugging cluster state synchronization and data corruption issues.
/var/log/containerd/	`containerd.logerror.log`	`containerd.log`: Logs container runtime events.`error.log`: Errors related to container runtime.	Debugging container runtime (image pulling, container start).
/var/log/network/	`cni.logflannel.logcalico.logerror.log`	`cni.log`: Container Network Interface logs.`flannel/calico.log`: Specific network provider logs.`error.log`: Network errors.	Troubleshooting pod networking and connectivity issues.
/var/log/node/	`syslogmessagesdmesg.logauth.logerror.log`	`syslog/messages`: General system logs.`dmesg.log`: Hardware/boot info.`auth.log`: SSH and sudo authentication logs.`error.log`: Node-level error logs.	Debugging hardware, authentication, and boot issues.

Detailed Explanation of Logs

Log Type	Component/Service	Key Focus
Container Logs	Individual Pods/Containers	Application-level logs (stdout and stderr).
Node Logs	Kubelet and System Logs	Logs at the node level for resource management.
Control Plane Logs	Scheduler, Controller Manager	Logs for cluster-level management and decisions.
API Server Logs	API Server	Tracks API requests, errors, and audit trails.
Networking Logs	CNI Plugins (Flannel, Calico)	Network-related logs and troubleshooting.
Datastore Logs	etcd	Logs related to cluster state storage and retrieval.
Runtime Logs	containerd	Logs for container lifecycle management.

Rajesh Kumar