Skip to main content

Fault Log Collection

APO-OneAgent enables fault logs and full log collection by default. This document is used to configure APO-OneAgent to collect logs during fault occurrences. The product is based on ilogtail for secondary development to collect logs generated by applications.

Fault Scene Log Collection Configuration in Kubernetes Environment​

Users who have followed the 《APO OneAgent #Kubernetes》 should use this section to configure fault scene log collection.

  1. Modify the ConfigMap named apo-ilogtail-user-config and add a new configuration file <your-file-name>.yaml.
  2. Refer to the configuration template in this section and save the content to <your-file-name>.yaml.
  3. Restart OneAgent.

1. Kubernetes Standard Output Collection Configuration Template​

OneAgent enables Kubernetes standard output collection by default. If there are no additional requirements, manual configuration of this template is not necessary.

Container standard output uses the service_docker_stdout plugin. Reference Documentation

enable: true
inputs:
  - Type: service_docker_stdout
    # k8s label filtering
    ExcludeK8sLabel:
      app.kubernetes.io/instance: apo
    BeginLineCheckLength: 10
    BeginLineRegex: "\\d+.*"
processors:
  - Type: processor_log_to_prom_metric
    PromPort: ":9499"
  - Type: processor_wait_for_signal
    ContentsRename:
      "__tag__:_container_id_": "_container_id_"
      "__tag__:__path__": "_source_"
flushers:
  - Type: flusher_otlp
    Logs:
      # Server-side vector receiving logs, replace with the actual server address
      Endpoint: apo-otel-collector-gateway-svc:4315

2. Kubernetes Container Log File Collection Configuration Template​

Text log collection uses the file_log plugin. Reference Documentation

Note

  1. When collecting log files within containers, log files mounted on PVs are not supported.
  2. The maximum depth of the monitored log directory should not be too large, otherwise it may exceed the number of files monitored by ilogtail, leading to collection failure. Different configuration file pipelines can be added for different collection paths.
enable: true
inputs:
    # Text log collection plugin, adjust as needed
  - Type: file_log
    # Log directory path
    LogPath: /var/log/nginx/
    # Maximum depth of the monitored log directory, 0 means monitoring files in the current directory
    MaxDepth: 0
    # Log file name pattern
    FilePattern: "*.log"
    # Set to true for monitoring log files within containers
    ContainerFile: true
processors:
  - Type: processor_log_to_prom_metric
    PromPort: ":9499"
  - Type: processor_wait_for_signal
    ContentsRename:
      "__tag__:_container_id_": "_container_id_"
      "__tag__:__path__": "_source_"
flushers:
  - Type: flusher_otlp
    Logs:
      # Server-side vector to receive logs, replace with the actual server address
      Endpoint: apo-otel-collector-gateway-svc:4315

Fault Scene Log Collection Configuration in Docker Environment​

For users who have installed OneAgent using the 《APO OneAgent #Traditional》, configure fault scene log collection based on the monitored service type using the configurations in this section.

  1. In the OneAgent ilogtail configuration directory apo-one-agent-compose/config/apo-ilogtail, add a new configuration file <your-file-name>.yaml
  2. Refer to the configuration template in this section and save the content to <your-file-name>.yaml
  3. Restart OneAgent

1. Docker Standard Output Collection Template​

Suitable for collecting standard output from Docker containers on traditional servers. Use the service_docker_stdout plugin for container standard output, refer to documentation

enable: true
inputs:
    # Log collection plugin, adjust as needed
  - Type: service_docker_stdout
    # Multi-line log merging
    BeginLineCheckLength: 10
    BeginLineRegex: "\\d+.*"
processors:
  - Type: processor_log_to_prom_metric
    PromPort: ":9499"
  - Type: processor_wait_for_signal
    ContentsRename:
      "__tag__:_container_id_": "_container_id_"
      "__tag__:__path__": "_source_"
flushers:
  - Type: flusher_otlp
    Logs:
      # Server-side vector to receive logs, replace with the actual server address
      Endpoint: apo-otel-collector-gateway-svc:4315

2. Docker Container Log File Collection Template​

Use the file_log plugin for collecting text logs, refer to documentation

Note

It is advisable not to set the maximum depth of the monitored log directory too large, as it may exceed the number of files that ilogtail can monitor, leading to collection failures. You can add different configuration file pipelines for different collection paths.

enable: true
inputs:
    # Text log collection plugin, adjust as needed
  - Type: file_log
    # Log directory path
    LogPath: /var/log/nginx/
    # Maximum depth of the monitored log directory, 0 means monitoring files in the current directory
    MaxDepth: 0
    # Log file name pattern
    FilePattern: "*.log"
    # Set to true for monitoring log files within containers
    ContainerFile: true
processors:
  - Type: processor_log_to_prom_metric
    PromPort: ":9499"
  - Type: processor_wait_for_signal
    ContentsRename:
      "__tag__:_container_id_": "_container_id_"
      "__tag__:__path__": "_source_"
flushers:
  - Type: flusher_otlp
    Logs:
      # Server vector receiving logs, replace with the actual server address
      Endpoint: apo-otel-collector-gateway-svc:4315

Fault Scene Log Collection Configuration in Traditional Server Environments​

For users who have installed OneAgent using the 《APO OneAgent #Traditional》, configure fault scene log collection based on the monitored service type using the configurations in this section.

  1. In the OneAgent ilogtail configuration directory apo-one-agent-compose/config/apo-ilogtail, add a new configuration file <your-file-name>.yaml
  2. Refer to the configuration template in this section and save the content to <your-file-name>.yaml
  3. Restart OneAgent

1. Template for Collecting Log Files from Host Paths​

Suitable for traditional services that save log files to host paths. Use the file_log plugin for text log collection, refer to the documentation

Note

It is advisable not to set the maximum depth of the monitored log directory too large, as it may exceed the number of files that ilogtail can monitor, leading to collection failures. You can add different configuration file pipelines for different collection paths.

enable: true
inputs:
  # Text log collection plugin, adjust as needed
- Type: file_log
  # Path of the log file within the container where the ilogtail process resides
  # By default, the host directory is mounted to /logtail_host within the container
  # Therefore, prepend /logtail_host to the host path
  LogPath: /logtail_host/home/
  # Filename pattern for collecting text logs, supports wildcards
  # Supports multi-level directory matching, meaning all files matching the pattern in the specified directory (including all subdirectories) will be found.
  # The number of levels to search can be determined by MaxDepth
  FilePattern: "*.log"
  # Maximum depth of the directory to be monitored
  # For example, when the maximum depth is 2, even if all files match the filePattern
  # Only $LogPath/xx/xx/test.log will be collected
  # Logs from $LogPath/xx/xx/xx/test.log will not be collected
  MaxDepth: 4
processors:
  # Associate with the application's pid
- Type: processor_log_to_prom_metric
    PromPort: ":9499"
- Type: processor_path_to_pid 
- Type: processor_wait_for_signal
  ContentsRename:
    "__tag__:_container_id_": "_container_id_"
    "__tag__:__path__": "_source_"
flushers:
  - Type: flusher_otlp
    Logs:
      # Server-side vector to receive logs, replace with the actual server address
      Endpoint: apo-otel-collector-gateway-svc:4315