ELK Stack · Part 5 of 5

Filebeat Log Shipper

Lightweight agent for collecting and forwarding log data from servers to Elasticsearch or Logstash with minimal resource usage.

Log Collector Lightweight Agent Modules Support SSL/TLS

What is Filebeat?

Filebeat

A lightweight agent installed on servers to collect logs from different sources and forward them to Elasticsearch or Logstash. Designed to be efficient with minimal resource usage.

Type: Log shipper / forwarder
Input Sources: App logs, system logs, Nginx, Docker logs, or custom log files
Outputs: Elasticsearch, Logstash, or other supported outputs
Modules: Predefined configs for Nginx, Apache, MySQL, System, Docker

01 Install Filebeat

bash

apt install filebeat -y

# Backup configuration
cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat_backup.yml

# Test connectivity
telnet ELK-SERVER 9200

# Basic testing
filebeat test config
filebeat test output

02 Custom Log Configuration

For collecting custom application logs with manual configuration.

bash

nano /etc/filebeat/filebeat.yml

Configuration Example

yaml

# filestream input for collecting log files
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/syslog

# =================================== Kibana ===================================
setup.kibana:
  host: "ELK-SERVER:5601"

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["ELK-SERVER:9200"]
  preset: balanced
  protocol: "https"
  ssl.certificate_authorities: ["/etc/elasticsearch/certs/http_ca.crt"]
  username: "elastic"
  password: "abcd@1234"

About setup.kibana

setup.kibana tells Filebeat where to load Kibana dashboards
Filebeat automatically imports pre-made dashboards, visualizations, index patterns, and searches into Kibana — so you can instantly view logs without building dashboards manually

03 Module-Based Configuration

Using prebuilt Filebeat modules for common log sources — system logs, Nginx, Apache, MySQL.

Advantages of Using Modules:

Predefined inputs, parsers, and dashboards
No manual filestream input configuration needed
Automatic log parsing with ingest pipelines
Pre-built Kibana dashboards for instant visualization

1 — List Available Modules

bash

filebeat modules list

2 — Enable System Module

bash

filebeat modules enable system

Creates /etc/filebeat/modules.d/system.yml with default settings

3 — System Module Configuration

yaml

- module: system
  syslog:
    enabled: true
    var.paths: ["/var/log/syslog*"]
  auth:
    enabled: true
    var.paths: ["/var/log/auth.log*"]

4 — Minimal filebeat.yml for Modules

When using modules, only configure outputs and Kibana connection:

yaml

setup.kibana:
  host: "ELK-SERVER:5601"

output.elasticsearch:
  hosts: ["ELK-SERVER:9200"]
  username: "elastic"
  password: "abcd@1234"
  ssl.certificate_authorities: ["/etc/elasticsearch/certs/http_ca.crt"]

Important:

You don't need to define filestream inputs for system logs
The module automatically creates inputs for syslog and auth logs
All module settings are in /etc/filebeat/modules.d/system.yml

5 — Setup Pipelines & Dashboards

bash

filebeat setup -e

What this does

Loads Elasticsearch ingest pipelines for log parsing
Imports Kibana dashboards for system logs
Configures index templates and patterns

6 — Start Filebeat Service

bash

systemctl daemon-reload
systemctl enable --now filebeat
systemctl start filebeat
systemctl restart filebeat
systemctl status filebeat

Filebeat is now running! It will automatically collect system logs and send them to Elasticsearch.

04 Testing & Verification

bash

# Test configuration
filebeat test config -e

# List modules
filebeat modules list

# View module configuration
cat /etc/filebeat/modules.d/system.yml

# Enable settings in module
sed -i '/enabled:/s/false/true/' /etc/filebeat/modules.d/system.yml
nano /etc/filebeat/modules.d/system.yml

Custom Log Paths

Specify custom paths in the module config: var.paths: ["/path/to/log1","/var/log/*.log"]

Configuration Approaches Comparison

Approach	Use Case	Config Required
Custom filestream	Custom app logs, non-standard paths	Full filebeat.yml with inputs
Module-based	Common apps (Nginx, MySQL, System)	Minimal — just outputs & Kibana
Direct to Logstash	When additional processing needed	Change output to logstash host:5044