Filebeat 一个日志收集工具的快速入门

1. Filebeat简介
Filebeat是一个本地系统日志的搬运工。通过安装到服务器上作为代码的角色,监控日志数据,并将它搬运到指定的接口,比如Elasticsearch,Logstash,Redis等。

prospector:启动Filebeat,将开启一个或者多个prospector发现指定目录的日志,仅仅读取本地文件,不支持远程的文件发现,同时负责决定是否需要开启一个harvester,Each prospector runs in its own Go routine;prospector还负责着文件的状态信息,如果数据发送失败,会从新发送;

harvester:每个日志文件会启动一个harvester读取数据,一行一行读取;

spooler:聚合事件,将数据发送到指定的输出;


2. 安装Filebeat
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.4.1-x86_64.rpm
sudo rpm -vi filebeat-5.4.1-x86_64.rpm
没有发现docker的官方安装方式


3. 配置Filebeat
filebeat使用YAML格式配置,大致分成input与output两大配置项。input表示读取日志,包括读取的方式;output表示输出日志,包括输出方式,输出的对象等。

启动前首先得做配置:
vi /etc/filebeat/filebeat.yml

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  template.enabled: true
  template.path: "filebeat.template.json"
  template.overwrite: false
  index: "filebeat"
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key"
#ssl这三个通常是不用配置的

多行日志配置,一个以时间2017-06-06开头的日志
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after

上面的配置将匹配下面的日志,它将把下面的两行当成一个事件数据,详细的使用见下图
2017-03-03 14:49:46,364 [grpc 2 (cq 2)] INFO  
 - use 0ms for SayHello with: { "Id": "123456" }.


4. 启动
sudo /etc/init.d/filebeat start


5. 停止
sudo /etc/init.d/filebeat stop

假定你安装了Elasticsearch,访问 http://192.168.0.62:9200/_cat/indices?v,如下图,可以看见日志已经写入Elasticsearch,索引名是filebeat开头


6. 目录结构
deb and rpm
TypeDescriptionLocation

home

Home of the Filebeat installation.

/usr/share/filebeat

bin

The location for the binary files.

/usr/share/filebeat/bin

conf

The location for configuration files.

/etc/filebeat

data

The location for persistent data files.

/var/lib/filebeat

logs

The location for the logs created by Filebeat.

/var/log/filebeat

docker
TypeDescriptionLocation

home

Home of the Filebeat installation.

/usr/share/filebeat

bin

The location for the binary files.

/usr/share/filebeat

conf

The location for configuration files.

/usr/share/filebeat

data

The location for persistent data files.

/usr/share/filebeat/data

logs

The location for the logs created by Filebeat.

/usr/share//filebeat/logs

Posted by 何敏 on 2017/06/07 07:16:26