Filebeat 一个日志收集工具的快速入门
1. Filebeat简介
Filebeat是一个本地系统日志的搬运工。通过安装到服务器上作为代码的角色,监控日志数据,并将它搬运到指定的接口,比如Elasticsearch,Logstash,Redis等。

prospector:启动Filebeat,将开启一个或者多个prospector发现指定目录的日志,仅仅读取本地文件,不支持远程的文件发现,同时负责决定是否需要开启一个harvester,Each prospector runs in its own Go routine;prospector还负责着文件的状态信息,如果数据发送失败,会从新发送;
harvester:每个日志文件会启动一个harvester读取数据,一行一行读取;
spooler:聚合事件,将数据发送到指定的输出;
2. 安装Filebeat
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.4.1-x86_64.rpm
sudo rpm -vi filebeat-5.4.1-x86_64.rpm
没有发现docker的官方安装方式
3. 配置Filebeat
filebeat使用YAML格式配置,大致分成input与output两大配置项。input表示读取日志,包括读取的方式;output表示输出日志,包括输出方式,输出的对象等。
启动前首先得做配置:
vi /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /var/log/*.log
output.elasticsearch:
hosts: ["http://localhost:9200"]
template.enabled: true
template.path: "filebeat.template.json"
template.overwrite: false
index: "filebeat"
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"
#ssl这三个通常是不用配置的
多行日志配置,一个以时间2017-06-06开头的日志
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
上面的配置将匹配下面的日志,它将把下面的两行当成一个事件数据,详细的使用见下图
2017-03-03 14:49:46,364 [grpc 2 (cq 2)] INFO
- use 0ms for SayHello with: { "Id": "123456" }.

4. 启动
sudo /etc/init.d/filebeat start
5. 停止
sudo /etc/init.d/filebeat stop
假定你安装了Elasticsearch,访问 http://192.168.0.62:9200/_cat/indices?v,如下图,可以看见日志已经写入Elasticsearch,索引名是filebeat开头

6. 目录结构
deb and rpm
docker
Type | Description | Location |
---|---|---|
home | Home of the Filebeat installation. |
|
bin | The location for the binary files. |
|
conf | The location for configuration files. |
|
data | The location for persistent data files. |
|
logs | The location for the logs created by Filebeat. |
|
Type | Description | Location |
---|---|---|
home | Home of the Filebeat installation. |
|
bin | The location for the binary files. |
|
conf | The location for configuration files. |
|
data | The location for persistent data files. |
|
logs | The location for the logs created by Filebeat. |
|
Posted by 何敏 on 2017/6/7 07:16:26