使用Elasticsearch + Logstash + Kibana搭建日志集中分析平臺實踐
在上周的上海Gopher Meetup的聚會上,聽了ASTA謝的演講。然后公司最近也需要實現一個日志集中分析平臺。ASTA謝恰好也講了他使用了Elasticsearch + Logstash + Kibana這個組合進行日志分析。回來之后就買了一本書然后各種google把它配置好了,當然只是把框架搭好了。這三個組建還有很多功能并沒有熟悉。本文只是簡單的介紹在Centos如果配置ELK(因為公司的服務器是Centos的,個人比較喜歡Ubuntu 哈哈)
什么是ELK:
Elasticsearch + Logstash + Kibana(ELK)是一套開源的日志管理方案,分析網站的訪問情況時我們一般會借助Google/百度/CNZZ等方式嵌入JS做數據統計,但是當網站訪問異常或者被攻擊時我們需要在后臺分析如Nginx的具體日志,而Nginx日志分割/GoAccess/Awstats都是相對簡單的單節點解決方案,針對分布式集群或者數據量級較大時會顯得心有余而力不足,而ELK的出現可以使我們從容面對新的挑戰。
- Logstash:負責日志的收集,處理和儲存
- Elasticsearch:負責日志檢索和分析
- Kibana:負責日志的可視化
官方網站:
JDK – http://www.oracle.com/technetwork/java/javase/downloads/index.html
Elasticsearch – https://www.elastic.co/downloads/elasticsearch
Logstash – https://www.elastic.co/downloads/logstash
Kibana – https://www.elastic.co/downloads/kibana
Nginx- https://www.nginx.com/
服務端配置:
安裝Java JDK:
cat /etc/redhat-release //這是我linux的版本 CentOS Linux release 7.1.1503 (Core) //我們通過yum 方式安裝Java Jdk yum install java-1.7.0-openjdk
cat/etc/redhat-release //這是我linux的版本 CentOS Linuxrelease7.1.1503(Core) //我們通過yum 方式安裝Java Jdk yum installjava-1.7.0-openjdk
Elasticsearch安裝:
#下載安裝 wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm yum localinstall elasticsearch-1.7.1.noarch.rpm #啟動相關服務 service elasticsearch start service elasticsearch status #查看Elasticsearch的配置文件 rpm -qc elasticsearch /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml /etc/init.d/elasticsearch /etc/sysconfig/elasticsearch /usr/lib/sysctl.d/elasticsearch.conf /usr/lib/systemd/system/elasticsearch.service /usr/lib/tmpfiles.d/elasticsearch.conf #查看端口使用情況 netstat -nltp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1817/master tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 27369/node tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 31848/nginx: master tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16567/sshd tcp6 0 0 127.0.0.1:8005 :::* LISTEN 8263/java tcp6 0 0 :::5000 :::* LISTEN 2771/java tcp6 0 0 :::8009 :::* LISTEN 8263/java tcp6 0 0 :::3306 :::* LISTEN 28839/mysqld tcp6 0 0 :::80 :::* LISTEN 31848/nginx: master tcp6 0 0 :::8080 :::* LISTEN 8263/java tcp6 0 0 :::9200 :::* LISTEN 25808/java tcp6 0 0 :::9300 :::* LISTEN 25808/java tcp6 0 0 :::9301 :::* LISTEN 2771/java tcp6 0 0 :::22 :::* LISTEN 16567/sshd
#下載安裝 wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm yum localinstallelasticsearch-1.7.1.noarch.rpm #啟動相關服務 service elasticsearchstart service elasticsearchstatus #查看Elasticsearch的配置文件 rpm-qcelasticsearch /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml /etc/init.d/elasticsearch /etc/sysconfig/elasticsearch /usr/lib/sysctl.d/elasticsearch.conf /usr/lib/systemd/system/elasticsearch.service /usr/lib/tmpfiles.d/elasticsearch.conf #查看端口使用情況 netstat-nltp Active Internetconnections(onlyservers) Proto Recv-QSend-QLocalAddress ForeignAddress State PID/Programname tcp 0 0127.0.0.1:25 0.0.0.0:* LISTEN 1817/master tcp 0 00.0.0.0:5601 0.0.0.0:* LISTEN 27369/node tcp 0 00.0.0.0:80 0.0.0.0:* LISTEN 31848/nginx:master tcp 0 00.0.0.0:22 0.0.0.0:* LISTEN 16567/sshd tcp6 0 0127.0.0.1:8005 :::* LISTEN 8263/java tcp6 0 0:::5000 :::* LISTEN 2771/java tcp6 0 0:::8009 :::* LISTEN 8263/java tcp6 0 0:::3306 :::* LISTEN 28839/mysqld tcp6 0 0:::80 :::* LISTEN 31848/nginx:master tcp6 0 0:::8080 :::* LISTEN 8263/java tcp6 0 0:::9200 :::* LISTEN 25808/java tcp6 0 0:::9300 :::* LISTEN 25808/java tcp6 0 0:::9301 :::* LISTEN 2771/java tcp6 0 0:::22 :::* LISTEN 16567/sshd
我們看到9200端口了說明我們安裝成功了,我們可以在終端輸入
#測試訪問 curl -X GET http://localhost:9200/
#測試訪問 curl-XGEThttp://localhost:9200/
或者直接瀏覽器打開我們可以看到
{
status: 200,
name: "Pip the Troll",
cluster_name: "elasticsearch",
version: {
number: "1.7.2",
build_hash: "e43676b1385b8125d647f593f7202acbd816e8ec",
build_timestamp: "2015-09-14T09:49:53Z",
build_snapshot: false,
lucene_version: "4.10.4"
},
tagline: "You Know, for Search"
}{
status:200,
name:"Pip the Troll",
cluster_name:"elasticsearch",
version:{
number:"1.7.2",
build_hash:"e43676b1385b8125d647f593f7202acbd816e8ec",
build_timestamp:"2015-09-14T09:49:53Z",
build_snapshot:false,
lucene_version:"4.10.4"
},
tagline:"You Know, for Search"
}說明我們的程序是運行正常的。
Kibana的安裝:
#下載tar包
wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
#解壓
tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/
cd /usr/local/
mv kibana-4.1.1-linux-x64 kibana
#創建kibana服務
vim /etc/rc.d/init.d/kibana
#!/bin/bash
### BEGIN INIT INFO
# Provides: kibana
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Runs kibana daemon
# Description: Runs the kibana daemon as a non-root user
### END INIT INFO
# Process name
NAME=kibana
DESC="Kibana4"
PROG="/etc/init.d/kibana"
# Configure location of Kibana bin
KIBANA_BIN=/usr/local/kibana/bin
# PID Info
PID_FOLDER=/var/run/kibana/
PID_FILE=/var/run/kibana/$NAME.pid
LOCK_FILE=/var/lock/subsys/$NAME
PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
DAEMON=$KIBANA_BIN/$NAME
# Configure User to run daemon process
DAEMON_USER=root
# Configure logging location
KIBANA_LOG=/var/log/kibana.log
# Begin Script
RETVAL=0
if [ `id -u` -ne 0 ]; then
echo "You need root privileges to run this script"
exit 1
fi
# Function library
. /etc/init.d/functions
start() {
echo -n "Starting $DESC : "
pid=`pidofproc -p $PID_FILE kibana`
if [ -n "$pid" ] ; then
echo "Already running."
exit 0
else
# Start Daemon
if [ ! -d "$PID_FOLDER" ] ; then
mkdir $PID_FOLDER
fi
daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &
sleep 2
pidofproc node > $PID_FILE
RETVAL=$?
[[ $? -eq 0 ]] && success || failure
echo
[ $RETVAL = 0 ] && touch $LOCK_FILE
return $RETVAL
fi
}
reload()
{
echo "Reload command is not implemented for this service."
return $RETVAL
}
stop() {
echo -n "Stopping $DESC : "
killproc -p $PID_FILE $DAEMON
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p $PID_FILE $DAEMON
RETVAL=$?
;;
restart)
stop
start
;;
reload)
reload
;;
*)
# Invalid Arguments, print the following message.
echo "Usage: $0 {start|stop|status|restart}" >&2
exit 2
;;
esac
#修改啟動權限
chmod +x /etc/rc.d/init.d/kibana
#啟動kibana服務
service kibana start
service kibana status
#查看端口
netstat -nltp#下載tar包
wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
#解壓
tar zxfkibana-4.1.1-linux-x64.tar.gz-C/usr/local/
cd/usr/local/
mv kibana-4.1.1-linux-x64kibana
#創建kibana服務
vim/etc/rc.d/init.d/kibana
#!/bin/bash
### BEGIN INIT INFO
# Provides: kibana
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Runs kibana daemon
# Description: Runs the kibana daemon as a non-root user
### END INIT INFO
# Process name
NAME=kibana
DESC="Kibana4"
PROG="/etc/init.d/kibana"
# Configure location of Kibana bin
KIBANA_BIN=/usr/local/kibana/bin
# PID Info
PID_FOLDER=/var/run/kibana/
PID_FILE=/var/run/kibana/$NAME.pid
LOCK_FILE=/var/lock/subsys/$NAME
PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
DAEMON=$KIBANA_BIN/$NAME
# Configure User to run daemon process
DAEMON_USER=root
# Configure logging location
KIBANA_LOG=/var/log/kibana.log
# Begin Script
RETVAL=0
if[`id-u`-ne0];then
echo"You need root privileges to run this script"
exit1
fi
# Function library
./etc/init.d/functions
start(){
echo-n"Starting $DESC : "
pid=`pidofproc-p$PID_FILEkibana`
if[-n"$pid"];then
echo"Already running."
exit0
else
# Start Daemon
if[!-d"$PID_FOLDER"];then
mkdir$PID_FOLDER
fi
daemon--user=$DAEMON_USER--pidfile=$PID_FILE$DAEMON1>"$KIBANA_LOG"2>&1&
sleep2
pidofprocnode>$PID_FILE
RETVAL=$?
[[$?-eq0]]&&success||failure
echo
[$RETVAL=0]&&touch$LOCK_FILE
return$RETVAL
fi
}
reload()
{
echo"Reload command is not implemented for this service."
return$RETVAL
}
stop(){
echo-n"Stopping $DESC : "
killproc-p$PID_FILE$DAEMON
RETVAL=$?
echo
[$RETVAL=0]&&rm-f$PID_FILE$LOCK_FILE
}
case"$1"in
start)
start
;;
stop)
stop
;;
status)
status-p$PID_FILE$DAEMON
RETVAL=$?
;;
restart)
stop
start
;;
reload)
reload
;;
*)
# Invalid Arguments, print the following message.
echo"Usage: $0 {start|stop|status|restart}">&2
exit2
;;
esac
#修改啟動權限
chmod+x/etc/rc.d/init.d/kibana
#啟動kibana服務
service kibanastart
service kibanastatus
#查看端口
netstat-nltp因為剛剛已經執行過
netstat -nltp
netstat-nltp
所以顯示的效果我這里就不貼了,如果我們能看到5601端口就說明我們安裝成功了。
Option 1:Generate SSL Certificates:
生成SSL證書是為了服務端和客戶端進行驗證:
sudo vi /etc/pki/tls/openssl.cnf
sudo vi/etc/pki/tls/openssl.cnf
Find the[ v3_ca ]section in the file, and add this line under it (substituting in the Logstash Server’s private IP address):
subjectAltName = IP: logstash_server_private_ip
subjectAltName=IP:logstash_server_private_ip
cd /etc/pki/tls sudo openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
cd/etc/pki/tls sudo opensslreq-config/etc/pki/tls/openssl.cnf-x509-days3650-batch-nodes-newkeyrsa:2048-keyoutprivate/logstash-forwarder.key-outcerts/logstash-forwarder.crt
Option 2: FQDN (DNS):
cd /etc/pki/tls sudo openssl req -subj '/CN=<^>logstash_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
cd/etc/pki/tls sudo opensslreq-subj'/CN=<^>logstash_server_fqdn/'-x509-days3650-batch-nodes-newkeyrsa:2048-keyoutprivate/logstash-forwarder.key-outcerts/logstash-forwarder.crt
Logstash安裝:
Logstash Forwarder(客戶端):
安裝Logstash Forwarder
wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm
yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm
#查看logstash-forwarder的配置文件位置
rpm -qc logstash-forwarder
/etc/logstash-forwarder.conf
#備份配置文件
cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.save
#編輯 /etc/logstash-forwarder.conf,需要根據實際情況進行修改
vim /etc/logstash-forwarder.conf
{
"network": {
"servers": [ "這里寫服務器的ip:5000" ],
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},
"files": [
{
"paths": [
"/var/log/messages",
"/var/log/secure"
],
"fields": { "type": "syslog" }
}
]
}安裝LogstashForwarder
wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm
yum localinstalllogstash-forwarder-0.4.0-1.x86_64.rpm
#查看logstash-forwarder的配置文件位置
rpm-qclogstash-forwarder
/etc/logstash-forwarder.conf
#備份配置文件
cp/etc/logstash-forwarder.conf/etc/logstash-forwarder.conf.save
#編輯 /etc/logstash-forwarder.conf,需要根據實際情況進行修改
vim/etc/logstash-forwarder.conf
{
"network":{
"servers":["這里寫服務器的ip:5000"],
"ssl ca":"/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout":15
},
"files":[
{
"paths":[
"/var/log/messages",
"/var/log/secure"
],
"fields":{"type":"syslog"}
}
]
}Logstash Server(服務端):
#下載rpm包
wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm
#安裝
yum localinstall logstash-1.5.4-1.noarch.rpm
#創建一個01-logstash-initial.conf文件
vim /etc/logstash/conf.d/01-logstash-initial.conf
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
#啟動logstash服務
service logstash start
service logstash status
#訪問Kibana,Time-field name 選擇 @timestamp 要在下一步操作 Nginx 日志配置之后訪問 不然會沒有數據不能創建
http://localhost:5601/
#增加節點和客戶端配置一樣,注意同步證書(可以通過SSH的方式同步)
/etc/pki/tls/certs/logstash-forwarder.crt#下載rpm包
wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm
#安裝
yum localinstalllogstash-1.5.4-1.noarch.rpm
#創建一個01-logstash-initial.conf文件
vim/etc/logstash/conf.d/01-logstash-initial.conf
input{
lumberjack{
port=>5000
type=>"logs"
ssl_certificate=>"/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key=>"/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter{
if[type]=="syslog"{
grok{
match=>{"message"=>"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}"}
add_field=>["received_at","%{@timestamp}"]
add_field=>["received_from","%{host}"]
}
syslog_pri{}
date{
match=>["syslog_timestamp","MMM d HH:mm:ss","MMM dd HH:mm:ss"]
}
}
}
output{
elasticsearch{host=>localhost}
stdout{codec=>rubydebug}
}
#啟動logstash服務
service logstashstart
service logstashstatus
#訪問Kibana,Time-field name 選擇 @timestamp 要在下一步操作 Nginx 日志配置之后訪問 不然會沒有數據不能創建
http://localhost:5601/
#增加節點和客戶端配置一樣,注意同步證書(可以通過SSH的方式同步)
/etc/pki/tls/certs/logstash-forwarder.crt配置Nginx日志:
#修改客戶端配置
vim /etc/logstash-forwarder.conf
{
"network": {
"servers": [ "自己服務器的ip地址:5000" ],
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},
"files": [
{
"paths": [
"/var/log/messages",
"/var/log/secure"
],
"fields": { "type": "syslog" }
}, {
"paths": [
"/app/local/nginx/logs/access.log"
],
"fields": { "type": "nginx" }
}
]
}
#服務端增加patterns
mkdir /opt/logstash/patterns
vim /opt/logstash/patterns/nginx
NGUSERNAME [a-zA-Z.@-+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:remote_addr} - - [%{HTTPDATE:time_local}] "%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}
#修改logstash權限
chown -R logstash:logstash /opt/logstash/patterns
#修改服務端配置
vim /etc/logstash/conf.d/01-logstash-initial.conf
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
if [type] == "nginx" {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}#修改客戶端配置
vim/etc/logstash-forwarder.conf
{
"network":{
"servers":["自己服務器的ip地址:5000"],
"ssl ca":"/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout":15
},
"files":[
{
"paths":[
"/var/log/messages",
"/var/log/secure"
],
"fields":{"type":"syslog"}
},{
"paths":[
"/app/local/nginx/logs/access.log"
],
"fields":{"type":"nginx"}
}
]
}
#服務端增加patterns
mkdir/opt/logstash/patterns
vim/opt/logstash/patterns/nginx
NGUSERNAME[a-zA-Z.@-+_%]+
NGUSER%{NGUSERNAME}
NGINXACCESS%{IPORHOST:remote_addr}--[%{HTTPDATE:time_local}]"%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}"%{INT:status}%{INT:body_bytes_sent}%{QS:http_referer}%{QS:http_user_agent}
#修改logstash權限
chown-Rlogstash:logstash/opt/logstash/patterns
#修改服務端配置
vim/etc/logstash/conf.d/01-logstash-initial.conf
input{
lumberjack{
port=>5000
type=>"logs"
ssl_certificate=>"/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key=>"/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter{
if[type]=="syslog"{
grok{
match=>{"message"=>"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}"}
add_field=>["received_at","%{@timestamp}"]
add_field=>["received_from","%{host}"]
}
syslog_pri{}
date{
match=>["syslog_timestamp","MMM d HH:mm:ss","MMM dd HH:mm:ss"]
}
}
if[type]=="nginx"{
grok{
match=>{"message"=>"%{NGINXACCESS}"}
}
}
}
output{
elasticsearch{host=>localhost}
stdout{codec=>rubydebug}
}我們看一下完成配置之后的效果:
好了,我是折騰了2天才折騰出來的,感覺自己好笨。寫篇總結為了下一次能夠快速的搭建起來。
我們可以ton
使用Elasticsearch + Logstash + Kibana搭建日志集中分析平臺實踐