唯一的真理

True or False

ELK 开源日志实时分析系统

信息版本

名称 信息
系统 CentOS release 6.5 (Final)
Java java version “1.8.0_101”
Elasticsearch Elasticsearch 2.3.5
Logstash logstash-2.3.4
Kibana Kibana 4.5.4
Redis redis-3.2.3

相关参考连接

下载地址:ELK
ELK中文文档:ELK doc

主机名称 IP 应用内容
test01 10.20.161.122 logstash(indexer),elasticsearch,redis,kibana
test02 10.20.161.126 logstash(agent),nginx

关于ELK

  • ELK详细的网上都有,这边我只做简单的阐述,ELK是通过Elasticsearch,Logstash,Kibana,3个组件搭建的日志分析系统 是具有分布式,实时分析以及索引搜索的功能日志系统,非常强大,可以支持到PB级的日志记录,并且在页面轻松分析现有的所有日志

ELK架构想法

  • ELK传统架构中,通过logstash(agent),搜集分析日志并发送至elasticsearch做存储和生成索引用于搜索,kibana是一个基于elasticsearch开发的展示页面,经历的相当多版本,以及非常全面,我们可以将数据通过这个页面进行轻松交叉对比,释放繁重的手工命令分析日志
  • 考虑到传统架构中,还是有许多不足,在后面的版本中也发现支持redis等NO SQL存储用于队列,我调整了一下架构如下图(ELRK),由于测试Redis我没在这次测试中做成集群

ELRK

ELK install

test01

需要安装jdk

1
2
3
4
5
6

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
vim /etc/profile.d/jdk.sh
export JAVA_HOME=/var/lib/jvm/jdk1.8.0
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
source /etc/profile

安装Redis

1
2
3
4
5
6
7
8
9
10
11

tar xf redis-3.2.3.tar.gz
cp -rp redis-3.2.3  /home/app/
存放目录 /home/app/redis-3.2.3
cd /home/app/redis-3.2.3
make
make install
#优化系统
echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf
echo "net.core.somaxconn = 262144" >> /etc/sysctl.conf
echo never > /sys/kernel/mm/transparent_hugepage/enabled
sysctl -p
  • 启动命令 /home/app/redis-3.2.3/src/redis-server /home/app/redis-3.2.3/redis.conf &

Elasticsearch 安装

  • ElasticSearch默认的对外服务的HTTP端口是9200，节点间交互的TCP端口是9300，注意打开tcp端口。
  • Elasticsearch不允许以root运行（其实也可以运行，需要配置）
1
2
3

tar xf elasticsearch-2.3.5.tar.gz
mv elasticsearch-2.3.5 /home/app/
chown app.app -R /home/app/
  • 修改配置
1
2
3
4
5
6
7
8
9
10

cluster.name: test #集群名称
node.name: node-1  #节点名称
node.rack: k1    #机架名称
http.port: 9200      #监听端口
network.host: 10.20.161.122 #监听IP
discovery.zen.ping.unicast.hosts: ["10.20.161.122", "10.20.161.126"] #对心跳检测,尽量写IP
marvel.agent.exporters:
  id1:
    type: http
    host: ["http://10.20.161.122:9200", "http://10.20.161.126:9200"]

  • 启动命令 su - app -c "/home/app/elasticsearch-2.3.5/bin/elasticsearch -d"

  • 测试是否成功(如下是成功的)

1
2
3
4
5
6
7
8
9
10
11
12
13

curl -X GET http://10.20.161.122:9200
{
  "name" : "node-1",
  "cluster_name" : "test",
  "version" : {
    "number" : "2.3.5",
    "build_hash" : "90f439ff60a3c0f497f91663701e64ccd01edbb4",
    "build_timestamp" : "2016-07-27T10:36:52Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

1

/home/app/elasticsearch-2.3.5/bin/plugin install mobz/elasticsearch-head

Logstash安装(indexer)

  • Logstash默认的对外端口是9292，如果防火墙开启了要打开tcp端口。
1
2

tar xf logstash-2.3.4.tar.gz
cp -rp logstash-2.3.4 /home/app/

测试是否成功(如下说明成功)

1
2
3
4
5
6
7
8
9

ln -s /home/app/logstash-2.3.4/bin/* /usr/local/bin/
logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
手工输入hello world
{
       "message" => "hello world",
      "@version" => "1",
    "@timestamp" => "2016-08-08T08:43:12.785Z",
          "host" => "test01"
}

设置indexer角色

  • 定义自己设定的变量
1
2
3
4

mkdir /home/app/logstash-2.3.4/patterns
vim /home/app/logstash-2.3.4/patterns/my_pattern
DOMAIN \b(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62})(?:\.(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62}))*(\.?|\b)
NGINX_NUMBER [0-9.,-:]+
  • 设置logstash配置文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

mkdir /home/app/logstash-2.3.4/etc
vim /home/app/logstash-2.3.4/etc/logstash_indexer.conf

input {
        redis {
                host => "10.20.161.122"
                port => "6379"
                data_type => "list"
                key => "logstash:redis"
                type => "redis-input"
        }
}

filter {
    if [type] =~ "nginx_access_log" {
        grok {
                patterns_dir => "/home/app/logstash-2.3.4/patterns/my_pattern"
                match => { "message" => "(%{NGINX_NUMBER:request_time}) (?:%{NGINX_NUMBER:upstream_time}|-) (%{NGINX_NUMBER:client_ip}) %{DATA} (%{NGINX_NUMBER:upstream_host})  \[%{HTTPDATE:local_time}\] %{DOMAIN:domain} \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NGINX_NUMBER:http_version}))\" (%{NGINX_NUMBER:upstream_cache_status}) (?:%{NGINX_NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} \"(%{NGINX_NUMBER:gzip_ratio})\" \"(%{NGINX_NUMBER:x_forword})\" %{DATA} \"(%{NGINX_NUMBER:lvs_vip})\""}
        }
    }
}

output {
        elasticsearch {
                hosts => "10.20.161.122:9200"
        }
        stdout { codec => rubydebug }
}
  • 这次分析的日志是nginx,日志格式如下
1
2
3

log_format  test  '$request_time $upstream_response_time $remote_addr - $upstream_addr  [$time_local] '
                  '$host "$request" $status $bytes_sent '
                  '"$http_referer" "$http_user_agent" "$gzip_ratio" "$http_x_forwarded_for" - "$server_addr"';

  • 启动命令 nohup /home/app/logstash-2.3.4/bin/logstash -f /home/app/logstash-2.3.4/etc/logstash_indexer.conf &

  • redis查看是否有数据

1
2
3

redis-cli -h 127.0.0.1
exists logstash:redis
显示(integer) 1说明数据进去了

Kibana安装

1
2

tar xf kibana-4.5.4-linux-x64.tar.gz
mv kibana-4.5.4-linux-x64 /home/app/
  • 更改配置
1
2

vim /home/app/kibana-4.5.4-linux-x64/config/kibana.yml
elasticsearch.url: "http://10.20.161.122:9200" #数据读取的地方

test02

  • JDK,Elasticsearch,Logstash的安装参考test01

logstash(agent) 配置修改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

vim logstash_agent.conf

input {
        file {
                type => "nginx_access_log"
                path => ["/data/logs/nginx/access.log"]
        }
}
output {
        redis {
                host => "10.20.161.122:6379" #redis server
                data_type => "list"
                key => "logstash:redis"
        }
}
  • 启动命令 nohup logstash -f /home/app/logstash-2.3.4/etc/logstash_agent.conf &

Elasticsearch node2配置

1
2
3
4
5
6
7
8
9
10
11

vim elasticsearch-2.3.5/config/elasticsearch.yml
cluster.name: test
node.name: node-2
node.rack: k2
http.port: 9200
network.host: 10.20.161.126
discovery.zen.ping.unicast.hosts: ["10.20.161.122", "10.20.161.126"]
marvel.agent.exporters:
  id1:
    type: http
    host: ["http://10.20.161.122:9200", "http://10.20.161.126:9200"]

测试结构化是否正确(下面说明正确)

  • test01
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

/home/app/logstash-2.3.4/bin/logstash -f /home/app/logstash-2.3.4/etc/logstash_indexer.conf
"message" => "0.000 - 10.20.150.91 - -  [18/Aug/2016:11:44:10 +0800] test02.corp.test.com \"GET /favicon.ico HTTP/1.1\" 404 727 \"http://test02.corp.test.com/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36\" \"-\" \"-\" - \"10.20.161.126\"",
                 "@version" => "1",
               "@timestamp" => "2016-08-18T03:44:10.705Z",
                     "path" => "/data/logs/nginx/access.log",
                     "host" => "test02",
                     "type" => "nginx access log",
             "request_time" => "0.000",
            "upstream_time" => "-",
                "client_ip" => "10.20.150.91",
            "upstream_host" => "-",
               "local_time" => "18/Aug/2016:11:44:10 +0800",
                   "domain" => "test02.corp.test.com",
                     "verb" => "GET",
                  "request" => "/favicon.ico",
             "http_version" => "1.1",
    "upstream_cache_status" => "404",
                    "bytes" => "727",
                 "referrer" => "\"http://test02.corp.test.com/\"",
                    "agent" => "\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36\"",
               "gzip_ratio" => "-",
                "x_forword" => "-",
                  "lvs_vip" => "10.20.161.126"

最后展示结果

  • 扩展head e_head

  • Kibana展示结果 Kibana

返回顶部