地址库
在ELK中,我们可以使用地址库,来对IP进行分析,对日志进行分析,在ELKstack中只有Logstash可以做到,但是出图,是Kibana来出的,所以我们首先需要下载地址库数据文件,然后对Logstash进行配置,使用geoip模块对日志访问IP进行分析后,再以中国地图或者是世界地图的形式,展现在Kibana中。
下载地址库
Logstash2版本下载地址:http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
logstash5版本下载地址:http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
#进入Logstash目录
[root@linux-node1 ~]# cd /etc/logstash/
#下载地址库
[root@linux-node1 logstash]# wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
#解压地址库文件
[root@linux-node1 logstash]# tar xf GeoLite2-City.tar.gz
#查看地址库文件
[root@linux-node1 logstash]# ll
[root@linux-node1 logstash]# ll
total 28912
drwxrwxr-x 2 root root 4096 Aug 2 19:32 conf.d
-rw-r--r-- 1 root root 1694 Aug 2 19:03 conf.d_tar.gz
drwxr-xr-x 2 root root 90 Jul 30 04:24 GeoLite2-City_20190730
-rw-r--r-- 1 root root 29577038 Jul 30 04:24 GeoLite2-City.tar.gz
-rw-r--r-- 1 root root 1738 May 22 2017 jvm.options
-rw-r--r-- 1 root root 1334 May 22 2017 log4j2.properties
-rw-r--r-- 1 root root 4484 Aug 1 15:01 logstash.yml
-rw-r--r-- 1 root root 1659 May 22 2017 startup.options
配置Logstash使用地址库
配置Logstash
#进入Logstash配置文件目录
[root@linux-node1 logstash]# cd /etc/logstash/conf.d/
#编辑Logstash配置文件
[root@linux-node1 conf.d]# vim redis_es_ip.conf
input {
redis {
host => "192.168.6.243"
port => "6379"
db => "3"
key => "all"
data_type => "list"
password => "zls"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLite2-City_20190730/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}
output {
elasticsearch {
hosts => ["192.168.6.241:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}
#启动Logstash
[root@linux-node1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis_es_ip.conf &
因为是单机环境,日志里面没有公网IP,所以我们需要自己往里输入公网IP
#北京公网IP
[root@linux-node1 conf.d]# echo '{"@timestamp":"2019-04-11T20:27:25+08:00","host":"222.28.0.112","clientip":"222.28.0.112","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"www.elk.com","url":"/index.html","domain":"www.elk.com","xff":"10.0.0.1","referer":"-","status":"304"}' >> /usr/local/nginx/logs/access_json.log
#海南公网IP
[root@linux-node1 conf.d]# echo '{"@timestamp":"2019-04-11T20:40:24+08:00","host":" 124.225.0.13","clientip":"124.225.0.13","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"www.elk.com","url":"/index.html","domain":"www.elk.com","xff":"10.0.0.1","referer":"-","status":"304"}' >> /usr/local/nginx/logs/access_json.log
#吉林公网IP
[root@linux-node1 conf.d]# echo '{"@timestamp":"2019-04-11T20:45:24+08:00","host":" 124.234.0.12","clientip":"124.234.0.12","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"www.elk.com","url":"/index.html","domain":"www.elk.com","xff":"10.0.0.1","referer":"-","status":"304"}' >> /usr/local/nginx/logs/access_json.log
#黑龙江公网IP
[root@linux-node1 conf.d]# echo '{"@timestamp":"2019-04-11T20:46:24+08:00","host":" 123.164.0.18","clientip":"123.164.0.18","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"www.elk.com","url":"/index.html","domain":"www.elk.com","xff":"10.0.0.1","referer":"-","status":"304"}' >> /usr/local/nginx/logs/access_json.log
验证Kibana中的数据
打开浏览器,访问:http://192.168.6.243:5601
配置Kibana使用地图
Kibana画中国地图
如图:报错:”No Compatible Fields: The “[report-]YYYY.MM.DD” index pattern does not contain any of the following field types: geo_point”
原因:索引格式为[report-]YYYY.MM.DD 的日志文件由logstash输出到Elasticsearch;在elasticsearch中,所有的数据都有一个类型,什么样的类型,就可以在其上做一些对应类型的特殊操作。geo信息中的location字段是经纬度,我们需要使用经纬度来定位地理位置;在elasticsearch中,对于经纬度来说,要想使用elasticsearch提供的地理位置查询相关的功能,就需要构造一个结构,并且将其类型属性设置为geo_point,此错误明显是由于我们的geo的location字段类型不是geo_point。
我们可以通过以下方式验证一下:
[root@linux-node1 conf.d]# curl -XGET http://192.168.6.241:9200/report-2019.08.05/_mapping/
{"report-2019.08.05":{"mappings":{"report.mcake":{"properties":{"@timestamp":{"type":"date"},"@version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"beat":{"properties":{"hostname":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"clientip":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"domain":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"geoip":{"properties":{"city_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"continent_code":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"coordinates":{"type":"float"},"country_code2":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"country_code3":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"country_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ip":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"latitude":{"type":"float"},"location":{"type":"float"},"longitude":{"type":"float"},"region_code":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"region_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"timezone":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"host":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"http_host":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"input_type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"offset":{"type":"long"},"referer":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"responsetime":{"type":"float"},"size":{"type":"long"},"source":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"status":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"tags":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"upstreamhost":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"upstreamtime":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"xff":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}
其中”location”:{“type”:”float”},”,字段类型是float,而不是geo_point,因此会报图中的错误。
解决方法:Elasticsearch支持给索引预定义设置和mapping(前提是你用的 elasticsearch 版本支持这个API,不过估计应该都支持)。其实ES中已经有一个默认预定义的模板,我们只要使用预定的模板即可,那为什么还会报错呢?因为默认预定义的模板必须只有匹配 logstash-* 的索引才会应用这个模板,由于我们在logstash中使用的是[report-]YYYY.MM.DD索引方式,因此不会匹配到默认模板,我们只需要改一下索引方式即可:
[root@linux-node1 conf.d]# cat redis_to_ip.conf
input {
redis {
data_type => "list"
key => "nginx"
host => "192.168.6.243"
port => "6379"
db => "0"
password => "zls"
codec => "json"
}}
filter {
json {
source => "message"
remove_field => ["message"]
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLite2-City_20190730/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}
output {
elasticsearch {
hosts => ["192.168.6.241:9200"]
#index => "report-%{+YYYY.MM.dd}"
index => "logstash-%{type}-%{+YYYY.MM.dd}"
}}
将输出到ES的索引: index => "%{type}-%{+YYYY.MM.dd}"
改为: index => "logstash-%{type}-%{+YYYY.MM.dd}"
重启Logstash,登录Kibana刷新即可。
继续画图
也可以根据自己喜好,画成热力图
保存,可以放入Dashboard