Elastic Stack

  • Elasticsearch, Logstash, Kibana, Beatsなどのあれこれ.

Elasticsearch

dump

Logstash

Plugins

Installation using logstash-plugin command

  • Example: logstash-output-slack
root@elk:/usr/share/logstash/bin# ./logstash-plugin install logstash-output-slack
Validating logstash-output-slack
Installing logstash-output-slack

flow

  • http://enog.jp/wp-content/uploads/2015/09/enog43_elk_0904.pdf
    • ASN, gio-ipはいれたい.
  • https://www.janog.gr.jp/meeting/janog39/application/files/7014/8481/0318/janog39-traffic-nishizuka-03.pdf

slack通知

if [dstip] and [dstip] !~ "(^127.0.0.1)|(^10.)|(^172.1[6-9].)|(^172.2[0-9].)|(^172.3[0-1].)|(^192.168.)|(^169.254.)" {

Tools

Common

install battle on ubuntu18.04

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.0.deb
sha1sum elasticsearch-5.4.0.deb
sudo dpkg -i elasticsearch-5.4.0.deb
sudo vim /etc/elasticsearch/jvm.options
java -version
sudo apt install default-jre
sudo apt list --installed | grep jre
curl -XGET 'localhost:9200/?pretty'
sudo vim /etc/elasticsearch/elasticsearch.yml
sudo service elasticsearch start

elk memo misc

ELK Install Battle

sudo apt install openjdk-8-jdk apt-transport-https
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update && sudo apt-get install elasticsearch
sudo vi /etc/elasticsearch/elasticsearch.yml
sudo service elasticsearch status
sudo service elasticsearch start
sudo service elasticsearch status
sudo service elasticsearch enable
curl "http://localhost:9200/"
sudo apt install kibana
sudo vi /etc/kibana/kibana.yml
sudo service kibana status
sudo service kibana start
sudo service kibana enable
curl http://localhost:5601/ -L
sudo apt install logstash
sudo vim /etc/systemd/system/logstash.service
sudo systemctl status logstash
sudo systemctl start logstash
sudo vim /etc/logstash/conf.d/syslog.conf
sudo vi /etc/elasticsearch/elasticsearch.yml
sudo systemctl start elasticsearch.service
sudo systemctl daemon-reload
sudo systemctl enable kibana
sudo systemctl start kibana
sudo systemctl start logstash
curl http://localhost:9200/applogs/_search?q=*
curl http://localhost:9200//_search?q=*
curl http://localhost:9200/syslog/_search?q=*
curl localhost:9200

kea-dhcpのallocログからouiを取得してベンダをフィールドに追加する

  • 前提
"message" => "INFO  [kea-dhcp4.leases] DHCP4_LEASE_ALLOC [hwtype=1 01:23:45:67:89:ab], cid=[01:01:23:45:67:89:ab], tid=0x11111111: lease 1.2.3.4 has been allocated"
  • keaのリースログとしてこれが流れてくる.
  • sample.conf
input {
  elasticsearch {
    hosts => "localhost"
    query => '{ <Queries> }'
  }
}

filter {
  grok {
    "match" => { "message" => '.*\[hwtype.* %{COMMONMAC:macaddr}]' } # COMMONMACはdefaultのgrok-patternsとして存在する
  }

  mutate {
    add_field => ["macaddr_prefix", "%{macaddr}"]
  }

  mutate {
    gsub=> ["macaddr_prefix", "^(.{8}).*", "\1" ]
  }
}

  translate {
    dictionary_path => "/etc/logstash/oui.yml" # `"xx:xx:xx": "vender_name"` がリストされたymlへのpath
    field      => "macaddr_prefix"
    destination => "oui"
    fallback => "N/A"
  }
}

output{
  stdout { codec => rubydebug }
}
  • elasticsearchの既存indexからリースログを引くクエリを指定してinputとしている.
  • kibana等で可視化したい場合はoutputを別indexとしてelasticsearchに投げることでkibanaでも扱えるようになる.
    • この手法をとるとouiフィールドがあるindexと無いindexでほぼ同様の情報が2つdupで保存されてしまうことに注意する.