Logstash basic configuration (Docker)

Configuration files

logstash.conf : have 3 section
1) input - Standard input, log file and Filebeat etc.
2) filter - filter stream contents
3) output  - to elasticsearch

input { stdin { } }
output { stdout { } }
-- or --
input { stdin { } }
output {
  elasticsearch {
    hosts => ["esearch:9200"]
  }
}
--or--
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
  elasticsearch {
    hosts => ["esearch:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Run process

$ docker run --restart=always --expose=5044 --name logstash-es -d -p 10.0.2.41:5044:5044 --net="my-net" --add-host="esearch:172.28.5.1" --mac-address="c2:00:c6:bb:c8:e2" --ip="172.28.5.3" -v /home/ubuntu/docker/logstash/config:/config logstash:2.3.3-1 logstash -f /config/logstash.conf

Generate SSL Certificates
This case we generate key on docker host rest logstash container

** Important : must change openssl.conf as

...
[ v3_ca ]
subjectAltName = IP: 10.0.2.41
...

ubuntu@node1:~/docker/logstash/config/pki/tls$ sudo vi /etc/ssl/openssl.cnf 
ubuntu@node1:~/docker/logstash/config/pki/tls$ 
ubuntu@node1:~/docker/logstash/config/pki/tls$ mkdir certs private
ubuntu@node1:~/docker/logstash/config/pki/tls$ sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
Generating a 2048 bit RSA private key
.................................+++
.....................+++
writing new private key to 'private/logstash-forwarder.key'
-----

Remark: *.key are private key and *.crt are public key can be distributed to Beat agent (Client)

Change configuration file

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/config/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/config/pki/tls/private/logstash-forwarder.key"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["esearch:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Restart container

$ docker stop logstash-es
$ docker start logstash-es





Filter customization
alternatively create by Grok Constructor

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "\A%{SYSLOGTIMESTAMP:syslog_timestamp}%{SPACE}%{SYSLOGHOST:syslog_hostname}%{SPACE}%{SYSLOGPROG}: %{GREEDYDATA:syslog_message}" }
      add_field => {
        "syslog_program" => "%{program}"
        "syslog_pid" => "%{pid}"
        "received_at" => "%{@timestamp}"
        "received_from" => "%{host}"
      }
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}