Benutzer-Werkzeuge

Webseiten-Werkzeuge


freebsd:filebeats_logstash_elasticsearch_kibana

Filebeat, Logstash, Elasticsearch, Kibana, Nginx

We will use Filebeat, Logstash, Elasticsearch and Kibana to visualize Nginx access logfiles.

Create the x509 Certificate

As I have all running on one server I use as the SSL common name localhost.

If you would like to deliver logfiles to another IP address use here the correct FQDN.
mkdir -p /usr/local/etc/pki/tls/certs
mkdir -p /usr/local/etc/pki/tls/private
cd /usr/local/etc/pki/tls
openssl req -subj '/CN=localhost/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/beat.key -out certs/beat-cacert.crt

The beat-cacert.crt will be copied to all computer you want to send logs from.

Install and configure Elasticsearch

pkg install elasticsearch2

We only change one line in the config file to make sure only localhost can connect to elasticsearch:

/usr/local/etc/elasticsearch/elasticsearch.yml network.host: localhost
 

Enable it with:

sysrc elasticsearch_enable="YES"

Start it with:

service elasticsearch start

Install and configure Filebeat

pkg install filebeat
Use only spaces and no tabs in the configuration file!
/usr/local/etc/filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - /var/log/auth.log
        - /var/log/messages
      input_type: log
      document_type: syslog
    -
      document_type: web_access_nginx
      input_type: log
      paths:
        - /usr/home/http/poudriere/logs/access.log

output:
  logstash:
    hosts: ["localhost:5044"]
    bulk_max_size: 1024
    tls:
      certificate_authorities: ["/usr/local/etc/pki/tls/certs/beat-cacert.crt"]

shipper:

logging:
    rotateeverybytes: 10485760 # = 10MB

Verify the format of the file with:

filebeat -configtest

Enable Filebeat with:

sysrc filebeat_enable="YES"

And start it with:

service filebeat start

It should now directly start to deliver logfile information defined in section prospectors. You can test it with:

curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

If you see something like this everything is fine:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 20,
    "successful" : 20,
    "failed" : 0
  },
  "hits" : {
    "total" : 18157,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "filebeat-2016.08.03",
      "_type" : "syslog",
      "_id" : "AVZcJLZL5UZfyQchYySN",
...
freebsd/filebeats_logstash_elasticsearch_kibana.txt · Zuletzt geändert: 2016/08/06 13:34 (Externe Bearbeitung)