#FreeBSD #Filebeats_logstash_elasticsearch_kibana

Filebeat, Logstash, Elasticsearch, Kibana, Nginx

Aug 6, 2016

2 min read

May 31, 2023 18:15 EEST

We will use Filebeat, Logstash, Elasticsearch and Kibana to visualize Nginx access logfiles.

Create the x509 Certificate

As I have all running on one server I use as the SSL common name localhost.

If you would like to deliver logfiles to another IP address use here the correct FQDN.

mkdir -p /usr/local/etc/pki/tls/certs
mkdir -p /usr/local/etc/pki/tls/private
cd /usr/local/etc/pki/tls
openssl req -subj '/CN=localhost/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/beat.key -out certs/beat-cacert.crt

The beat-cacert.crt will be copied to all computer you want to send logs from.

Install and configure Elasticsearch

pkg install elasticsearch2

We only change one line in the config file to make sure only localhost can connect to elasticsearch:

/usr/local/etc/elasticsearch/elasticsearch.yml

network.host: localhost

Enable it with:

sysrc elasticsearch_enable="YES"

Start it with:

service elasticsearch start

Install and configure Filebeat

pkg install filebeat

Use only spaces and no tabs in the configuration file!

/usr/local/etc/filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - /var/log/auth.log
        - /var/log/messages
      input_type: log
      document_type: syslog
    -
      document_type: web_access_nginx
      input_type: log
      paths:
        - /usr/home/http/poudriere/logs/access.log

output:
  logstash:
    hosts: ["localhost:5044"]
    bulk_max_size: 1024
    tls:
      certificate_authorities: ["/usr/local/etc/pki/tls/certs/beat-cacert.crt"]

shipper:

logging:
    rotateeverybytes: 10485760 # = 10MB

Verify the format of the file with:

filebeat -configtest

Enable Filebeat with:

sysrc filebeat_enable="YES"

And start it with:

service filebeat start

It should now directly start to deliver logfile information defined in section prospectors. You can test it with:

curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

If you see something like this everything is fine:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 20,
    "successful" : 20,
    "failed" : 0
  },
  "hits" : {
    "total" : 18157,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "filebeat-2016.08.03",
      "_type" : "syslog",
      "_id" : "AVZcJLZL5UZfyQchYySN",
...

Aug 06	Filebeat, Logstash, Elasticsearch, Kibana, Nginx	4 min read
Mar 10	Apache	5 min read
Feb 03	Letsencrypt	3 min read
Nov 15	Migrate system from i386 (32-bit mode) to amd64 (64-bit mode)	4 min read
Mar 12	Samba 4	1 min read