We will use Filebeat, Logstash, Elasticsearch and Kibana to visualize Nginx access logfiles.

As I have all running on one server I use as the SSL common name localhost.

mkdir -p /usr/local/etc/pki/tls/certs
mkdir -p /usr/local/etc/pki/tls/private
cd /usr/local/etc/pki/tls
openssl req -subj '/CN=localhost/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/beat.key -out certs/beat-cacert.crt

The beat-cacert.crt will be copied to all computer you want to send logs from.

pkg install elasticsearch2

We only change one line in the config file to make sure only localhost can connect to elasticsearch:

/usr/local/etc/elasticsearch/elasticsearch.yml network.host: localhost

Enable it with:

sysrc elasticsearch_enable="YES"

Start it with:

service elasticsearch start

pkg install filebeat

/usr/local/etc/filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - /var/log/auth.log
        - /var/log/messages
      input_type: log
      document_type: syslog
    -
      document_type: web_access_nginx
      input_type: log
      paths:
        - /usr/home/http/poudriere/logs/access.log

output:
  logstash:
    hosts: ["localhost:5044"]
    bulk_max_size: 1024
    tls:
      certificate_authorities: ["/usr/local/etc/pki/tls/certs/beat-cacert.crt"]

shipper:

logging:
    rotateeverybytes: 10485760 # = 10MB

Verify the format of the file with:

filebeat -configtest

Enable Filebeat with:

sysrc filebeat_enable="YES"

And start it with:

service filebeat start

It should now directly start to deliver logfile information defined in section prospectors. You can test it with:

curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

If you see something like this everything is fine:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 20,
    "successful" : 20,
    "failed" : 0
  },
  "hits" : {
    "total" : 18157,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "filebeat-2016.08.03",
      "_type" : "syslog",
      "_id" : "AVZcJLZL5UZfyQchYySN",
...