Kibana – Chromium

When you have installed your Elasticsearch cluster with Kibana on your favourite Linux distribution, you will probably at some time want to generate a report as a PDF.

Elastic decided to switch from PhantomJS to Chromium as headless browser in the latest releases. And Chromium has some dependencies, that Kibana install is not handling for you.

So if you goto Kibana/Managment/Reporting, it’s possible, you will see this error:

Chromium fails to start

This can be caused by a variety of reasons. But a typical cause is missing dependencies for Chromium. This is documented at this link.

The page has sections documenting the dependencies for the most popular distributions.

So we add these packages to our Kibana Ansible playbook:

Kibana Ansible playbook

After having run this updated playbook, we see, that we are now able to generate PDF reports as expected.

Sample PDF

Modular Elasticsearch Architecture

This topic is very interesting to us and we have spent a lot of time trying to get it right. We have gone through a number of architecture designs, while we were maturing with the Elasticsearch product. Needless to say, this topic is very complex and just because something works for us doesn’t mean that it will work for you. So with that in mind, let’s delve into this topic.

A lot of people hear stories about Elasticsearch and how easy and straightforward it is to get going. And it really is that easy. Most will have an Elasticsearch installation running after an hour or so. It’s just a matter of following the excellent documentation at Elastic’s website.

So once you installed the first node , you want to add a few other nodes, still simple operation.

Single node cluster

You end up with for example a simple 3 node cluster, where all nodes holds the same roles. All are master , data, kibana and logstash. And that is perfectly fine , if you are just trying out stuff.

3 node cluster

However. As you deploy more of these small clusters for various usecases , you tend to get frustrated by this approach as these small cluster can’t talk to each other. You can’t search all your data from one single location. And you need to maintain and monitor each cluster.

So next logical approach is to try and go bigger and follow recommendations from Elastic. So you add dedicated masters nodes , dedicated Kibana nodes, dedicated data nodes for hot and warm. This is cool, we can extend this cluster unlimited and put all our data in this huge cluster. And you probably can, but you probably dont want to.

Getting bigger

Somewhere down along the line, the masters will have trouble keeping up. Some usecase will end up influencing the other usecases running in the same cluster. Upgrading the nodes becomes a pain. If bad stuff happens to the cluster, everything is gone. And maybe you will feel a need to rethink the architecture.

So we did that..

What we came up with is a simple modular approach, where you will get these benefits:

  • Search all your data from one place
  • Centralized Logstash pipeline management
  • Centralized Cluster Monitoring
  • Centralized Machine Learning
  • Usecase/workload isolation
  • Simple maintenance
  • Unlimited scalability

Through the power of Cross Cluster Search, you can easily build this architecture. If you have not heard of CCS , I encourage you to read the docs. It is a faily new feature, that allows you to connect to a cluster and search remotely connected clusters. Completely transparent. Your users dont need to know or care about, which Elasticsearch cluster, their data is located in.

Only caveat is that it is a licensed feature, so you will need a subscription from Elastic in order to deploy this topology.

Modular Architecture

So the topology is basically a frontend cluster with these resources:

  • Master nodes
  • Kibana nodes
  • ML nodes
  • Small data nodes , no end user data

Then you build a number of usecase clusters or backend clusters.

  • Master nodes
  • Coordinator nodes – if required, depends on size
  • Hot Data
  • Warm Data

You attach these backend clusters to the frontend cluster by adding them as remote cluster. You configure the backend clusters to use the frontend for monitoring.

This architecture isolates the clusters from each other and thus the workload. You can easily do rolling upgrades without affecting the whole setup. You could even run with different versions of Elasticsearch in the clusters. However that is not something we would recommend.

You will need to install Kibana in the backend clusters also as the developer tools don’t work across clusters yet. This is purely for administrative purposes, no end user access. We are hoping that remote connect in developer tools will be available in a future release of Kibana.

Whenever you have a need for a new usecase, that doesnt play well with the existing backend clusters, you simply deploy a new backend cluster and plug it into the frontend. All your existing usecases will continue running completely unaware of this added workload. The data will be instantly searchable from the same Kibana locations, that your user are accustomed to.

In regards to the Logstash nodes. You install a group of them and configure them to connect to the frontend cluster for their pipeline management. In the output section of the actual pipelines , you target the specific backend cluster. Typically the coordinator nodes in the backend cluster.

Your end users connect to Kibana in the frontend cluster and searches transparently all the backend clusters, that they are allowed to. The roles that you assign them in the frontend will be carried with their search requests to the backend clusters.

Hopefully you will have the Kibana setup with Spaces, so it gives your users a more structured view of all the data availabke to them. And you as the admin , can monitor the state of your entire Elasticsearch setup in one place.

As we started out. Elasticsearch architecture is complex and there is no size fits all. We hope this article gave you some ideas on, how to build a highly scalable and modular Elasticsearch setup allowing you to take advantage of the power of Elasticsearch.

Adding Windows DHCP logs to Elastic – part 2

If you followed the previous blog, we have created a pipeline for sending Windows Server DHCP logs into Elastic. But that was just the basic logs. Suppose we want to enrich the logs with the Mac Address Vendor Information. So you will be able to monitor what devices appear on your network. Please note that Mac Addresses can be manipulated by 3 party.

A Mac Address consists of 2 parts . You have the Mac Vendor Prefix of 3 bytes followed by a unique identifier of another 3 bytes.

On a Windows computer you have multiple ways to display the Mac Address. These include:

  • ipconfig /all
  • getmac
ipconfig /all
Ipconfig /all

In this example , we have a Mac Address of A0-B3-CC-85-30-F4. Our Vendor prefix will be A0-B3-CC. But how to translate this prefix into Vendor information?

Fortunately we can download a list of known Vendors http://standards-oui.ieee.org/oui.txt

So we download that list and look for our example prefix, we find this prefix as registered by HP.

Vencor a0-b3-cc
Vendor a0-b3.cc

To be able to use the downloaded OUI list in logstash we need to convert the file to a YML file. The file will also contain some characters that logstash don’t appreciate, so we will try to strip those characters also. We will use a small powershell script to do this conversion. The script contains hardcoded locations, which you need to update.

# assume we have downloaded script to this location
$file = "c:\source\oui.txt"
# target location
$dest = "c:\source\oui.yml"

# regex to filter lines that contains the "(base 16)" wording
$regex = "(^.{6})\s+\(base 16\)\s+(.*)$"

if (Test-path $dest)
{
    remove-item $dest
}

$streamWriter = [System.IO.StreamWriter] "$dest"
foreach($line in Get-Content $file) 
{
    if($line -match $regex)
   {
       $company = $matches[2]
       # strip out strange characters
       $nice = $company -replace "[^0-9a-zA-Z ]+" , ""
       # write to dest YML file
       $streamWriter.WriteLine("`"$($matches[1])`": $($nice)");
    }
}
# we are done
$streamWriter.close();

We are now able to expand our logstash filter, to use our new Mac Address vendor lookup information. We will do this with the help of the translate filter. This allows us to lookup a value in the YML file.

if [Description] == "Assign" or [Description] == "Renew"
{
   grok 
   {
    # split the mac address into 2 elements
    match => [ "MAC_Address", "(?<ouiprefix>.{6})(?<ouisuffix>.*)" ]
   }
   # lookup the ouiprefix in the YML file and add new oui field
   translate 
   {
     dictionary_path => "c:\elastic\logstash\config\oui.yml"
     field      => "[ouiprefix]"
     destination => "[oui]"
     fallback => "N/A"
   }
}

That’s all you need to do to enrich the DHCP logs with Mac Address Vendor. As usual we leave it to you to make a visualization of this info and add that to your DHCP dashboard.

Next part of this series will adress trying to find unusual hostnames in the DHCP logs. We will try to develop a pattern on the typical hostname and use Elastic Machine Learning to find uncommon patterns.

Adding Windows DHCP logs to Elastic – part 1

Prerequisites

In order to add Windows DHCP server logs to Elastic, we assume that you have the infrastructure needed.

  • Windows DHCP Server 2012 R2 or higher
  • Elasticsearch cluster
  • Logstash

We are going to work with Elastic 6.x in this setup.  

Filebeat

Install filebeat on your DHCP server in a directory of your liking.

The DHCP logs are located in %systemroot%\system32\dhcp\dhcpsrvlog*.log. You will also find ipv6 logs, but we will focus on ipv4 logs.

Add the following code to your filebeat.yml. This is the prospector, that is going to watch for your DHCP logs. Notice that I am adding a field called Type under root with the value dhcp. This is a field , we will using in the Logstash configuration.

    -
      # Paths that should be crawled and fetched. Glob based paths.
      # To fetch all ".log" files from a specific level of subdirectories
      # /var/log/*/*.log can be used.
      # For each file found under this path, a harvester is started.
      # Make sure not file is defined twice as this can lead to unexpected behaviour.
      paths:
        #- /var/log/*.log
        - c:\windows\system32\dhcp\DhcpSrvLog-*.log
      input_type: log
      include_lines: ["^[0-9]"]
      document_type: dhcp
      close_removed : false
      clean_removed : false                 
      ignore_older: 47h
      clean_inactive: 48h     
      fields:
        type: dhcp
      fields_under_root: true

And add an output section also. There are multiple ways of shipping data from Filebeat. In this example we are shipping the logs to Logstash for parsing. DNS names are logstash01 and logstash02.

output:

  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["logstash01:5044" , "logstash:5044" ]

    # Number of workers per Logstash host.
    worker: 2

    # Set gzip compression level.
    #compression_level: 3

    # Optional load balance the events between the Logstash hosts
    loadbalance: true

    # Optional index name. The default index name depends on the each beat.
    # For Packetbeat, the default is set to packetbeat, for Topbeat
    # top topbeat and for Filebeat to filebeat.
    #index: filebeat

    # Optional TLS. By default is off.
    #tls:
      # List of root certificates for HTTPS server verifications
      #certificate_authorities: ["/etc/pki/root/ca.pem"]

      # Certificate for TLS client authentication
      #certificate: "/etc/pki/client/cert.pem"

      # Client Certificate Key
      #certificate_key: "/etc/pki/client/cert.key"

      # Controls whether the client verifies server certificates and host name.
      # If insecure is set to true, all server host names and certificates will be
      # accepted. In this mode TLS based connections are susceptible to
      # man-in-the-middle attacks. Use only for testing.
      #insecure: true

      # Configure cipher suites to be used for TLS connections
      #cipher_suites: []

      # Configure curve types for ECDHE based cipher suites
      #curve_types: []

After these steps, filebeat should be able to watch the DHCP server and ship them to Logstash.

Logstash

In order for logstash to process the data coming from your DHCP server , we create an input section and specify it as beats input.

input {
  beats {
   port => 5044
  }
}

Next we define the filter section, where we will parse the logs. Notice that we are using the dissect filter here. This is very convinient for this kind of stuff and you dont have to use grok for simple stuff like this.

filter
{
    if [type] == "dhcp"
    {
        dissect {
          mapping => {
          "message" => "%{ID},%{Date},%{Time},%{Description},%{IP_Address},%{Host_Name},%{MAC_Address},%{User_Name},%{TransactionID},%{QResult},%{Probationtime},%{CorrelationID},%{Dhcid},%{VendorClass_hex},%{VendorClass_ascii},%{UserClass_hex},%{UserClass_ascii},%{RelayAgentInformation},%{DnsRegError}"
                   }
           } 
         mutate
         {
            add_field => { "log_timestamp" => "%{Date}-%{Time}" }
         }
         date {
              match => [ "log_timestamp", "MM/dd/YY-HH:mm:ss" ]                
              timezone => "Europe/Copenhagen"
         }
         if "_dateparsefailure" not in [tags]
         {
             mutate
             {
                remove_field=> ['Date', 'Time', 'log_timestamp', 'message']
             }
        }
     }
}

And finally we define the output section, where we ship data from Logstash to Elastic. We are using a daily index in this example, but you could use a weekly or even monthly approach instead as there will not be a huge amount of data in this index.

output {
  if [type] == "dhcp"
  {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "dchp-%{+YYYY.MM.dd}" 
    }
  }
}

Elasticsearch

In order for Elasticsearch to correctly handle our DHCP data , we need to provide a index template. Notice that we are just using Keywords here.

{
  "dhcp": {
    "order": 10,
    "index_patterns": [
      "dhcp-*"
    ],
    "settings": {},
    "mappings": {
      "dhcp": {
        "dynamic_templates": [
          {
            "strings_as_keyword": {
              "mapping": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "match_mapping_type": "string"
            }
          }
        ],
        "properties": {}
      }
    },
    "aliases": {}
  }
}

That’s is pretty much it for you to get data going. We will leave  it for you to define a dashboard to display the data in a meaningful manner.

Stay tuned for next part of this series where we will be expanding the logstash filter by enriching the data with Mac Vendor lookup.