Filebeat 7.8 on pfSense 2.4.5

Welcome

Hey and welcome to this blog post by me, my name is Michael Pedersen I am 34 years old and I love open source and security stuff. In August of 2019 I started my journey into the danish InfoSec community and along the way I came across Security Distractions because they too love open source and security! Now one year later I have been invited to join the blog and I am very pleased to be able to publish this first post and hopefully have many more to come. So again – Welcome πŸ™‚

Introduction

On many of my projects I often need a firewall to segment networks, apply different rules, intercept traffic etc. For this I alway use pfSense, because it’s easy to setup, has a ton of functionality, is very well documented and of course free and opensource, which we like.

But unfortunately this is not a blog post about pfSense and how awesome and fun it is to play around with. Actually, this is more of a fusion of different sources that I have put together, to solve a problem I was facing – so let us get started.

For one of my projects I was going to intercept traffic from a client network using pfSense and the squid proxy service and have all traffic written to the squid access log. The second step was then to ship the logs into an Elasticsearch instance and the best way to do this is using Elastic Beats, in this case Filebeat. This is where the challenge awaited.

The challenge…

As you probably know, pfSense is running on FreeBSD and as of the moment, the package manager does not provide any easy install of Elastic components. I searched the internet for a while, not having much luck until I fell over this blog post written by a Swedish pentester called Christoffer Claesson. It seemed he had found an answer to this within an ongoing github issue, but felt that it was to good not to have on its own blog – I totally agree. The original idea was posted by another guy called jakommo which you can read here.

Now, jakommos idea was straight forward. Simply download the github repository and build the beat you wanted yourself. Hell, why did I not think of that! Christoffers and jakommo already has some of the steps covered, but I would like to follow up on them and add a few things.

There is a few prerequisites that you need to get up and running before you start.

Prerequisites…

  1. Get a fresh FreeBSD machine up and running, preferably the same versions as your pfSense box.
  2. Make sure you can reach your FreeBSD box via SSH. You might need to edit the /etc/ssh/sshd_config and allow root login (not best practice – I know)

Let’s go…

The following steps updates the pkg repository and installs the tools needed to be able to clone the filebeat repository and built it from source. This will furthermore install a bunch of dependencioes, which is why this is done on a separate BSD machine to avoid polluting the pfSense box.

root@freebsd:~ # pkg update -f 
root@freebsd:~ # pkg install git gmake go bash
root@freebsd:~ # git clone https://github.com/elastic/beats.git
root@freebsd:~ # cd beats/filebeat/
root@freebsd:~/beats/filebeat/ # gmake

When the process is done, you should now be able to do this:

Now step back one dir, and create an archive of the filebeat folder.

root@freebsd:~/beats/filebeat/ # cd ..
root@freebsd:~ # tar -czvf filebeat.tar.gz ./filebeat/

This concludes the work with our temporary FreeBSD box, now we have to copy the archive out to our local machine so we can get it to our pfSense box. You can do this any way you like, I prefer to use secure copy from my local machine:

Next up, we need to upload our filebeat archive to our pfSense box. This can be done the same way, or you can use the WebUI like this:

in pfSense WebUI -> Diagnostics -> Command Prompt
browse to the archive on your local machine and upload the archive

All right, we are almost there. Log into your pfSense box using the console or SSH. If you used the WebUI to upload the archive, you will find the file in the /tmp/ folder. Now you can move it to wherever you want on pfSense and extract the archive. For this tutorial I simply stayed inside the /tmp/ dir.

extracting the archive
test it is working

And there you have it. All you have to do now is configure the filebeat to fit your needs and type ./filebeat.

Caveats… Always with the caveats…

  • This has only been tried in a virtual environment but I can’t see why this should not work on hardware as well.
  • When you start filebeat it will run in the foreground of the shell you are using. I will try to do a post on how to get it running as a service instead.
  • I have tried this with Filebeat 7.8 and the master branch (8.0 as of this writing).

Outro… (you still listening?)

Lastly, I want to give a shout out to my friend David Thejl-Clayton and his custom squid log format that you can find in our github repo here. David has spent a great deal of time to define a custom log format optimized with all relevant fields critical for detection and analyzing of proxy alerts.

Cheers – Michael

Adding Windows DHCP logs to Elastic – part 1

Prerequisites

In order to add Windows DHCP server logs to Elastic, we assume that you have the infrastructure needed.

  • Windows DHCP Server 2012 R2 or higher
  • Elasticsearch cluster
  • Logstash

We are going to work with Elastic 6.x in this setup.  

Filebeat

Install filebeat on your DHCP server in a directory of your liking.

The DHCP logs are located in %systemroot%\system32\dhcp\dhcpsrvlog*.log. You will also find ipv6 logs, but we will focus on ipv4 logs.

Add the following code to your filebeat.yml. This is the prospector, that is going to watch for your DHCP logs. Notice that I am adding a field called Type under root with the value dhcp. This is a field , we will using in the Logstash configuration.

    -
      # Paths that should be crawled and fetched. Glob based paths.
      # To fetch all ".log" files from a specific level of subdirectories
      # /var/log/*/*.log can be used.
      # For each file found under this path, a harvester is started.
      # Make sure not file is defined twice as this can lead to unexpected behaviour.
      paths:
        #- /var/log/*.log
        - c:\windows\system32\dhcp\DhcpSrvLog-*.log
      input_type: log
      include_lines: ["^[0-9]"]
      document_type: dhcp
      close_removed : false
      clean_removed : false                 
      ignore_older: 47h
      clean_inactive: 48h     
      fields:
        type: dhcp
      fields_under_root: true

And add an output section also. There are multiple ways of shipping data from Filebeat. In this example we are shipping the logs to Logstash for parsing. DNS names are logstash01 and logstash02.

output:

  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["logstash01:5044" , "logstash:5044" ]

    # Number of workers per Logstash host.
    worker: 2

    # Set gzip compression level.
    #compression_level: 3

    # Optional load balance the events between the Logstash hosts
    loadbalance: true

    # Optional index name. The default index name depends on the each beat.
    # For Packetbeat, the default is set to packetbeat, for Topbeat
    # top topbeat and for Filebeat to filebeat.
    #index: filebeat

    # Optional TLS. By default is off.
    #tls:
      # List of root certificates for HTTPS server verifications
      #certificate_authorities: ["/etc/pki/root/ca.pem"]

      # Certificate for TLS client authentication
      #certificate: "/etc/pki/client/cert.pem"

      # Client Certificate Key
      #certificate_key: "/etc/pki/client/cert.key"

      # Controls whether the client verifies server certificates and host name.
      # If insecure is set to true, all server host names and certificates will be
      # accepted. In this mode TLS based connections are susceptible to
      # man-in-the-middle attacks. Use only for testing.
      #insecure: true

      # Configure cipher suites to be used for TLS connections
      #cipher_suites: []

      # Configure curve types for ECDHE based cipher suites
      #curve_types: []

After these steps, filebeat should be able to watch the DHCP server and ship them to Logstash.

Logstash

In order for logstash to process the data coming from your DHCP server , we create an input section and specify it as beats input.

input {
  beats {
   port => 5044
  }
}

Next we define the filter section, where we will parse the logs. Notice that we are using the dissect filter here. This is very convinient for this kind of stuff and you dont have to use grok for simple stuff like this.

filter
{
    if [type] == "dhcp"
    {
        dissect {
          mapping => {
          "message" => "%{ID},%{Date},%{Time},%{Description},%{IP_Address},%{Host_Name},%{MAC_Address},%{User_Name},%{TransactionID},%{QResult},%{Probationtime},%{CorrelationID},%{Dhcid},%{VendorClass_hex},%{VendorClass_ascii},%{UserClass_hex},%{UserClass_ascii},%{RelayAgentInformation},%{DnsRegError}"
                   }
           } 
         mutate
         {
            add_field => { "log_timestamp" => "%{Date}-%{Time}" }
         }
         date {
              match => [ "log_timestamp", "MM/dd/YY-HH:mm:ss" ]                
              timezone => "Europe/Copenhagen"
         }
         if "_dateparsefailure" not in [tags]
         {
             mutate
             {
                remove_field=> ['Date', 'Time', 'log_timestamp', 'message']
             }
        }
     }
}

And finally we define the output section, where we ship data from Logstash to Elastic. We are using a daily index in this example, but you could use a weekly or even monthly approach instead as there will not be a huge amount of data in this index.

output {
  if [type] == "dhcp"
  {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "dchp-%{+YYYY.MM.dd}" 
    }
  }
}

Elasticsearch

In order for Elasticsearch to correctly handle our DHCP data , we need to provide a index template. Notice that we are just using Keywords here.

{
  "dhcp": {
    "order": 10,
    "index_patterns": [
      "dhcp-*"
    ],
    "settings": {},
    "mappings": {
      "dhcp": {
        "dynamic_templates": [
          {
            "strings_as_keyword": {
              "mapping": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "match_mapping_type": "string"
            }
          }
        ],
        "properties": {}
      }
    },
    "aliases": {}
  }
}

That’s is pretty much it for you to get data going. We will leave  it for you to define a dashboard to display the data in a meaningful manner.

Stay tuned for next part of this series where we will be expanding the logstash filter by enriching the data with Mac Vendor lookup.