TF-CSIRT – Whats it all about?

I have recently taken a break from blogging to focus on other things, before jumping back into my Incident Response 101 blog series. I want to write a little bit about TF-CSIRT and the reasons for joining a community like this. It is a process I am slowly becoming familiar with and it definitely deserves more words written about it…

First off…. What is TF-CSIRT?

Task Force Computer Security Incident Response Teams or TF-CSIRT for short, was established by the European CERT community back in the year 2000. The idea was to create a community of incident response groups/teams, which can work together for a common goal. That goal being, spreading information/knowledge sharing, assisting each other in incidents, and any other way they can leverage such a strong community to help in the incident response world.

In order to provide access to the community, a service was created called “The Trusted Introducer Service”. This service is used to provide a directory of incident response teams which are members of TF-CSIRT. The Trusted Introducer Service acts as a clearing house to ensure that members meet the correct requirements when joining. and then offering further processes for becoming accredited or certified TF-CSIRT members.

So what are the benefits?

The main backbone of the TF-CSIRT community is the member database, where emergency contact details for each incident response team are displayed. This information can prove vital in an incident response situation. To maintain this vital community spirit TF-CSIRT hosts regular conferences and meetups for its members, these are great for getting to know other teams and sharing knowledge.

Another huge benefit of TF-CSIRT actually lies within the certification process. This process provides strict requirements based on the SIM3 audit model and essentially means that when you hit the magic certification level, you are one of the best prepared incident response teams in Europe (at least on paper). This is a standard that a lot of teams aspire towards, but unfortunately don’t make it, due to time commitments usually.

The TF-CSIRT community also works very closely with FIRST (Forum of incident response and security teams). This partnership helps deliver a yearly joint conference.

There are many other benefits from becoming a member at TF-CSIRT, and I would highly recommend it!

So how do I join?

Joining TF-CSIRT is broken up into 3 different “memberships” or processes.

Listed Member

The first processs is to become a listed member. This means you become part of the community and you will get your team listed in the TF-CSIRT database. This also means you can begin attending the European conferences and meetups that are offered.

To become a listed member, you need to fulfill some requirements:-

  1. You need to be sponsored by at least 2 other already accredited or certified teams. A good idea here, would be to look at the Trusted Introducer directory and see if you know teams that have already gone through this process. The TF-CSIRT community is becoming larger and larger within Europe, so the chances are you already know the relevant teams to get the process moving.
  2. Get PGP/GPG keys for your team to communicate with TF-CSIRT. This one is a tiny bit of hassle as there is a large debate out there about using PGP, it can be quite difficult to get PGP supported within certain organizations and ad-hoc processes may end up being needed to facilitate this requirement.

Once you have these two main requirements met, you simply fill out a form and email it to the Trusted Introducer email address and VOILA… Well not quite VOILA, there is still an internal process which is undertaken within TF-CSIRT where various members are voting about your membership. But after a period, you will find yourself a listed member!

Accredited Member

A lot of teams who aim for the certification membership, will first need to become accredited members. By becoming accredited you recieve access to the members only part of the Trusted Introducer service where you have access quite a lot of nice information about other teams within the directory which is not publically available. Many teams reach this stage aiming for certification, but for multiple reasons do not progress to that step. You should look at the accreditation step as “we are who we say we are”, an incident response team who wants more than simply being listed, but wants to show the community they mean business.

To become accredited your team must:-

  1. Already be a listed member
  2. Use RFC2350 (I will blog about this soon)
  3. Fill out a large amount of information about your team and their capabilities and service offerings

Once these requirements are met, this information is supplied to the Trusted Introducer team. This time it is not quite VOILA at all. There is a long process where the information you have provided is vetted and assessed. This assessment takes around 3 months to complete and can result in further questions being asked by the Trusted Introducer team. After it is completed and you are accepted, then you gain a shiny new status of “Accredited” within the directory!

Certified Member

Saving the best type of membership for last, a certified member is a team who has met the gold standard for incident response teams. They have adhered to the strict SIM3 model and achieved a maturity rating within this model that is set by the Trusted Introducer team, and essentially means “your team is one of the best in Europe at incident response” (on paper!).

The requirements to become certified:-

  1. Must already be an accredited member
  2. Have a positive SIM3 assessment based on current Trusted Introducer thresholds

The idea with number 2. is that the team will spend time assessing their current maturity within incident response. To do this they use the SIM3 model, something which I will be blogging about very soon! This model is used to ensure that a team has all necessary processes documented and in place, plus that there is a measurable maturity within these processes.

If the team discovers they are not quite ready after completing a SIM3 assessment, they can then spend some time improving processes and documentation to a higher standard. Another low hanging fruit is ensuring that the processes you define are signed off and audited by someone independant from your incident response team. Once you are confident you have met the correct maturity level within your documentation, you can then apply to be certified.

A SIM3 auditor will then be appointed to you, this auditor will perform an onsite workshop at your location and audit all of your documented processes. Performing interviews of certain team members, and really digging deep to ensure that processes are not just something written on paper, but are understood too.

Once this audit is passed, your status will then be changed within the directory to “Certified” and you can then go and show off to your friends! *cough* I mean constituents…

I may make the certification process sound like a long drawn out process, but in fact how else could you achieve such an important gold standard, without being audited externally and being put before a committee who decides if you are mature enough to be certified, any other process like this would also take time. However the benefits that come after being certified are huge, your constituents and management can have safer knowledge that they are being served by a certified team.

Final words…

I hope that you learned something from this blog post, I have become familiar with the whole Trusted Introducer/TF-CSIRT grouping over the last 2 years and I think it is incredibly exciting to be a part of this community. The certification process is also an incredible learning experience and will ensure that you really have everything in order to run your incident response team!

The Trusted Introducer website has far more details and interesting information about the processes, and can be found here:- https://www.trusted-introducer.org/index.html

My next blog post in this area will talk about the SIM3 model and how awesome it is for measuring the maturity of your incident response team…

Squid Proxy Log Format

Squid - ASUSTOR NAS

In my previous post, I mentioned a custom Squid Proxy log format created by my friend and fellow blogger at Security Distractions, David Thejl-Clayton. I know that he spent a lot of time to define this and getting it parsed via Logstash into Elasticsearch to make data searchable. I have been using this very same approach in some of my labs, and therefore I thought I would give him some credit and also show how I have been using this.

Well, If we have configured our clients to use a proxy on the network, it can be an unbelievably valuable data source for security incidents – if we have enabled logging that is. Depending on the size of the network, we will also quickly be generating a vast amount of data, so being able to define exactly what we want written to log, is worth having a look at. This is exactly what the squid log format will let us do.

Squid Proxy is an open source content filter and caching engine, which can be found integrated into more well known proxying solutions on the market today. Squid Proxy comes bundled within pfSense and is something that features heavily in the lab I am building, which I will be writing more and more about over the next few months!

From a security perspective, there are some data that we would really like to include in our proxy logs. Things related to source and destination, but also things like request and responses, and any other details to help us answer if something malicious happened.

So let’s jump right into it, and get our hands a little dirty – Here is a log format that I have been using in some of my labs:

%ts.%03tu %6tr %>a %>p %03>Hs %<st %rm %ru %rp %rv %<a %<p %<A %mt %ssl::>sni "%{User-Agent}>h"

First, a few basics

Every new field begins with a percentage (%) symbol and the order they appear will be how they are written in the log. The numbers represent min/max field width, that can be used to control how much data you want to include e.g. in timestamps or to create space between field entries. I have not been looking too much into this, and in fact just used a few default settings. The bigger and less than (> <) symbols tells us something about the direction of the field value which we will get back to shortly. To get all this right, you will have to look at the documentation which I have provided a link to at the end the post. Lastly, the letters represent the actual field value that we want to log – so let’s start to have a look at them.

The good (and nerdy) stuff..

I divided the format into some smaller chunks and will walk through them step by step.

%ts.%03tu %6tr
1599026873.430     26

The first part here gives us a Unix timestamp of when an event occurred, followed by a dot and a maximum of three milliseconds precision. It is of course crucial to have timestamps in our log, as we need to know when something happened, so we can add things to a timeline and compare different entries over time.

The last field represents the response time in milliseconds. A bit of space is given between the timestamp to ease the view. One could perhaps argue if this information is needed or should be placed elsewhere, but I decided to leave it here for now.  


%>a %>p
192.168.100[.]100 56106

Next up is an IP and port number. Note the bigger than (>) sign here tells us a direction which in this case is the source IP and port. We are naturally interested in where a request came from if we suspect something malicious, so this is definitely two pieces of information that I expect being present.


%03>Hs %<st %rm
200 8860 GET

Here, the first field represents the three digit HTTP status code which was sent to the client/source. This will help our indication if something bad actually happened or not. Was the request successful? Perhaps we believe it was blocked? Any redirection or errors on the client or server side? This is another piece of information that I believe we should definitely include.

The following field is the total size of the reply, to see how much data the client received. At the end we have the request method (GET/POST etc.) to help us see what kind of requests was made, and over time help us learn the nature of the conversation.


%ru %rp %rv
https://www.securitydistractions.com/2020... /2020... 1.1

I also grouped these three fields together as they are somewhat related. The first field gives us the full URL requested by the client/source. Naturally, we are very interested in knowing exactly what was accessed or requested. Perhaps we are pivoting to or from this URL to see if other clients accessed it, which again can help us indicate if something actually happened.

The second field is simply the URL path without the host name. This is merely a convenient way to have this information sorted out separately into a field of its own, and will come in handy later in the process. At the end the protocol version is simply included as well.


%<a %<p
46.30.215[.]134 443

This we have seen this before. Pay attention to the less than (<) symbol which now tells us, that it is the destination IP and port. This gives us the ability to investigate and pivot to other sources of information for this IP. Also, the port information might give us a hint that this is something worth investigating, like traffic to non-standard ports etc.


%<A %mt %ssl::>sni
www.securitydistractions.com text/html www.securitydistractions.com

Almost done. These three covers server FQDN, the MIME content type and SNI from the client. Basicly we would want to know the FQDN to be able to act on this information and also have it sorted away from the URL which we recored earlier. The SNI is nice to know, if the client requested hostname is actually the same as the one accessed . The MIME type is an indication to the type of data that was accessed, and can help us validate our findings.


"%{User-Agent}>h"
Mozilla/5.0 (Windows NT 10.0; Win64; x64)....

The last part here, as you might have guessed, is the observed User-Agent. The UA can help indicate what kind of application was used to access a resource. From a security perspective, looking at the UA alone an interesting observation is aggregation of less frequently used. If we know our infrastructure well, we should perhaps be able to rule out some of them, and then investigate the ones we are less certain of. Also, I think it makes sense to log the UA here, because were else would you have it?

The finale

Phew, that was it. Now, to actually use our custom squid format we need to put it into the squid.conf file. If you are using the squid proxy on pfSense, like me, this is very easy and you can do it using the WebUI. Go to ServicesSquid Proxy ServerShow Advanced Settings and then copy paste the format in like shown below. The first line defines the format, and the bottom line calls the new format to be used.

Now our logs will be written using our custom format, and we can start to work with them.

I hope this has been worth a while and gave some ideas on how you are able to customize your squid logs and why this is a great way to help benefit any needs that you might have. This is merely an example, and we could customize this much further and have more or less information included. I highly recommend having a look at the squid log format documentation which you can find here.

If you think this was interesting, stay tuned for a follow up. In one of my next blog posts I will be using the output created here and parse it using Logstash to start using the logs for some good. Until then, read my previous post about running filebeat on pfSense, to help ship logs off the pfSense box and on to Logstash.

Cheers – Michael

Filebeat 7.8 on pfSense 2.4.5

Welcome

Hey and welcome to this blog post by me, my name is Michael Pedersen I am 34 years old and I love open source and security stuff. In August of 2019 I started my journey into the danish InfoSec community and along the way I came across Security Distractions because they too love open source and security! Now one year later I have been invited to join the blog and I am very pleased to be able to publish this first post and hopefully have many more to come. So again – Welcome πŸ™‚

Introduction

On many of my projects I often need a firewall to segment networks, apply different rules, intercept traffic etc. For this I alway use pfSense, because it’s easy to setup, has a ton of functionality, is very well documented and of course free and opensource, which we like.

But unfortunately this is not a blog post about pfSense and how awesome and fun it is to play around with. Actually, this is more of a fusion of different sources that I have put together, to solve a problem I was facing – so let us get started.

For one of my projects I was going to intercept traffic from a client network using pfSense and the squid proxy service and have all traffic written to the squid access log. The second step was then to ship the logs into an Elasticsearch instance and the best way to do this is using Elastic Beats, in this case Filebeat. This is where the challenge awaited.

The challenge…

As you probably know, pfSense is running on FreeBSD and as of the moment, the package manager does not provide any easy install of Elastic components. I searched the internet for a while, not having much luck until I fell over this blog post written by a Swedish pentester called Christoffer Claesson. It seemed he had found an answer to this within an ongoing github issue, but felt that it was to good not to have on its own blog – I totally agree. The original idea was posted by another guy called jakommo which you can read here.

Now, jakommos idea was straight forward. Simply download the github repository and build the beat you wanted yourself. Hell, why did I not think of that! Christoffers and jakommo already has some of the steps covered, but I would like to follow up on them and add a few things.

There is a few prerequisites that you need to get up and running before you start.

Prerequisites…

  1. Get a fresh FreeBSD machine up and running, preferably the same versions as your pfSense box.
  2. Make sure you can reach your FreeBSD box via SSH. You might need to edit the /etc/ssh/sshd_config and allow root login (not best practice – I know)

Let’s go…

The following steps updates the pkg repository and installs the tools needed to be able to clone the filebeat repository and built it from source. This will furthermore install a bunch of dependencioes, which is why this is done on a separate BSD machine to avoid polluting the pfSense box.

root@freebsd:~ # pkg update -f 
root@freebsd:~ # pkg install git gmake go bash
root@freebsd:~ # git clone https://github.com/elastic/beats.git
root@freebsd:~ # cd beats/filebeat/
root@freebsd:~/beats/filebeat/ # gmake

When the process is done, you should now be able to do this:

Now step back one dir, and create an archive of the filebeat folder.

root@freebsd:~/beats/filebeat/ # cd ..
root@freebsd:~ # tar -czvf filebeat.tar.gz ./filebeat/

This concludes the work with our temporary FreeBSD box, now we have to copy the archive out to our local machine so we can get it to our pfSense box. You can do this any way you like, I prefer to use secure copy from my local machine:

Next up, we need to upload our filebeat archive to our pfSense box. This can be done the same way, or you can use the WebUI like this:

in pfSense WebUI -> Diagnostics -> Command Prompt
browse to the archive on your local machine and upload the archive

All right, we are almost there. Log into your pfSense box using the console or SSH. If you used the WebUI to upload the archive, you will find the file in the /tmp/ folder. Now you can move it to wherever you want on pfSense and extract the archive. For this tutorial I simply stayed inside the /tmp/ dir.

extracting the archive
test it is working

And there you have it. All you have to do now is configure the filebeat to fit your needs and type ./filebeat.

Caveats… Always with the caveats…

  • This has only been tried in a virtual environment but I can’t see why this should not work on hardware as well.
  • When you start filebeat it will run in the foreground of the shell you are using. I will try to do a post on how to get it running as a service instead.
  • I have tried this with Filebeat 7.8 and the master branch (8.0 as of this writing).

Outro… (you still listening?)

Lastly, I want to give a shout out to my friend David Thejl-Clayton and his custom squid log format that you can find in our github repo here. David has spent a great deal of time to define a custom log format optimized with all relevant fields critical for detection and analyzing of proxy alerts.

Cheers – Michael

Incident Response 101 – The Why?

In the previous post we discussed the background for my knowledge within incident response. Now we will jump into the exciting stuff and talk about “The Why?”

I guess a pretty good place to start in defining the incident response process, is understanding why do we need incident response at all?

Incident response wouldn’t exist without something to actually trigger the process. To trigger the process you need an incident, and what will generate that incident?

Threats

Incidents are generated from a threat, whether this threat is a nation state attacker, a script kiddie, a pandemic, or even some sort of natural disaster. So then what is a threat, and how do we define it?

I like to start out this explanation by showing the following diagram:-

Diagram by IncibeCERT

Intent + Capability + Opportunity = Threat

Each one of these conditions needs to be met to fulfill the criteria to create a threat. To make it more understandable I use an example of whether there is a threat at home from my child trying to steal Nutella from the cupboard.

Intent

Intent is pure and simple, does my child want the Nutella? Do they have the desire and drive to get it? Without intent, I could leave Nutella all over the house and not be worried about anything happening to it.

Capability

Does my child have the capability to get the Nutella? I may have left the cupboard door open, and my child may desperately want the Nutella. But they haven’t learned to open a jar yet. So the threat is not there…

Opportunity

Did I leave the Nutella jar open on the kitchen top? So now my child has the perfect chance to get hold of it. The opportunity has been given to them, now they can combine it with their intent and capability to create the threat!

Well what can we do about this?

You may look at these three points, and think there is alot to be done to protect against each part of the “threat process”. But there isn’t… You cannot take actions to reduce the capabilities of your attackers yourself.

You also cannot influence an attackers intent against you. In some niche cases, you could argue that by “doing good things” you might reduce the intent. But this is a relatively difficult issue to measure.

So this leaves only “opportunity” where you can have some sort of impact. I say “some sort of” because an attacker will always get an opportunity. An opportunity can be something as simple as a misconfigured firewall, a vulnerability in a public facing server and many more.

But you can do your best to restrict the number of opportuntities presented to an attacker. A good example of this, is vulnerability management, when an exploit or vulnerability is released and this effects you, taking actions to patch or mitigate it can help reduce the attackers opportunity to become a threat.

But what about incident response?

You may be thinking, well wait a minute, where does incident response fit into this? Incident reponse assumes that the attacker had the opportunity to become a threat and then carried out actions against you which have resulted in an incident needing to be handled. Incident response is purely a reaction process and is driven by threats.

In some cases the lessons learned from the root cause analysis within the incident response process can also assist with reducing the attacker opportunities. An example of this… Imagine having a perimeter firewall hole, which is too wide and allows external access to a number of test servers which are not patched. The subsequent incident from an attacker compromising these servers, can lead to a report which identifies the broad firewall rule and gives advice on how to fix it. Thus reducing the next attackers opportunity to become a threat!

Closing remarks…

In the next post we will look at how we can have an understanding of the threat landscape, and how to figure out which threats might be relevant to us…

Incident Response 101 – The Background

In the previous post, I gave an introduction to my planned set of blog posts around incident response.

But…

The first question is, how have I made it to this stage in my understanding of the incident response process. Which materials, courses, books etc have lead me to develop my current knowledge level in this field. I will try to give a short description of each resource and why it is important…

All authors start with some background about them, so the audience trusts them a little more when they begin reading, “oh this guy has read alot, and is certified in xx and xx, they must know what they are talking about”.

This is a list of resources, that I turn to at least once a week in my work within incident response.

Materials:-

FIRST CSIRT Services Framework 2.0

https://www.first.org/standards/frameworks/csirts/FIRST_CSIRT_Services_Framework_v2.1.0.pdf

It took me quite some time to find this document, and it was quite a way into my journey of discovery within building a Cyber Defence Center before I found it. But once I did, it answered so many of the outstanding questions I had. This document lays it out flat, what you need to do to deliver a large selection of services within the CSIRT world. It opened up a door to a large community for me too, as I found the authors to be very interesting and the FIRST group a very welcome aid in my service architecture. I treat this book like the bible for the services I needed to build.

Just like any religious text, there is always room for intepretation and this resource is very good, but it does not answer every single question. In some areas it raises more questions, which require deeper research and more technically focused answers. But this we will touch in on later in the blog posts on this subject.

SIM3 – Security Incident Management Maturity Model

http://opencsirt.org/wp-content/uploads/2019/12/SIM3-mkXVIIIc.pdf

I started learning about the SIM3 Model whilst beginning research into joining the TF-CSIRT community (something we will look into in later blog posts). This model lays out the perfect foundation for the building blocks you need to assemble an international class incident response team. Attaining a good maturity rating within this model, enables you to join the TF-CSIRT community and know that you have a very well oiled incident response process. The SIM3 Model is written by Don Stikvoort, who has also been highly influential in the FIRST CSIRT Services Framework.

This model is the golden standard for creating an incident response service, and I will reference it alot throughout the blog posts coming up. It gives you some of the backbone structure that you need to then build upon, to create your own service.

Books:-

Intelligence-Driven Incident Response

http://shop.oreilly.com/product/0636920043614.do

I bought this book after attending the SANS FOR578 course that I mention above. I wanted a supplmental resource to aid my studies in Cyber Threat Inteligence, and this book went beyond my expectations. It really breaks down the incident response process in detail and shows where you can begin to look at it as a driver for gaining threat intelligence. This book really helped solved the problem I will later discuss, around “incident recording” language.

I recommend this book to everyone who I meet within the incident response world.

MITRE – Ten Strategies of a World-Class Cybersecurity Operations Center

https://www.mitre.org/sites/default/files/publications/pr-13-1028-mitre-10-strategies-cyber-ops-center.pdf

This book is available for free from the link above. I was lucky enough to recieve a printed copy from someone I met at the FOR578 training course. This book goes into a lot of great details on how to build a SOC and which resources you should look at to do it. Although the book was written back in 2014 and a lot has changed since then, it still holds alot of relevancy today. The section called “Strategy 4” is very useful in determining which functions should an incident response team have, and how can they be developed if needed.

Courses:-

SANS SEC401 – GSEC

https://www.sans.org/course/security-essentials-bootcamp-style

This course was the first “none vendor” focused training course I ever took, before this I was heavily focused on studying Network Security through the CCNA books. This course helped me understand that the security world was bigger than specific vendors offerings and opened up the gates to my eventual drive into cyber security and incident response. For anyone starting out in this field, this course is very useful as it is very broad and tries to get around most of the important topics in cyber security.

SANS FOR578 – Cyber Threat Intelligence

https://www.sans.org/course/cyber-threat-intelligence

If I look back at any course, or anyhing I have ever studied in general. This course holds the top honours for how much I learned. I went into this course with an understanding of how I thought cyber security worked, and then came out the other side with an entirely deeper knowledge and thought process. This course really helped me understand that data can be so powerful when absorbed from the incident response process. Providing that the data is organized into structures and frameworks to present it in a clear way. I also had the added bonus that the course was being taught by Jake Williams (@malwarejake), and his anecdotes helped to further the understanding of the materials. I would say that this course was the straw which broke the camels back and changed me from being a purely technical orientated person to being much more focused on process and structure. I do not have enough great words in my dictionary for this course!

Other resources:-

Don’t ever underestimate the value you can get from just talking to people, whether they are in the incident response field, or in other fields. A great example is the crossover between incident response and incident management in an ITIL sense. Essentially they are the same process and flow, just that incident response has the “cyber” tag.

Closing words…

This is just a list of the resources that I have used, and they are not complete, you need to find the bits you need from each of them and use it to define your own process.

I have also had the massive benefit of learning from some great people and spending time with organizations like CIRCL, Mandiant, Red Canary to name a few… I just try to absorb as much from the experts as possible…

Incident Response 101 – Intro

I have been wanting to write a set of blog posts about this for a while, possibly I will one day turn this into a book! But for now, it can live here.

Over the last year, I have given a few presentations and lectures about incident response, some of which live on our Github in the presentations folder. But they are not tied together and they aren’t “alive” like a series of blog posts could be…

I would like to share alot of the knowledge I gain whilst working within this field, and studying alongside. A lot of the words coming in the next few blog posts, will be coming from experience of delivering exactly what they say.

A problem that I have found whilst trying to understand incident response deeply, is that most incident response books, courses and sales folk seem to really focus on the deep technical parts of incident response… The forensics, the detections, the reverse engineering, the indicators of compromise etc etc. The “sexy” analysis parts, and the easy sell. What I have been missing is a comprehensive guide to the underlying process behind the whole incident response stack.

Then it struck me, most of the people working within incident response are deeply technical and do get down and dirty with the analysis stage. But they aren’t really strong when it comes to the process. A process which is made up of far more stages that just analysis. This ends up creating a vacuum, where incident response seems highly expensive and complex to the outside observer.

So I have decided to write some blog posts to the “2019 me”. So I can help others who are in my shoes, those who need to build something much more than just an analysis team. Those who need to architect the entire process from alert to end report that delivers great actionable results.

Creating detection rules in Elastic SIEM App

It has been quite a long time since I wrote my last blog post, as with everything, life gets in the way! But I have been spending some quiet time rebuilding my lab, and I have upgraded my ELK stack to 7.6, and I am totally blown away by how awesome the Elastic SIEM app is. So I thought I would put together a few blog posts about how to use it!

Prerequisites

  • You must be running 7.6 (duh)…
  • You must be running the basic license.
  • You must be running at a minimum basic authentication within your setup, between Kibana, Elastic, Logstash etc.
  • You must be running TLS on Elastic.


Enabling each one of these prereqs takes time, and if you are using your stack just for testing purposes and haven’t set up TLS or auth before, then good luck! You are in the lucky position I was last week, and welcome to 2 days of work…
However once you are done, you are ready to move on to the real good stuff…

The good stuff

We will use an example to aid in the instructions, this example is based on creating a detection for each time that there is a Windows Defender event ID 1116 – Malware Detected entry in my logs.

First you will need to open the Elastic SIEM app, and then click on “Detections”.

Once you are in the detections window, on the right hand side you will find “Manage signal detection rules”.

In this window “Signal detection rules”, you can see all the rules you currently have created, or imported. You can manage whether they are activated rules, and many other configuration changes can be done here.

To create a new rule click on “Create new rule”

Within the “Create new rule” section, the first thing you will need to do is to define the index you wish the rule to point at, and then the query you want the rule to run. In this example as I am splitting Defender into a separate index, I have chosen my “sd-defender” index, and then my query is written in KQL (Kibana query language). This query is set to use the ECS (elastic commond schema) field of event.code and will match when it finds event.code 1116. Once you have built this first part, click on “Continue”.

The 2nd stage of building a rule, is to add some more description to the rule…

Here you can name the rule, and write a description of what it is/does. You also assign a “Severity” from low to critical, and a “Risk score” from 0-100. In this case I have chosen “Severity” = High and “Risk score” of 75. When you have finished in this section, click on “Continue”.

In this section you can also add some “Advanced settings”… Where you can supply some reference materials to the alert, if you created it from a blog post, or if it came from a Sigma rule, you could supply a URL here. You can also add some examples of false positives, and then also enrich the rule with some MITRE ATT&CK TTPS! In this example, we won’t add them. But I will be blogging again soon about how to do this part using Sigma rules!

The last part of rule creation, is the “Schedule rule” section. Here you can setup how often you would like the rule to run, and when it does run, how far back in time should it run. This is interesting because if you have just created a new rule, and you would like to see how it would have performed over the last days of logs, then you can adjust that setting here. When you are done setting up the schedule, you can then choose to simply “Create rule without activating it” or “Create and activate rule”, both options are pretty self explanatory!

Once the rule is created, we can try to provoke it and see how it turns out… If you head back to the “Detections” page of the SIEM app. In my example, I am lucky because it is my lab and there is nothing else going on…

Now we will trigger a malware detected alarm, by downloading the EICAR test file to one of my lab machines.

BINGO!

And here is the alert landing in the “Signals” pane, from here we can then begin investigation. Right now there is not very much information about how these alerts will then make it to the attention of someone not using the SIEM app directly. But the SIEM app has some incredible offering here, for free! I have also added a bonus item on how to extract the alerts out to case management tools, slack, etc etc.

Bonus bonus bonus

If you want to extract the alerts out of the SIEM app, you can use a tried a tested tool “Elastalert”. The SIEM app uses a system index called “.siem-signals-default-00001”. This index can be read via Elastalert and the alerts can make it out to your SOC team!

We only need to append

Introduction

As Elasticsearch matures over time, they are fixing some of the less obvious stuff. Seemingly little things can be tremendously important though.

One of the new things, that I want to highlight here is the new security privilegie : create_doc. Read about it the Elasticsearch 7.5 release notes.

As Elastic describes it:

With the previous set of index privileges, users that were allowed to index new documents were also allowed to update existing ones.

With the new create_doc, cluster administrators can create a user that is allowed to add new data only. This gives the minimum privileges needed by ingest agents, with no risk that that user can alter and corrupt existing logs. These administrators can (now) rest assured knowing that components that live directly on the machines that they monitor cannot corrupt or hide tracks that are already into the index.

Have a look at the documention as there is one important change, that is needed in the Elasticsearch Logstash output section.

Implementing it

It is very easy to take advantage of this new feature. Create a role called append_writer and assign a user to the new role:

Or if you prefer developer tools

The final to modify is the output section in Logstash. You need to add an action attribute to it:

Of course , the credentials of the append_writer should be kept in secret store of Logstash!

Conclusion

This simple change is trivial to make, but gives great value. You can rest assured, that the user used in Logstash can never be used to change existing documents in your Elastic clusters.

Using Logstash @metadata

Introduction

In a previous post, I showed how to do simple Kafka and Elasticsearch integration. It showed how to use a single Kafka topic to carry many different types of logs into Elasticsearch.

Have a read if you want to catch up or haven’t read it.

This approach had an undesired sideeffect of putting attributes into Elasticsearch, that are not needed and wasting precious diskspace.

Metadata

However there is very simple and elegant way to fix this. Have a read of the description of Logstash metadata fields here

Previous article suggested this approach. This meant storing kafkatopic,myapp and myrotation in every single document, that went through pipeline.

filter {
    mutate {
        copy => { "[@metadata][kafka][topic]" => "kafkatopic" }
    }

   if ![myapp]
   {
     mutate {
       add_field => { "myapp" => "default" }
     }
   }

   if ![myrotation]
   {
     mutate {
       add_field => { "myrotation" => "weekly" }
     }
   }
}

If we convert to using metadata fields, it could look like this instead. No more kafkatopic,myapp or myrotation being stored.

filter {

   if ![myapp]
   {
     mutate {
       add_field => { "myapp" => "default" }
     }
   }

   if ![myrotation]
   {
     mutate {
       add_field => { "myrotation" => "weekly" }
     }
   }
   # take advantage of metadata fields
   if [myapp]
   {
      mutate {
        rename => { "myapp" => "[@metadata][myapp]" }
      }
   }
   if [myrotation]
   {
      mutate {
        rename => { "myrotation" => "[@metadata][myrotation]" }
      }
   }
}

We can then use the new metadata stuff in the output section

output
{
      if [@metadata][myrotation] == "rollover"
      {
                  elasticsearch {
                                   hosts => ["https://elastic01:9200" , "https://elastic02:9200"]
                                   manage_template => false
                                   index => "%{[@metadata][kafka][topic]}-%{[@metadata][myapp]}-active"
                   }
      }

      if [@metadata][myrotation] == "daily"
      {
                   elasticsearch {

                                   hosts => ["https://elastic01:9200" , "https://elastic02:9200"]
                                   manage_template => false
                                   index => "%{[@metadata][kafka][topic]}-%{[@metadata][myapp]}-%{+YYYY.MM.dd}"
                   }
     }

      if [@metadata][myrotation] == "weekly"
      {
                  elasticsearch {
                                   hosts => ["https://elastic91:9200" , "https://elastic02:9200"]
                                   manage_template => false
                                   index => "%{[@metadata][kafka][topic]}-%{[@metadata][customapp]}-%{+xxxx.ww}"
                   }
      }
  }

Debugging

As all outputs automatically remove the @metadata object and you are trying to debug your conf file, you now need to do a simple trick to display the contents of metadata.

output
{
  # also show contents of metadata object
  stdout { codec => rubydebug { metadata => true } }
}

Conclusion

So by using this approach we are no longer storing kafkatopic,myapp and myrotation as attributes in every single document, that is passing through this pipeline.

We save diskspace,processing time and documents are clean.

Simplifying Logstash by adding complexity

Background

A lot of logs that goes into Logstash will be done using the beats protocol. So you will have a pipeline in Logstash listening for beats on port 5044 typically. This could be stuff coming from filebeat,winlogbeat,metricbeat or heartbeat.

In your Logstash filter section, you will over time end up with a huge mess trying to add the relevant parsing of logs inside a bunch of if statements. In the output section, you could see the same mess again,where you output the different types of logs inside another bunch of if statements.

If you have done stuff like this, your code will be increasingly difficult to read and debug. Not to mention the problems, you will face, if multiple persons need to be able to contribute to the configuration of Logstash. Also if you need to move parsing of a specific type to another Logstash node. Then you need to grab the relevant parts by copy/paste, which is errorprone.

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] =="winlogbeat" {
    #enrich winlogbeat
    ....
  }
  if [type] =="heartbeat" {
    #enrich heartbeat
    ....
  }
  if [type] =="mylogfile" {
    #enrich mylogfile
    ....
  }
  if [type] =="dns" {
    #enrich dns
    ....
  }
  if [type] =="dhcp" {
    #enrich ddhcp
    ....
  }
}

output {
  if [type] =="winlogbeat" {
    #output winlogbeat
    ....
  }
  if [type] =="heartbeat" {
    #output heartbeat
    ....
  }
  if [type] =="mylogfile" {
    #output mylogfile
    ....
  }
  if [type] =="dns" {
    #output dns
    ....
  }
  if [type] =="dhcp" {
    #output dhcp
    ....
  }
}

Simplifying

So what to do about this problem you may ask. Earlier people did some stuff by using named conf files that would be picked up by Logstash to form a large configuration. However we want to be modern and use new features made available by Elastic.

Pipeline to pipeline

I read about pipeline-to-pipeline feature in Logstash a long time ago. There is an excellent article about the options here. This feature is now generally available in 7.4.

It’s actually very simple to implement. You create a pipeline file to receive the beats input and then distribute the events to small tailor made pipelines.

input {
  beats {
    port => 5044
  }
}

filter {
}

output {
        if [type] == "dns" {
          pipeline { send_to => dns }
        } else if [type] == "dhcp" {
          pipeline { send_to => dhcp }
        } else if [type] == "mylogfile" {
          pipeline { send_to => mylogfile }
        } else {
          pipeline { send_to => fallback }
        }
}

Then create a new pipeline to handle the specific log type. This code is restricted to parsing DNS logs.

input {
  pipeline { address => dns }
}

filter {
   # do only your parsing of DNS logs
}

output {
  # output dns
}

You must remember to add all your pipelines to your pipelines.yml file. Remember to think about whether you need in-memory queue or persisted queue per pipeline.

- pipeline.id: beats-input
  path.config: "/etc/path/to/beats-input.config"
  pipeline.workers: 3
- pipeline.id: dns
  path.config: "/etc/different/path/dns.cfg"
  queue.type: persisted
  queue.max_bytes: 4gb
- pipeline.id: dhcp
  path.config: "/etc/different/path/dhcp.cfg"
  queue.type: persisted
  queue.max_bytes: 1gb
- pipeline.id: mylogfile
  path.config: "/etc/different/path/mylogfile.cfg"
  queue.type: persisted
  queue.max_bytes: 2gb

Conclusion

We have started using this approach and will be doing this going forward. We get a much simpler way of handling many different types inside Logstash and we are able to distribute the work to more people.

On top of this we are seeing better latency times in logstash. I suggest to read this article while you are at it. You are effectively using parallel pipelines like the article suggests by this approach.

As always, use this approach if you find it applicable to your usecase.