Teach me everything you know about SIEMs

General
1

While I understand the purpose of a SIEM, getting any kind of advice/recommendations/lamentations on the setup, configuration, and usage of one is next to impossible.

Every vendor has their own solution and there are just as many FOSS ones as well, and while the usual adage of “we all pay to play” should always be considered when you’re comparing FOSS tools vs turn-key vendor solutions, evaluating any of them even on a testing basis is a fairly large lift, if not outright impossible.

So, inspired by Taggart’s response here let talk about SIEM’s. Let’s share everything we know.

  • Which ones are good?
  • Which ones are hot garbage?
  • How much ingest is too much?
  • What are the most important things to ingest?
  • Cloud solutions vs On-prem solutions
  • FOSS vs Vendors
  • How long do you keep the ingest data?
  • What policies/playbooks/automations do you use to separate the wheat from the chaff?
  • Is a SOAR worth it? Or can you DIY a SOAR’s functions with playbooks and automations?
2

This one is a real big annoying “it depends”.

But… it depends.

Ultimately a SIEM is not just something you can quickly implement and be happy that it just works. EDRs you can (kinda, mostly) do this, but a SIEM very often requires a lot of handholding and care. The cloud solutions take away a lot of the burden of care in exchange for money siphoned from your OPEX wallet.

So let me state this - my preference is and probably always will be for on-premise SIEMs. I have used Splunk and ELK in professional settings and my preference is for Splunk, but I have not used ELK in a few years.

I prefer Splunk because I find SPL to be a pretty intuitive query language and because, most importantly, it is what I know and so I’m good at it. When you get into the deep architecture of building Splunk indexers at scale and search head clusters and some of the true enterprise grade solutions, that is out of my depth and where I lean heavily on consultants or SMEs on my team. This is the true cost of using an on-prem SIEM.

To your questions on ingestion - this is hard. My Detection Engineering hat says “only ingest data that you know you can write detections for” but my Incident Responder hat says “only ingest data that is useful in a forensics investigation”. My coworker’s SIEM Engineer hat says “ingest everything, figure out storage and costs and optimization later”. You’ll need to find some middle ground. Decide on the data sources that you NEED in an investigation and in your daily operations - things like SYSMON + DNS + Firewall are probably the most typical starting points but even those require a lot of optimization. In this case the “it depends” is “it depends on your budget and tolerance for missing data”.

How long to keep that data is similar. You start first with your compliance requirements and then work beyond that based on your tolerance for costs and historical data. Keep in mind that you will collect threat intelligence that describes historical attacks and you will be curious if you were a part of that discovered attack 6months or 12months (or more) later.

SOAR is an entirely different topic and, based on my experience with SOARs, if you can build your own you should. If you have the technical expertise and the software engineering experience to build (not vibe code) a secure mechanism for communicating with a dozen API endpoints across your enterprise to enrich events and perform automated actions, you should. Any off-the-shelf tool you buy will have dozens of features that don’t work in general or for your specific usecase and SOARs by nature are inherently specific and not generic tools.

3

I was having a discussion with a co-worker about buying turn-key solutions vs building your own in terms of SIEM’s and SOAR’s and this was the exact point I made at the time. Ultimately we want the SOAR to auto-action on sus data from the SIEM. Turn key vendors that claim to solve this problem for us can solve the very generic problems, but the real value is going to come from our specific environment, our specific tech-stack, and our specific traffic. We will always have to build those ourself. So if we have to build the valuable stuff anyways, why are we paying some shithouse company to store our internal data in the cloud where we no longer own our data and are tethered to them forever.

By and large I agree with the rest of your assessment. A lot of the specifics are going to depend on that balance between cost, risk, and overhead.

I’ve used Splunk a tiny bit in the past, and while it is very cool. One of the main problems I’ve had with it is that you need to know what you’re looking for to get value out of it. I don’t think it is able to surface issues to you based on x telemetry, or y heuristics. But maybe I’m wrong on that.

4

I only have hands-on experience with three, but here’s my take on them in the order of my experience with them from earliest to current:

Elastic (Open Source)

We initially set up Elastic when it was open source and came with Wazuh. It worked well as a SEIM, in that we could get alerts from it and then find the events in the logs that triggered the alert. It was happy days as we were able to shape it the way we wanted, but that also meant we had to write a lot of our own connectors. For example, we were using Forcepoint for Internet security at the time, and to get the logs from Forcepoint to Elastic I had to write a 3000+ line long multithreaded Python script that pulled their XML wrapped CSV files (I am not kidding) from their service, put it on a separate file server where a customized CSV ingest connector could pull the data into Elastic. But once it was going, and so long as our file management was on point with the ingest server, it just worked. We kept 90 days of Internet logs, the most we’ve ever had. Life was good.

Elastic (SaaS - Cloud based)

When Elastic finally went commercial, we stuck with it. Our price didn’t increase too much as we were already a customer, but it did increase. However, we no longer had to maintain the infrastructure on which our data resided, so it freed up time to do more analyst and less engineer work, and freed up our security engineers to move to new projects rather than maintaining infrastructure. Overall it worked well and we were content. As SaaS their connector library got better and better and we were able to ingest logs more easily and add to our growing list of data sources considerable. Those were happy days.

Rapid7

But eventually the global nature of our company caught up with us. We (the U.S.) was the first to have a SOC and any sort of dedicated security team, but eventually the company “got around” to looking to do the same for everyone else around the globe. And since they were in charge, we had to do it their way. But frankly their implementation of Rabid7 was hot garbage and the MSP they hired to run it was as well IMO. So we kept Elastic or ourselves and did double long ingests to give them what they wanted. I had access to Rapid7, and would occasionally have to use it whenever it actually generated an alert that wasn’t a false positive, but it was neither a consensual or happy experience.

Sentinel and XDR with Microsoft Defender for Endpoints

in 2024 company changes brought new visions to the table and the desire to unite globally under the one company everyone was already paying licensing fees to: Microsoft. With that came access to Sentinel and XDR - two products that are both SIEMs, do the some thing for the most part, have a shared query language, and only differ in that Sentinel is focused on Azure tenant level SIEM data, and XDR is focused more on client level logs. But boy howdy are those blurry lines. So much so that Sentinel is being merged into the XDR dashboard. That was supposed to start happening next month, but Microsoft postponed it to next March as they always so. Knowing that Sentinel would eventually get moved into XDR, I have spent 90% of my last two years in XDR, so I’ll talk mostly about that. What I will say about Sentinel is that it can pretty much do everything that XDR can do but with an interface that is even more engineer than XDR if you can imagine that (meaning an engineer designed it but they have never daily driven it so don’t really understand the ramifications of their design decisions.)

Of the three SIEMs I have used I find XDR to be the most capable with the very Microsoft forward environment the company has. However, if we were not so Microsoft dependent, I would hands down go back to Elastic just because we never had to wait days to get responses on trouble tickets and weeks to get things fixed. The biggest issue with XDR isn’t the platform, it’s the servicing of that platform. My company is big, but it’s not Fortune 500 big so we often feel like the red headed step child if you know what I mean. But fortunately XDR hasn’t had that many issues. That seems to be the purview of Sentinel, which I don’t rely on. I write all my rules in XDR and run them as XDR rules. And while I liked Elastic’s query language (EQL), I actually really like KQL. With EQL it was hard to do statistical analysis, and it sometimes felt like I had to jam a square peg into a round hole. But KQL does statistical analysis. It’s also a fairly capable programming language. While I might be able to make our logs dance a jig while singing Sea Shanties with a robust Python script, I can certainly make KQL hum along and hit the correct notes. It’s a better hunting environment than EQL, and that makes me happiest about the switch. What makes me the unhappiest is that most logs only give us 30 days of data, but that is changing.

Summary

That’s a lot of words and I’m not even sure I answered your points. So here’s my stab at your list.

  • Which ones are good?

    • Elastic for non-Microsoft, XDR if married to Microsoft

  • Which ones are hot garbage?

    • Rapid7

  • How much ingest is too much?

    • There is no such thing as too much ingest, but 90 days is good.

  • What are the most important things to ingest?

    • IMO, environment dependent, and in order - Internet access logs, sign in logs, device activity logs (processes, network activity), email gateway logs, database (SQL, etc.) logs, everything else.

  • Cloud solutions vs On-prem solutions

    • Small and medium business on-prem, large business needs cloud or infrastructure costs can eat you.

  • FOSS vs Vendors

    • Yes? We literally have both in our security stack. Use what fits your needs. Donate to your FOSS!

  • How long do you keep the ingest data?

    • As long as infrastructure and/or the accountants let us.

  • What policies/playbooks/automations do you use to separate the wheat from the chaff?

    • OMG, start another post please.

  • Is a SOAR worth it? Or can you DIY a SOAR’s functions with playbooks and automations?

    • Good questions. With Elastic we used TheHive as a SOAR so our analysts had playbooks, etc., to guide them and automated lookup tools (Cortex). A lot of that comes built into XDR. I can pull geo and registration data by clicking on an IP in a query result for example. We have DIYed anything else we needed. I feel that the role of a SOAR is being pulled more and more into the SIEM itself. One of the regional SOCs in my company uses a custom SOAR to management Sentinel/XDR alerts. IMO it’s more of a hinderance than a help and I can analyze circles around it. So no, a SOAR is not worth it to me using XDR. However, it provided missing functionality with Elastic; even their cloud offering. (Edit: and having reread that, I want to add that the reason for the SOAR makes a big difference. With Elastic, it added functionality Elastic did not have. That’s a good reason to have a SOAR. If your company’s reason to have a SOAR is because they think it’ll allow them to hire low skill analysts and save money, they have just taken their first step to being ransomed.)

Hope that’s at least entertaining!

5

Splunk has the benefit of significant market share and a vast array of integrations. Many of the apps built for Splunk can do reasonably automated surfacing of issues with pre-built queries.

But hoo boy are you gonna pay for the privilege.

I will say though, that I shouldn’t be looking at a SIEM to generate initial alerts. Something should, but not a human. This is why EDR rules-based alerting that correlates to SIEM data is so valuable.

6

Thank you for this phrasing.

I too really enjoy KQL, but I must say that Elastic’s shift to ES|QL has been an absolute sea change, to further the imagery. Both API and Kibana-based data collection is 1000% smoother with the new language.

7

I was just getting into ES|QL when we moved to Sentinel/XDR! I 100% agree tough I only had a few months with it. I was really starting to get some interesting queries going beyond the ‘unknown IP signed into an account’ stuff and I wish I could go back and spend some more time with it.

The other thing I really liked about Elastic was that, because I am that old, our security engineering manager gave me access to the admin console so I could make sure we didn’t exceed data limits. Not only did that lit me manually manage our logs, but it was like having a bonus level of query capability, so long as I didn’t need to look at individual records. Since we parsed out data into multiple files and rolled over frequently, I started monitoring individual indexes for abnormal growth patterns - a malignancy of maliciousness so to speak - that might let me know some extra unwanted traffic was happening. But I could also do that with index record counts to. I just got intrigued by the possibility.

I would 100% love to spend more time in Elastic. I was even running Wazuh and Elastic in my home lab up until the time I ditched my last Windows box.

8

I’ve used Splunk a tiny bit in the past, and while it is very cool. One of the main problems I’ve had with it is that you need to know what you’re looking for to get value out of it. I don’t think it is able to surface issues to you based on x telemetry, or y heuristics. But maybe I’m wrong on that.

So Splunk has tried to propose a philosophical approach to this problem where they use tagging and common field name standards to solve this but it is a hard problem and I’ve yet to see anyone actually follow their model.

The problem with Splunk and most SIEMs is that YOUR data looks distinctly like YOUR data and it is not easy to just jump into an environment and use their SIEM.

I’ve had better experience with the KQL environment for this reason because it is all very standardized - M$ is responsible for all of the fields so they can set the standard themselves. That being said - you’re not ingesting NGINX logs into Sentinel and if you were to you lose that optimization immediately.

I literally created a Taggart Institute course on how to handle the complexity of converting Sigma standard fields into whatever it looks like in your Splunk because it is the kind of thing that needs a training wheras in KQL it is quite a bit more intuitive.

That being said - Microsoft already has enough of my data and they don’t need more. They especially don’t need to charge me money to store my own data.

9

And OMG do they charge for the privilege!

That aside since I don’t pay the bill, HQ has managed to get our Zscaler and a fair number of other logs into the less-expensive uber-expensive M$ storage option - Aux.

Aux_logs
Aux_logs342×1411 44.2 KB

They are using Cribl to ingest the logs into AUX storage. It works so long as the Crible ingestion nodes don’t have a hiccup. Unfortunately it is not bullet proof enough for our SOC to rely on the data we need being there at the time we need it. This also does not translate to XDR whatsoever. I’ve raised the question of what happens to all this expensive infrastructure when Sentinel and XDR eventually merge. I got shrugs. You know, it’s not my money, but I can’t help thinking that it may all become wastes effort. Time will tell. They pay be regardless.

10

I am going to say something most people don’t want to hear.

At some point, you will need to make a judgment call about what goes into the SIEM, and what you can’t afford to put in there. Tools like Cribl can reduce log noise, but ultimately you will likely have more data sources than capacity. At that point, you need to decide what makes an impact for you. This could mean being judicious about Windows Event logs; it could mean reducing retention for certain indices.

When you need to make that decision, you’ll want a table somewhere that broadly has the following columns:

Source Description Detection Value Loss Cost
Sysmon Enhanced Windows event Logging High Endpoint activity

What is it, what does it provide, and what would you lose if you removed it? That can help make clear what really matters in the SIEM.

11

I fundamentally agree. From the DE perspective, there a lot of logs that I would love to have but I have to consider what the impact of those logs will have on the detections I write or if I even have ideas on how to make useful detections from them.

I’d love WAF logs but for my business they don’t provide much value.

12

Just a guick update on this. Last month HQ finally connected XDR to our Sentinel workspace. We now have access to all Sentinel tables in XDR. It is a huge improvement. While there are still some issues with log ingestion latency (that’s not an XDR/Sentinel problem) it is improving. While I can’t alert off the Sentinel tables except in Sentinel, I can at least hunt the data there. It’s still M$ expensive…

13

If you ever want some cool KQL challenges that teach teach really effective hunting strategies, I recommend: