myname.website

I've been using Cloud9 as my main IDE for years, since before Amazon bought them. Sitting here with an anemic bottom-barrel 2015 Macbook Pro with just 8GB of RAM and 128GB of SSD doesn't leave a lot of wiggle room. It's hard to just take a new language for a spin, having to install all the dependencies and frameworks and nonsense that comes with programming. Cloud9 was a breath of fresh air when I found it. And since Amazon bought it, it's gotten even better (although slowly).

The biggest driver keeping me on Cloud9 is AWS's auto-sleep feature. If I stop using the IDE for 30 minutes, it'll shut down the EC2 instance that's running behind the scenes and stop billing me for it. That's a pretty huge deal. Right now I have 8 Cloud9 instances on my AWS account and my monthly bill is about $3. It's also free for a year if you qualify for the free tier. It's a fully-fledged IDE with full root access to a full Linux system under the hood. It's a full development/test environment, letting you run your code and see it work in a real browser. Postgres and Redis running too. I mean, it's the real deal.

But one thing that's bothered me for a while was the lack of iPad support. I mean... it's a browser based tool! Why shouldn't it work? It kind of worked in iOS12, except Safari didn't support some keyboard keys necessary to navigate the code. iPadOS 13 seemed promising, but Cloud9 needed third party cookies and Safari on iOS13 and 13.1 didn't actually support enabling third party cookies. But along comes iOS13.2 and a quick update from AWS and we're in business!

There are still some downsides of Cloud9 though. It's hard to get your running instance accessible from outside of your browser window, for example. Meaning if you want to see how your code runs on a real phone (not just Chrome's impersonation of a phone), there are a ton of poorly documented hoops to jump through. And if you have a decoupled front/back end, you'll either need to jump through these hoops or run both servers on the same Cloud9 instance. You can't test your APIs with Postman without the same headaches. There's also extremely limited and experimental plugin features. For the most part, what you see is what you get. Which is fine for me, so hooray!

But why would you want to code on an iPad?

Hipster cred, mostly. Joking aside, I read this article on coding on an iPad years ago and I've never been able to shake the feeling that this should be possible. And I shouldn't have to use Vim when Cloud9 exists in a browser.

I travel a lot for work, and I also maintain an ever-growing stable of side project that need occasional maintenance. If I'm expecting to be on-call for something on a side project, I'm not doing that work on my corporate laptop. So I end up bringing two laptops, plus an iPad to read comics/books during the flight, plus two phones. It just gets to be a lot. Even when I head down to my local co-working space, I'm often lugging two laptops, just to work in an application that's browser based but for some reason doesn't work on every browser. Until now.

But while his VPS cost him $20/mo way back then and a modern equivalent is just $5, I don't program often enough to rent a server full time. And I like the ability to spin up/down new instances whenever I want but only pay for them when I'm using them. That's a pretty awful experience on Linode or DigitalOcean etc. With an EC2 instance you can sleep, but your IP address goes away unless you pay $5/mo to keep it, which defeats the purpose. Cloud9 makes those problems go away.

I should mention the Lightning->HDMI adapter works nicely, too.

My setup

Most of the time I'm coding on my Macbook using Cloud9. Maybe that'll change now that the iPad works, but we'll see. I do all of my personal projects in Cloud9. Three Rails apps, an Elixir/Phoenix app, a Flask microservice, a Nodejs microservice, all developed 100% entirely on Cloud9. One of the Rails apps actually pre-dates the AWS acquisition, and I'm still actively maintaining it using Cloud9 with thousands of lines of code. The only things I write locally are things I don't plan on putting into production (bash and python scripts, mostly).

Besides the base model 2015 rMBP, I have a base model 2019 iPad with the official Lightning-HDMI adapter. Unfortunately the adapter only mirrors and does not do full 1920x1080 (it stays at the same aspect ratio as the iPad itself). I have a Logitech K380 keyboard that lets me quickly switch between three different devices (my personal Mac, my corporate laptop, and my iPad). On my laptop I use Chrome for the developer tools. Now that iPadOS supports mice, I also connect a Bluetooth mouse sometimes, although getting used to long-clicking rather than right-clicking is weird. Other than that, the mouse and keyboard work swimmingly.

Before Cloud9 worked on an iPad, I used to SSH into the instance with the help of Termius and mosh, but I've never been super comfortable at editing multiple files at a time in Vim.

Do I actually do this?

Mark O'Connor in that blog post got quite a bit of flak on Hacker News for wanting to code on an iPad. But I get it, Mark. It's liberating. It's not especially ergonomic unless you have an external monitor or some kind of stand for your iPad, but having basically a single-purpose machine is great. The number of times I've had to wait for a React app to recompile and in those 10 seconds I've switched to another browser window and then 15 minutes later I come back to finally try the code... I have a short attention span. The easier it is for me to multitask, the more likely I am to multitask. The iPad is awful at multitasking, so it keeps me focused. And everything is full screen by default.

If nothing else, it's a great idea. Much like coding in LISP or Haskell, you might not use it every day but the concepts you learn can be applied to your normal routine. For instance, I pretty much always keep my active window full-screened on a virtual desktop on my Macbook, and I hide the dock and top bar. Removing distractions helps keep my mind in the zone.

So we'll see, now that Cloud9 supports iPad, we'll see how much I take advantage of it. Especially, like with Mark O'Connor's blog post from so long ago, my Macbook is getting a little long in the tooth and ready to be replaced... but there's not a single laptop on the market today that I'm excited for. The iPad may become my daily driver sooner rather than later. Maybe I can talk my wife into letting me get a 12” iPad Pro...

Other iPad programming resources

Cloud9 isn't the only tool, and there's a pretty nice market for iPad apps for programmers. It's growing for sure, and with the iPad Pro and enhancements in iPadOS, I think it'll keep growing. People use 11” Macbook Airs and 12” Macbooks, so is a 10” iPad or 12” iPad Pro so strange?

Play.js is a native iPad app for writing React and Javascript.

Pythonista is pretty good, or so I've heard.

Panic's Code Editor is a pretty full-featured programming suite.

I recently had a post make it to Hacker News, lobste.rs, Pinboard, and reddit. This was not my doing, I have no idea who submitted the article. Honestly I was a bit surprised that anyone even knows this blog exists. I write this as a therapy session for myself when I need to rant but no one needs to hear it. If I knew I was writing for an audience, I would have written a lot differently.

The biggest thing I noticed from reading the comments especially on HN was people who don't understand what I'm talking about and think I'm just ranting because I hate everyone and everything. That may be true, but that's not exactly the point of the article. I can easily forgive the misunderstanding because, again, I wasn't writing for an audience I was writing for myself. So let me take the opportunity to clear some things up.

This misconception was repeated a few times in various comments: “syslog is a standard and if you don't like it you just don't like your job”. So let me explain, from a security standpoint, what's wrong with syslog.

Syslog for security logging

Let's start by explaining what syslog is and what it does in a security context. The product I work with is a SIEM, which basically collect logs from around your environment and combines them together to find security problems. So we'll collect logs from firewalls, domain controllers, IDPS systems, Linux servers, AV systems, Tripwire, everything. Everything we can get. The majority of these send their logs in syslog format. The logs we get aren't necessarily sensitive on their own, but they do provide a lot of contextual data that attackers would love. If you want to see alerts around one account failing to authenticate, you necessarily need to send that account's username, as well as the source and destination of the traffic. Now any attacker who sees that log knows the address of your domain controller, a valid username, and what IP address they're connecting from. This is... problematic from a security standpoint.

Now mainly my biggest beef is security vendors who want to ship syslog over the Internet. I've only encountered it a handful of times, but once is bad enough. But that's just an add-on to how outdated syslog is, especially for security logging. Syslog is a clear text protocol. There are methods of encrypting syslog traffic, but it's not standardized and it's up to the vendor to support it. Most don't. The best way is using a VPN tunnel, but then you have the overhead of a VPN tunnel and another point of failure. Again, we're talking security logs. If your logging goes down, so does your security.

And if you don't want the overhead of syslog over TLS or stunnel or OpenVPN, that means you're sending clear text security logs over the wire. On your internal network that's bad enough, but over the Internet?! You'd have to be insane (or work for Cisco).

No standard formatting

So that's the security beef with syslog over the Internet. Let's move on to all the other reasons syslog is terrible. The next one is the lack of a logging standard. IBM's product uses LEEF format, that looks like this:

LEEF:2.0|Trend Micro|Deep Security Manager||192|cat=System name=Alert Ended desc=Alert: CPU Warning Threshold Exceeded\nSubject: 10.201.114.164\nSeverity: Warning sev=3 src=10.201.114.164 usrName=System msg=Alert: CPU Warning Threshold Exceeded\nSubject: 10.201.114.164\nSeverity: Warning TrendMicroDsTenant=Primary

HP/MicroFocus's product uses CEF, which looks like this:

CEF:0|Trend Micro|Deep Security Manager||600|User Signed In|3|src=10.52.116.160 suser=admin target=admin msg=User signed in from 2001:db8::5

Linux syslog generates unstructured garbage which looks like this:

Failed password for root from 192.168.50.65 port 34780 ssh2

Cisco's IOS products generate logs like this:

00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down 2

So now your syslog receiver (the SIEM in this case) has to parse these multitude of logging formats. CEF and LEEF are nice because it's at least some structure to key your searches/regex on. Other syslog formats... not so much. And in the world of security, parsing a message wrong means potentially labeling the victim as the source of the attack and the attacker as the destination. In a real SOC, this would be the difference between an external hacker breaching a machine and getting a foothold in your network versus malware on one of your workstations talking to a command and control server. That is a massive difference and working under the wrong assumption could cost hours or days of investigation. Meanwhile the hacker is still crawling around your network.

Unreliable for security logs

So now that we know syslog is insecure and difficult to parse, let's move on to its reliability. Syslog has no function built in to retry a message. If your network goes down or your syslog receiver (your SIEM) goes down, those messages are lost. The sender has no idea that no one is listening. Sure you can use TCP but... I wouldn't recommend it.

So in the real world, this means if you're under a DoS attack and your syslog receiver/SIEM goes over license, or over capacity of the hardware, or the network link drops, or the attacker changes your firewall rules or any number of ways the link between the sender and receiver can be severed during an active attack... all those logs are lost. Gone forever. You will never get them back. Which means once the dust has settled, chances are you still have no idea where the attack came from, what happened, or if the attacker is still inside your network. Because your devices were dutifully sending syslog blindly assuming the receiver was listening, and the system has no idea those logs were lost. Unless you're using TCP, in which case when the link gets severed for whatever reason, your entire network crashes.

What would a robust logging solution look like?

There's plenty of robust logging solutions. Sure lots of them have their own issues, but let's take it one step at a time. JDBC is a good one. It's a pull not a push, so as long as your SIEM can remember where it left off, it will always get the missing logs after the dust settles. Many Windows agents are good at this too (many are not... YMMV). Properly written API clients can do this as well, remembering where they left off and resuming when the network connection drops. With the added benefit of, if you're using HTTPS you're not sending clear text logs across the network or Internet.

So, for the love of god, stop using syslog for security logging.

(Hey HN/Reddit/Lobste.rs! I didn't expect this to take off, there are a couple of unclear things in here. The biggest seeming to be my hatred for syslog. I've published a follow-up explaining why syslog is terrible for security here)

A couple of weeks ago I had a meeting with my boss where I told him I was burned out and was looking for a new opportunity. “Burn out” is the polite way to say it. In reality, I'm disappointed. I'm mad. I'm pissed off.

I'm pissed off at the state of information security. I'm pissed off that our tooling is falling behind. I'm pissed off that my clients don't seem to take it seriously, and I'm pissed off that the vendors don't seem to want to help. Let me ask you: is the state of information security really any better today than it was 8 years ago when I started? The easy answer is no. The better answer is, it's worse.

I'm an information security consultant. I work for a vendor. I won't say who because this isn't about the who. I'm absolutely not picking on any one person or company or team or product. I'm picking on the mindset of the entire industry, and the companies who use our services. This is ultimately going to end up being multiple posts, but every therapy session has to start somewhere.

So what am I mad about? I'm mad at how often I have to say no. No, we can't do that. No, it doesn't support that. No, the vendor doesn't allow for that. No, you don't have the right license. No, no, no. Isn't the point of technology to enable businesses? So why am I saying no so often?

I'm mad that my clients are still getting breached. That I've installed the best-in-breed security software, configured it with the best-in-breed rules and add-ons and analytics tools and staffed it with a best-in-breed SOC using the best-in-breed processes and a Wordpress vulnerability brings down the entire company. Or a misconfigured S3 bucket. Or an employee sending an email or browsing Facebook, or sharing a document.

I'm mad at my company's software. It was built for a time that no longer exists, where machines kept the same hostname and the same IP address and served one function. A time where infrastructure never changed. I'm mad the software STILL expects the infrastructure to never change. I'm mad that it keys off of IP addresses first and foremost. In a world with software defined networking, where the IP address can change on a moment's notice or the service is load balanced, it just doesn't make sense. I'm mad that it will create records of machines it has seen on the network and send out alerts when those machines drop off the network or change hostname. Of course they do! It's fucking VDI! That's the entire point!

I'm mad at other vendors. I'm going to call out Cisco by fucking name and elaborate more on this in a follow-up post, but the simple question is how do you move a god damned security product to the cloud and not have a logging API?! I'm mad that I sat on a call representing my company's cloud offering listening to Cisco tell us that the only way to get logs from god damned IRONPORT in the cloud was to use syslog! OVER THE INTERNET. FOR SECURITY LOGS. (For more info on why syslog and syslog over the Internet are bad things for security, see my follow-up here)

I'm mad that syslog even exists anymore. But I'm even more mad that Windows can only natively log to other Windows systems and logging to a Linux system requires an agent. I'm mad that my clients refuse to install our agent. I'm mad that agents have been so poorly written in the past that they've given agents a bad name forever, and I'm mad that OUR agent was that bad in the beginning too.

Mostly I'm mad that our product can't consume logs from an API without being programmed to do so by a team that's so far behind that they're still trying to support Compuserve, and I'm mad that Microsoft changes their Azure logging format so often that our integration team spends half their time re-writing our Event Hub and O365 support.

So what am I going to do about it?

That's the meat of a follow-up, but it all started with the last point I mentioned: API support. I've spent the past few years becoming more and more disillusioned with how traditional info sec has been run, and I've spent the past few months helping to bring the product I have some control over kicking and screaming into the modern world. I've been writing apps that connect to various APIs to convert their logs into old-school syslog so our product can consume it.

And along the way I learned a ton about cloud technologies. About Azure Sentinel and Chronicle Backstory. About microservices and Kubernetes and zero-trust networking and BeyondCorp and real actual security.

And at the end of that road is the end of my interest in SIEMs and syslog and hardware appliances and software agents and legacy OSes. Leave it all for dead, discarded at the side of a road like that broken down Olds 88. Take off the license plate, file off the VIN, and walk away. It's someone else's problem now.

Here's a question: the laptop or cell phone or tablet in front of you... do you own it? Maybe you're financing your phone from Verizon, but even if you paid full price and do not owe another dime to anyone else for it... do you own it?

Every so often I encounter hackers who demand nothing but free software, unlocked bootloaders, and open source hardware. They quote Richard Stallman's GNU Philosophy at me and claim they “own” their laptops, they “own” their cell phone. I disagree. No matter how much you control the software, you don't own a god damned thing.

The problem starts with the Internet. In the 90s, the Internet was described as “the information super highway” and it's actually a perfect metaphor today. Because I own my car. No matter how many payments are left on it, no matter how often I open the hood, no matter if I change my own tires or not. I own that car. Because I am singularly responsible for that car.

Let's take a metaphorical detour from our metaphor's detour... Stallman and “free software” folks like to talk about two different uses of the word “free”. There's free-as-in-beer which means you did not pay to use it. And there's free-as-in-freedom which means you are unrestricted in your use. So when we talk about “ownership”, I want to give two definitions of “ownership”. The first being it's-in-my-sole-possession, and the second being it's-my-sole-responsibility. People love to talk about the possession aspect of ownership, but often forget the responsibility aspect of ownership.

Back to the car analogy – if a tire flies off my car on the normal non-super highway and my car crashes into someone else's car, I have to pay to repair my car because I own-as-in-sole-possession-of that car and no one else will pay to fix it. But I also have to pay for the other person's car that my car damaged, because I own-as-in-responsible-for my car. If my car that I own causes damage to someone else's personal property, I am responsible for that damage.

So if my Internet-connected device is on the information super highway and causes damage to someone else's personal property, what happens? Nothing. Absolutely nothing. If my device causes damage to yours, I'm not held responsible for a single thing. If your device damages mine, likewise I cannot sue you for damage. If I don't clean malware off my computer and it spreads to yours, tough luck. That's just how the Internet works.

So do you actually own your device? You installed your own free software, the hardware is open source, and the bootloader is under your complete control... but if you don't take responsibility for damage your device causes... do you actually own it?

Unless you're willing to own-as-in-responsible-for your devices, you don't actually own your devices.

The Maginot Line in World War II was the French defense against invading German forces. It covered almost the entire perimeter of the country, was impervious to attacks from the air or ground, and had backup supply lines on the inside of the perimeter. It was well staffed with trained and experienced soldiers. It was hugely expensive, state-of-the-art, and considered impenetrable. It had been built by the world's most powerful military at the time, and they spared no expense.

Sound familiar? I'll bet it sounds like your own corporate network, as seen by management.

Now the French did leave one tiny gap... but it was only exposed to a trusted third party and not directly to the enemy. The French expected that if that ally got breached, there would be enough notice to secure that part after the third party vendor nation went down.

Sound familiar yet?

Now the Germans knew this defensive line was up. They also knew there was a way around if they only could breach the perimeter of Belgium and the Netherlands. Of course that would set off alarms, but they were prepared for that too. A section of the German army stood as a decoy along the Maginot Line so when the tanks rolled through Belgium and into France, the French were too busy defending against a decoy attack on the already impenetrable part of their defenses and spent no time at all watching the weakest link. The Germans strolled right past. And after the German tanks were in and the French forces moved to fight them, no one was defending the Maginot Line. One month from the time the Germans reached the Belgian border, Paris was captured and all was lost.

One of the most successful denial of service attacks of all time.

We can learn a lot from history, even in information security. Is your company's perimeter a Maginot Line? Well defended (except in the parts it's not) yet easily distracted? Every firewall deny alert you respond to from your SIEM could be the enemy forces distracting you. That's exactly how DoS breaches happen. Your SIEM becomes useless, your firewall overwhelmed, your analysts distracted... and then they turn your vendor-facing firewall into Belgium circa 1940.

How much money did you spend on your Maginot Line? How much time do you spend watching attacks that are already being blocked? And how many actual attacks do you miss because of it?

On January 29th 2019, the Director of National Intelligence sat down in front of the United States Senate and warned them that China and Russia had the ability to shut down natural gas distribution to the United States. That same day, a record cold snap hit the Midwest United States, a cold front that lingered for a number of days. And then on January 30th, the very next day, Michigan's largest natural gas supplier (Consumers Energy) was struck with a disaster: a fire at a natural gas substation that knocked out 67% of the natural gas supply to the state of Michigan. GM, Ford, and Fiat Chrysler agreed to cut back on gas usage at their factories, the entire state agreed to lower their thermostats on a night where the wind chill dipped down once again to -20F. The cold weather across the Midwest has been linked to 10 deaths at the time of writing. We're not talking about a bit of a chill, we're talking deadly cold.

Now, I have no proof of any cyber attack. And even if it's true, I doubt we'd ever hear about it. That's not the kind of thing you want to admit. But let's just pretend for a second.

The US Director of National Intelligence admits publicly that China and Russia can, at any moment, take control over and shut down America's power grid and natural gas supply. The very next day, during a record cold wave, one of the states most impacted by the cold loses more than half of its natural gas supply in a freak disaster just as everyone is headed to bed.

If I was China and I wanted to confirm what the DNI said was true without facing serious repercussion, if I wanted to slap the US across the face, what would I do? Forcing everyone in a single state to lower their overnight furnace temperature by a few degrees would do it for sure. No one is going to die, it's just a minor inconvenience, but during a time when people are afraid to go outside, afraid that if their heat goes out their family might freeze to death in their sleep, it sure is scary.

It could just be bad luck. Mistakes happen, demand was higher than the machines could handle, and a disaster was inevitable. But if China or Russia wanted to confirm what the US DNI said and show the world they really do have control over the US's energy grid... I can't think of a better time or better way to show that off.

The media is quick to cry out about the TRITON attack at Triconex, calling it a “watershed moment” for cyber security. Nonsense. Remember Stuxnet? Whatever happened with that? Did we stop using off-the-shelf Windows for critical infrastructure? Did we stop transferring files to airgapped machines using USB drives? Did we stop hard-coding default passwords? Did we start monitoring communications between PLCs and their controllers? Hell, did we even take a second to try to understand what the normal operations for our critical IT infrastructure looked like?

Or how about CryptoLocker? How do we measure the money and man-hours lost due to an encrypted drive that needs to be rebuilt? How many of us went out and set up TimeMachine, Crashplan, iCloud backups, tape drives, anything after ransomware came out? How many of us back up every single machine? How many organizations still run their NAS on Windows and connect to it from Windows, providing the perfect vector for the spread of worms?

WannaCry. 200,000 victims. The NSA discovered an exploit but kept it to itself instead of letting Microsoft know so they could patch. The consequence was 200,000 victims of WannaCry, and hundreds of hours of paid consulting work for people like me (thanks for the nice Christmas bonus!) cleaning it all up. And once it was done, you had… a few more rules in your SIEM. A new IPS signature. And you feel safe because it hasn’t come back, right?

Then Petya. The same god damned thing as WannaCry, the attack you all feel safe from now because you paid hundreds of thousands of dollars to “fix it”. And you’re strung out in the cold again because of a new attack that was “totally unpreventable”. Then NotPetya, after you were all trained to pay the ransom and check the box in your business continuity plan like good little victims… **surprise! **NotPetya doesn’t actually decrypt your files after you pay! Suckers!

Again, the attacks came through Windows. But Mac is too expensive, and Linux is too hard to use, right? If I just ignore every other option that exists, I can pretend there are no other options. There’s absolutely nothing we can do to stop it. Just accept that you’re already dead, because it’s slightly more convenient, and besides, IT infrastructure is a sport, right? Gotta root for your favorite team!

Bu-bu-bu-but Target! Aaaaaand.. Home Depot! But Equifax!

All changed nothing in the security world. Cards are still getting skimmed. Payment machines are still running on Windows. Equifax had their incident response website breached while the incident they were responding to was still happening, and then they got breached again after that.

So why the hell would anyone call TRITON a “watershed moment”? What exactly is so different about this one? What makes us change if all these recent attacks haven’t? Because it’s a nation-state? So was Stuxnet and Flame and Duqu. No one gave a shit. Because it shut down critical control systems? Anyone remember the 2015 attacks that shut down the power grid in Ukraine? No? That’s okay, it was apparently so forgettable that Wikipedia doesn’t even list the name of the malware, BlackEnergy.

You want to know why TRITON will be forgotten? Why it’s as far from a “watershed moment” as you can get? Because all of those other attacks happened and nothing changed. Nothing happened. We’re still doing exactly what we did before, only now we’re giving more money to security consultants who write rules looking for exactly that one attack, and when the next one happens and it’s slightly different, we all lose our collective shit because “no one could foresee this happening!”

What will it take to reach the real “watershed moment”? Deaths. Not just one, likely multiple. And not just any deaths. Media-friendly deaths. Something we can all relate to. The headline will read “Hackers holding daycare hostage after pwning their IoT smart locks”, the nation watches in fear as the timer for the ransom ticks down. All streamed live to your Android 2.3 phone, because “iPhone sux lolol” but Samsung never released an update after you were locked into a two-year contract.

Then, and only then, will anyone care about preventing cyber attacks before they happen.

(At the time I wrote this, I had never heard of “zero trust networking”. Looking back, that's what I'm describing in this post. If you're interested in these concepts, please read up on zero trust)

The way we are dealing with security is all wrong. No one can deny it, but no one is doing anything about it. Why? Because “the business”.

Security cannot and should not interfere with the experience of the end user, that is a fact. That is the only way we can sell security to users. The alternative is no security at all. But other IT teams are not end users. They’re technical and they understand limitations and workarounds. Security can and completely should impact them. Security should be a gate for IT teams, letting them through only after checking their badge and ID. You don’t let visitors into your office without checking their badge and ID. Why do IT’s technical whims get a free pass?

Application owners far too often get away with not knowing how their applications behave on the network. Networking teams far too often get away with not communicating their network architecture or changes to their architecture to the rest of the organization. Server teams far too often get away with not knowing what applications run on their servers.

How often on a security team do you see an IP address in your SIEM and have no idea who owns it, what the purpose is, and what applications are running on it? Far too often. It’s unacceptable. It should be considered the highest priority for teams to work with security not reluctantly, not when they’re told to, but as a rule. Security not knowing what is running on a server should be as critical as an application going down. It should be hair on fire. It should make or break careers.

Looking for projects for the new fiscal year? How about this one: give security a list of every port needed, every external network range needed, every server that needs to talk to every other server. And after that, block everything else. Put rules in the SIEM that create alerts for every firewall deny and everything that does not match that profile. Every new server and every application will have to be specifically allowed before it will work. And every firewall deny is a severity 1 security incident.

“But that will break everything!” Good. It should. It is criminally irresponsible to not know every single application, every workstation, every server on your network. Attackers can exploit one single hole that no one knew about that was allowed for the sake of another IT department because they don’t want to follow cumbersome processes, and now your entire company is completely fucked. Because we can’t impact the server teams. We can’t impact the network team. We really needed that any-any rule. Right?

Block everything. Make other teams defend their needs. Every quarter they need to attest that it is still a need, with their name signed beside every port, every server, every app. So when the breach happens, you know exactly who to file criminal negligence charges against. You know exactly who fucked your company.

There are no more mistakes. There are processes, and they are either followed or they are not. This shit isn’t a joke anymore, this is real life. Block everything. Make security break your company, or the lack of security will end your company.

Excessive firewall denies. Multiple login failures. Excessive firewall accepts. Outbound communication to known malicious sites. You’ve probably got these rules on your SIEM. You’re getting alerts from them and you’re spending your time ruling out false positives and updating the rule when something in your environment changes and you’re happy knowing that if someone tries breaking into your systems, you’re going to catch them trying to guess the domain admin account. You’re going to catch them moving laterally (and failing to move). Attackers stumble around, they make noise that normal users don’t make. Right? Right?

How many offenses do you have in your SIEM? How many alerts? How many emails do you get? And how many of them do you ignore? How long is your “run book”? How long do you spend telling new employees what alerts represent “acceptable risk” or “no further action”? Because I guarantee it’s too long. Too many. Too much. Because those rules are crap.

Let’s talk about maturity levels. I’m a security consultant and we are just as bad as anyone else at this. Far too often we recommend for low-maturity environments to use their SIEM to find policy violations. We might not consciously understand this, but that’s what we’re doing. Any time we write a rule that shows telnet connections, that shows communication to sites on a blacklist, that tattles when a user is sending more than 500mb out of the company’s network, we are codifying policy violations into a security tool. And that’s fine, right? Those are security issues!

But they’re not.

But the security team needs to know about it!

No we don’t. Those are policy violations. If someone is breaking policy, their manager needs to know, not the security team. And transferring a huge file out of the company might be an indicator of compromise, but most often it’s just someone uploading a pirated copy of Zootopia to their Dropbox account. Policy violation. HR issue. Not a security issue. So why does security get alerts on that? Is your million-dollar tool there to tattle on your users or to find security incidents? You want to talk about “next level of maturity”, let’s skip “policy violation” and go straight to a step I like to call actual value.

When’s the last time you responded to a SIEM alert and it was actually a breach? Was actually a security incident? It’s probably been a long time, if ever. So what value are you getting out of your SIEM?

Let’s start with excessive firewall denies and multiple login failures and other serious nonsense like that. They’re seriously nonsense. No one cares. It’s not an indicator of anything but poorly configured systems, which is a really terrible use of the resources of a security department. Why does it take two, three, four smart, trained, full time people to run a SIEM? Because most of it is hunting down non-issues. Most of it is tuning and busy work. Very little of it is responding to security incidents.

Turn.

Those.

Rules.

Off.

Simple as that. They are not indicators of compromise. They are not security incidents. Just do it. Turn them off. They are worthless. They are worse than worthless; they are actively harmful because it uses up the resources of your security team. They hide the actual incidents behind trivial nonsense and force you to spend your days hunting down what service account had its password changed but never updated on one system. If no one has noticed yet, no one cares. Move on. It’s not a security incident. It’s not. You know it’s not.

So what rules should we turn on? You’re not going to like the answer. Because those rules won’t fire. They really won’t. And that’s a good thing! Your SIEM shouldn’t fire! Ever! If you get an offense or an alert or an email from your SIEM you should be screaming! It should keep you up at night! If you’re getting alerts that you skim and move past, your SIEM is working against you, not for you. Turn that noise off. How many alerts per week should you have? NONE.

Instead of multiple login failures, why not a honey account? An account with a tantalizing name like “domain admin” with privileges to nothing? As soon as that account logs in anywhere, you hit the red button and start the klaxon. Because the only person who would ever use that account is an attacker inside your network. If the honey account even fails to log in, even once, that is an indicator of compromise.

Instead of multiple login failures, why not successful logins from different geographical regions? You don’t have to worry about people failing to log in. If someone fails to log in, there is quite literally no compromise. It’s only a compromise if someone succeeds. And then, only if they actually do something bad. Never ever ever is a mere login (or login failure) an IoC. But one account logging in from the US and Russia at the same time? One account connected from the UK and China at the same time? Best case scenario it’s a credential sharing policy violation. Most likely this will never fire unless an account has been stolen.

User profiling is an important tool too. If you don’t have a user profiling tool, get one. Get one that integrates into your SIEM and can fire an alert when someone does something they shouldn’t do. Because multiple login failures is absolutely worthless. But if Bob from accounting is logging into the machine of Alice from sales… THAT is an IoC. That’s a problem.

How about clearing event logs? Every system worth its salt will create an event when its event logs are cleared. Key your rules off that because what’s the first thing an attacker will do to hide their tracks. They will clear the event log that can tattle on them. Many attackers know this and won’t do it, but it’s worth looking for. Because this one will never fire unless you’ve been breached. Or you have lazy admins. But that’s a policy violation right? Make their manager shut that behavior down.

Instead of excessive firewall denies, how about trusting your IDPS? Yes. You should do this. You should log that to your SIEM, but you should also turn off the rule that turns those IDPS alerts into SIEM alerts unless your IDPS only alerts on successful breaches. Too often IDPS alerts on blocked attacks! Why?! You don’t need to tell me you’re doing your job! Shut that noise off! Alert on successful breaches. Yes, your SIEM will go quiet for a long time. Hopefully years. And that’s okay!

So what should the security team do in the meantime? You’re already writing corporate policy. That’s great. You’re reviewing your firewall policies on a regular basis, right? Making sure they make sense and are locked down? If not, do that. You’re reviewing your proxy rules right? All of those whitelist exceptions are still needed? You’re doing formal risk analysis of changes to your environment? Do that. Develop a risk analysis framework, or better yet adopt one. SANS has one. Use it. You’re phishing your employees, right? Do that. As a follow up, you’re formalizing end-user education? Making sure your users are working for you, not against you? Seriously, DO THAT. You’re running vulnerability scans on your entire network? Do that. You’re setting up application whitelisting? Do that! You have database activity monitoring (again with proper rules and no stupid babysitting)? You’re running DR tests and security incident drills and purple-teaming? All of the things you’ve said “I’d love to do that but I have to babysit the SIEM”? Stop babysitting that SIEM! Do your job!

Your manager wants to see value from the SIEM. Implementation was a long process, they’re spending a lot of money for employees to manage it, and let’s be honest, SIEMs are expensive. But the security team has too much real work to do to sit there responding to false positives. Your rules should only alert on actual security incidents. If they’re alerting on anything else, turn those rules off and figure out rules that make sense. Want to show value to your manager? Print off a weekly report of firewall denies and have it sent to your networking team. Send your IAM team a copy of the login failures. You never need to see those reports, you never need an alert, and it doesn’t matter if anyone ever takes action on them. That’s not your responsibility. Your only responsibility is the security of your company, not the behavior of your users.

Stop being a slave to your SIEM and actually get to work doing what you were hired to do: secure your damn company!