IT professionals still have in fresh memory the failure of CrowStrike's Falcon endpoint protection product which caused an US$10 billion global IT breakdown. The incident has been extensively covered in news outlets, on YouTube channels, and in blog posts – and rightly so.
However, no-one seems to question the idea of having such software installed on servers in the first place. That has long been a mystery to me.
Lack of independent justification for AV
After seven years, the following well-written ITPro article is still relevant: Does antivirus software do more harm than good? In the article, several sources describe how antivirus (AV) software itself has security holes, and that AV may interfere with other security measures. The problem with AV software is that it often runs in close proximity to the guts of the underlying system, so vulnerabilities in AV software (which are not uncommon) can have extraordinarily nasty security implications, as well as stability implications, see CrowdStrike.
In other sectors like research and medicine, it's considered natural to ask "Is there any independent evidence supporting the net gain from spending time and money on X?". Even though the IT world spends huge amounts of resources on AV, I've yet to see studies which provide such evidence. Sure, there are reports demonstrating that a given product detects malware XYZ (though not necessarily malware ZYX). But where's a study which documents how that detection makes a system more secure? Bear in mind that a file with malware might not get a chance to infect, because it relies on specific user interaction which is unlikely to happen, or it may rely on operating system vulnerabilities which have been patched long ago. On the other hand, when the AV product sometimes introduces gaping security holes, then the overall gain may well be negative.
Down-to-earth evidence of AV's questionable role
The above may appear overly academic, so here are some down-to-earth observations:
- Billions of Internet-connected devices run without AV software, and it generally works very well. Phone operating systems get software through well-understood, curated channels, and they have built-in robustness such as sandboxing and ASLR. So even though the smartphone ecosystem mainly consists of happy-go-lucky non-technical end-users, we have seen very little (if any) akin to an ILOVEYOU pandemic, and we see very little ransomware on phones.
- I did not have to clean up after the CrowStrike incident. But in my professional life, I've lately had to deal with detrimental effects from another AV product: That AV product had suddenly decided to "eat" a DLL file from a perfectly valid piece of software which had been installed weeks before. As a consequence, a rather important service stopped working. Paradoxically, the impacted service was helping us conduct safe IT practices. It took time and money to get things back running, and it has happened several times. I'm sure many of you have also seen AV software constitute a source of arbitrary and unpredictable outage.
AV at odds with pillars of secure IT practice
Let's step back and review decade-old basic system administration principles, and how they are often broken by AV:
- Install as little software as possible in order to limit attack surfaces – especially minimize software which runs with high privileges. AV breaks this principle, because it's perfectly possible to run a computer without an AV add-on (which millions of Macs and Linux servers, and probably also some Windows servers, do).
- Keep your software up-to-date in order to close the holes which malware will try to exploit. I've seen several cases where AV would conflict with software updates from Microsoft, making this principle harder to follow.
- Run software with the lowest possible privileges: All non-trivial software has bugs, so processes should be constrained as much as possible. I claim that many AV products run with excessively high privileges and/or in kernel space, breaking the principle. See this article about DLL search order hijacking for extra suspense.
- Make software hard to infect, primarily by disallowing software to write to its own binaries, but also by using techniques like address space layout randomization (ASLR). The article from 2017 quotes a software developer who spent lots of time fighting tricks performed by AV which could conflict with the product's built-in robustness.
- Prefer to have software be delivered via well known channels, such as yum repositories, app stores, etc. Many AV products introduce proprietary update services which see little scrutiny and may even decide to replace itself with another product, see Kaspersky's recent surprise act. On top of that, CrowdStrike's update system is hardly the only one which made it impossible to roll out in non-production, before applying updates to production.
- Stay away from software which shows signs of sloppy software engineering. Kudos to AV vendors for publishing CVE reports. But less so for what seems to be sloppiness: CrowdStrike has huge financial ballast, yet it failed to conduct basic testing steps, and it was not the first time in recent history. Other AV products also have questionable track records, but I've yet to hear such aspects being part of IT departments' decision making.
- Absolutely remove abandoned or end-of-life software. Yet, AV software is often rather hard to remove, causing friction when trying to uphold the principle. I've seen many servers running AV software which was no longer being updated, after a vendor change long ago (in one case, because removal required down-time where the server had to be booted into safe mode, before the abandoned product could be removed).
- Spend time wisely, because there is never time for it all. Time spent administering/maintaining AV software and battling AV-derived problems may result in less time spent on security enhancing activities like log analysis, systems/code cleanup, restore tests, AD hardening, etc.
I've been in a meeting where a server break-in was discussed. The server had known vulnerabilities which would take time to address. Someone proposed installing AV on the server, buying time. If the decision for AV had been chosen, I'm rather sure it would have stolen much needed attention on getting the root cause handled, because it would have added a false sense of security.
On top of that, AV software is often seen incurring significant system overhead, adding latency and affecting end-user productivity. System overhead also requires extra CPU and RAM, which goes against green IT ambitions.
Culture and profit
Given all the technical arguments against AV, we may instead be dealing with a cultural issue: What is considered a given in one community is foreign to another. You will probably agree that in some parts of the IT landscape, it would be surprising to run across AV. For example:
- Internet routers don't run AV, even though they are directly exposed to all sorts of traffic.
- I claim most VMware administrators would strongly object, if someone requested AV be run directly on hypervisors.
- Printers don't run AV (and are often not patched, but that's a separate, sad story).
- Millions of compute nodes in high-performance compute environments deliver gazillions of compute hours, typically without having AV interfere.
The list goes on. And fortunately, many Linux and Mac hosts are still allowed to run without AV, although many overly-general IT policies are starting to force AV into those environments.
AV got started in the old days when Microsoft had not yet introduced proper kernel/userland and administrator/user separation into Windows, and where organizations had no good way to curb users from installing software from arbitrary sources. Fast-forward to 2024 where Windows systems can be very robust, but AV thinking prevails -- why? Inertia is not foreign to the IT business, of course: "Nobody Gets Fired For Buying IBMAV". Let me propose an additional reason: Some software vendors and certain security consultants have found AV to be a very profitable business, and they would naturally hate to see profits go down.
Time to start questioning
AV uselessness may appear to be a controversial proposition, and many will disagree with me. On the other hand, I also know of many system administrators and programmers who agree.
IT professionals, can we at least agree that it's our responsibility to start questioning AV as a general requirement? The compliance crowd will not have the guts to do it. So-called IT security consultants often profit from antivirus products, so they don't want to rock the boat.
It's on IT professionals' plate to address this.
PS 1
There may be situations where AV makes good sense to have. I argue that AV can make sense to have
- on mail gateways, as part of general anti-junkmail efforts
- on file shares which are shared by many users, depending on the environment
- on hosts which – for some reason – can neither be regularly patched, nor brought onto a segregated network (but don't expect it to be very effective)
PS 2
While I generally object to the notion of buying security as an add-on product, I acknowledge that some products can provide real security. A product like Nessus' vulnerability scanner, for example, is a good addition to an IT organization's arsenal, especially because it can run in an unintrusive manner which neither installs anything on a server, nor tries to intercept all network traffic.
