How Computer Criminals Defeat Intrusion Detection Systems

Contact:[email protected]

How Computer Criminals Defeat Intrusion Detection Systems
by Carolyn Meinel

We covered "The ABCs of Intrusion Detection Systems (IDSs)" in a previous
column. Now we get to the really, truly, scary part: how computer criminals
defeat IDSs, and how to fight back.

The wily attacker may be able to slip undetected into your network via the
following techniques:


Incomplete IDS coverage
Lost or unknown network elements
An overwhelmed IDS
Excessive false positives (crying "wolf" too many times)
Unicode protocols disguising signatures
Fragmented packets
The 0-day problem
Highly switched networks
Compromise of the IDS itself
An improperly configured IDS
The unique challenges of the middleware environment

Incomplete IDS Coverage
One IDS sitting on top of your connection to the Internet is going to miss
internal attacks. It also may not detect attacks against systems it is not
designed to defend. You probably will need to use products from more than
one vendor, and use both a network IDS (NIDS) and a host-based IDS (HIDS).
There also may be portions of attacks that no IDS can monitor, except by
observing traffic once it enters the more standard portions of your network.
This vulnerability may be a blessing in disguise if it motivates management
to modernize your network.

In the meantime, any decent IDS in a middleware environment will require
products from many vendors. So how do you coordinate the output from these
systems?

"Let me give you the honest answer," offers Marcus J. Ranum of Network
Flight Recorder (NFR). "There's not a lot of cooperation in the security
products industry. Most vendors would rather have their customers use only
one product than support gateways between two products.

"Right now," he continues, "most IDS users that have several systems make
them all e-mail their alerts to one place, and then they process the alerts
together. There is a market evolving for systems to do alert correlation. I
predict that market will grow rapidly soon. NFR's actually looking to break
that model with some very unique stuff. I can't talk about it, sorry, but
pay attention around April."

The Intrusion.com NIDS is expected to soon include the ability to aggregate
input from Netranger, RealSecure and Firewall-1.

And according to the hacker known as Talisker on the Networkintrusion.com
Web site: "The fact that they are from differing vendors is immaterial; they
have such different roles. From experience, I don't like HIDS [hybrid IDS]
and NIDS output on the same console, the only exception being possibly
router output with the NIDS. There is very little correlation between the
two--98 percent of the HIDS traffic is by trusted internal users doing
things they shouldn't. NIDS output is cluttered with false positives, but
when you get a real bite, it's usually a good one."

Lost or Unknown Network Elements
You can't fully protect what you don't know exists. Large, complex systems
often change every day, sometimes in unexpected ways. One company recently
discovered that modem abuse was consuming the equivalent of several T3s of
bandwidth on its PBX. It discovered employees sneaking in external modems to
evade firewall bans on Napster, porn, stock trading and sports sites. (To
learn more about the hazards of modems, see "It's 2 a.m.: Do You Know Where
Your Modems Are?"

Computer criminals also look for orphan computers. A busy sysadmin (and most
are always too busy) may set up a test server and then leave it on the wire.
Once forgotten, it doesn't receive critical security updates.

To make sure your IDS covers everything, regularly scan your network for
computers and modems. Many commercial security scanners will give you a
decent--but not always perfect--view of your network. Be wary of free
network scanners such as nmap, which has been known to crash servers. Their
identification of operating systems is hit-and-miss.

Modem scanning is exceptionally difficult because many users only attach
modems briefly during work hours.

An Overwhelmed IDS
The best IDS architecture won't do much good if it can't keep up with the
load. Vendors argue about how much one another's products can handle. Says
Ron Gula, president of Network Security Wizards: "Solutions like Dragon
Sensor [200 Mb/s] and hardware load balancing, like those switches from
Toplayer, can get customers into the 1Gb range today. We're planning to hit
500 Mb/s by the end of this year? This experience comes from actual paying
customers who deploy Dragon on live networks and not artificial laboratory
testing."

It is safe to assume that if your IDS sensor must sift through more than
half a gigabit per second, it is missing much of what comes through. For
this level of throughput, you need to place IDS sensors on network segments
with lower throughput and aggregate their output. For example, BlackIce
Sentry--Multisegment can simultaneously monitor up to four 100-megabit lines
at once, providing protection for high-speed, switched environments and
making it one of the highest-capacity IDSs on the market.

If you have gigabit Ethernet to the desktop, today you may in big
trouble--unless those high-speed segments are running well below capacity.
And heaven help you if you bought the propaganda and have ATM over fiber to
the desktop.

Says Talisker, "This is where the hybrid solution or even personal firewalls
come into their own." A personal firewall will just look at packets
addressed to the computer on which it is installed.

Excessive False Positives
Even if you have invested in placing enough IDS sensors on sufficiently low
throughput network segments, an attacker may be able to run the CPU
utilization of your IDS console to 100 percent and overfill disk space. Says
John Flowers, chief scientist of Hiverworld, "Many NIDSs today are referred
to as "network false-positive recorders." The problem with false positives
(attacks that pose no threat to your network) is that a determined attacker
can flood an IDS with meaningless signatures, running CPU usage to 100
percent. While the IDS is thus overloaded with what the sysadmin assumes is
a slumber party of 13-year-old "haxors," a successful attack can slip
through.


A common trick is to use nmap from a few dozen tiny Linux machines scattered
across the Internet to scan your network. It has some truly obnoxious
options (recall the "Christmas tree" option) that hog IDS console CPUs.

If you run a Web server, an attacker can set up a script to send it
high-volume junk attacks full of strings such as //../bin/sh/ and so on.
These attacks can be automated in an apparently legal manner. For example,
in 1999 several hackers with high-traffic Web sites agreed to entice their
users to click on links such as the following:

http://www.antionline.com/cgi-bin/phf-is-really-ereet/../
this_is_friendly_greetings_from_ATTRITION.ORG/../giving_you
_the_link_you_deserve/../visit_www.attrition.org/negation/
../pass_us_some_hacker_profiler_$DATA_please/../and_have_a
_nice_day/../how_do_you_like_them_apples_mr_vranesevich?/
../and_it_always_amazes_us_that_the_href_buffer_is_so_big_
because_only_monkey_sites_use_urls_this_long/../phf_php_
search_dig_campus_faxsurvey_wguest_guestbook_anyform_cgitap
_query_cgiwrap_glimpse_lasso_dbadmin_nph-test-cgi_www-sql_
count.cgi_man.sh_info2www_Web.sql_and_textcounter.pl_are_all
_vulnerable_cgi_programs_you_should_be_searching_for/../imagine_
each_click_through_adding_a_full_1k_to_your_logs_this_would_
make_a_fun_Web_harassment_program (snip) ... (the link went on and on)

The ways to overwhelm a NIDS are legion. "Two thoughts here," says Network
Security Wizards' Gula. "If someone has layer 2 access (they can plug into
the hub), then it is 'game over.' They can simulate as many false attacks as
they want to, possibly crashing the NIDS when the hard drive fills up. The
question is, if all these attacks are occurring, can the NIDS handle it? The
other thought is more along the lines of a single packet DOS attack. Certain
types of packets may take more CPU horsepower to process. Choosing these
attacks may cause unexpected performance issues with any NIDS."

A solution to this problem is to configure your IDS to discard bogus attacks
without discarding data you really need. If under nmap assault, for example,
disable reporting of these scans and then reenable reporting later.

Or, if too many failed logins are overwhelming the IDS, set it to not report
failed logins. The problem with this solution, however, is that the more
logins an attacker tries, the more likely he or she is to finally stumble on
one that works.

A danger of being too enthusiastic in eliminating this reporting is that a
storm of bogus attacks may be a warning of serious attacks under way
elsewhere on your network. The solution? Make sure you have someone on your
staff with a deep understanding of computer security. This person should
exercise judgment and have the skills to quickly reconfigure your IDS
systems to deal with crises.

"We do a lot to reduce false positives in our system," says NFR's Ranum.
"Unlike traditional systems that just match a simple string pattern, our
NIDS does full-session state monitoring of applications so it's not just a
matter of telneting to port 23 and typing 'root' if you want to generate an
alarm."

Another solution is to use a target-based IDS (TIDS). Says Hiverworld's
Flowers: "The most notable difference between our TIDS and normal NIDS is
that we can maintain a detailed network picture of the customer network,
including host operating systems and vulnerabilities. Next, we use this
information to our advantage by not processing any packets that are destined
to a host that either (a) doesn't exist or (b) doesn't have any open ports."
Unfortunately, Hiverworld's product is not yet on the market.

Others warn that some attackers use an "everything but the kitchen sink"
approach, blindly throwing one attack after another at the victim network.
This resembles the theory that enough monkeys on enough typewriters will
eventually write a Shakespearean sonnet. This attack philosophy may
succeed--given enough time. By discarding "irrelevant" data, TIDS may throw
away an early warning of a concerted attack.

"You may even want to have both systems [target-based and a standard network
IDS] for various reasons," Flowers says.

Will these measures entirely solve the problem of false positives? "The fact
is that someone who understands the sensor's logic will always be able to
generate a fake attack that might raise a false positive," Ranum says.

Unicode Protocols Disguising Signatures
Many IDSs miss attacks simply because they were coded slightly differently
than the signatures in their databases. A root cause of this is the Unicode
problem, which Bruce Schneier explains in the July 15, 2000, issue of his
Crypto-Gram newsletter:

"Unicode is an international character set. Like ASCII, it provides a
standard correspondence between the binary numbers that computers understand
and the letters, digits and punctuation that people understand. But, unlike
ASCII, it seeks to provide a code for every character in every language in
the world. To do this requires more than 256 characters, the 8-bit ASCII
character set; default Unicode uses 16-bit characters, and there are rules
to extend even that."

Schneier sees ways to exploit Unicode to evade an IDS:


Start attaching semantics to the new characters as delimiters, white space,
etc. With thousands of existing characters and new characters being added
all the time, it will be extremely difficult to categorize all the possible
characters consistently. And where there is inconsistency, there tend to be
security holes.
Somebody uses "modifier" characters in an unexpected way.
Somebody uses UTF-8 or UTF-16 to encode a conventional character in a novel
way to bypass validation checks.

A major hazard of Unicode is to Web servers. Many attacks on Web servers
consist of simply inputting a URL into the location window of a browser (or,
directly and with more power to insert malicious code, via a telnet
connection to port 80 and the GET command). By using one of the several
possible Unicode variations of these many attacks, it may evade both a
firewall and an IDS. Some Web servers either do not implement Unicode or
allow it to be turned off. The Microsoft IIS server, however, currently has
no way to disable Unicode.

"Eric Hacker," writing for Security Focus, offers a couple of solutions to
this problem: Don't use IIS Web servers until Microsoft allows Unicode to be
disabled, and use a host-based IDS that watches the internal logs rather
than merely observing incoming packets.

True, use of a host-based IDS is in some cases like alerting you after the
barn door is open and the horse has gone gallivanting. But at least you'll
know about the barn door.

Fragmented Packets
If you are up against the A-Team, they may try to hide attacks by sending
them in fragmented and/or out-of-order packets that an element of your
network may obligingly reassemble once past your IDS. It doesn't matter if
your IDS handles fragmented packets if you give up when traffic is out of
order.

"Of course this is not a problem for our [Dragon] host-based IDS product,"
says Gula. "And as far as I am aware of, all commercial NIDSs will correctly
reconstruct fragmented traffic. The issue is, if traffic is maliciously
fragmented, then can the NIDS make sense out of it? The other question is,
will the NIDS tell someone that malicious/suspicious fragments have been
sent on the network?"

Says Mark Teicher of Network Ice: "Actually, one thing that can confuse most
IDS systems, especially during the BackTrace (or Source IP resolution), is
using scripts like targa. Targa is a multiplatform DoS attack which
integrates bonk, jolt, land, nestea, netear, syndrop, teardrop and winnuke
all into one exploit. It allows the user to generate IP-based attacks with
the various options: invalid fragmentation, invalid protocols, invalid
packet sizes, invalid header values, invalid options, tcp offsetsinvalid top
segments, routing flags, etc."

The 0-Day Problem
Until an attack has been publicly revealed, it is known in hacker slang as a
"0-day" exploit and might not be detected by your IDS. "No problem in some
cases, big problem in others," says Gula. "For example, almost any buffer
overflow can be detected with FTP, SMTP or HTTP by simply looking for
non-ASCII traffic traveling to those ports. Unfortunately, these events
still require a smart person to analyze."

Highly Switched Networks
Switched networks are the latest and greatest way to defeat bad actors
inside your LAN. Some networks are so highly switched that each computer,
even if running a promiscuous network interface, will only see packets
addressed to it. The upside is that if an intruder should get inside your
network, he or she will not be able to sniff anything worthwhile. The
downside is that in this environment, a network IDS will reveal little.

The solution? "There is much talk of NIDS failing in a switched
environment," says Talisker. "I disagree with this? By delegating NIDS down
to the end host, many of the problems higher in the network architecture are
covered. The USAF has just bought 500,000 copies of Tiny Personal Firewall
for this purpose."

"Many switched allow port mirroring or link mode, which will mirror the data
from all ports to a single port where the NIDS will be," he says. However,
mirror ports do not work well for full-duplex connections--one of the
primary uses of fast Ethernet links. Talisker points to Shomiti taps as a
solution to the port-mirroring problem.

Says Gula: "Host-based IDS works here real well. So does firewall
correlation... We have not seen a 'highly switched network' we could not
monitor."

Compromise of the IDS Itself
Who watches the watchmen? Who detects a criminal who has seized control of
the IDS itself? "Tough problem," Gula says. "All NIDSs can use a 'stealth'
interface, but many folks don't deploy them that way. With an IP stack, many
NIDSs also have remote management ports. We don't have those with Dragon,
but we do use SSH (and encrypted connection) to establish a remote TCP
session."

The Intrusion.com NIDS uses a connection between its sensors and console
that doesn't even use MAC addresses. Unless an attacker is extremely good at
compromising another host on the wire and reconfiguring its NIC, an attacker
won't even see this NIDS.

"With our sensors," says NFR's Ranum, "they can operate in software stealth
mode, in which they are invisible on the network through software means
(patch out the parts of the IP stack that talk out that interface), or
hardware mode, in which a special cable with a clipped transmit lead is
used. Obviously, if you're invisible on the network, you have to have a
duplicate interface on an alternate network for out-of-band management."

How does he keep attackers from disabling the consoles used to evaluate the
sensor data?

"All communications into our NID require a valid encryption key or they're
discarded," Ranum says. "So spoofing traffic will just result in the NID
throwing the traffic away. Since we're using secret key, not public key on
the encryption, you can't do a PKI resource exhaustion attack (PK uses too
much CPU). For our consoles, we recommend that the user have some kind of
personal firewall installed on the console (for example, ZoneAlarm or
whatever)."

An Improperly Configured IDS
No matter how good your IDS, it won't help you if it's not configured
properly. Flowers, speaking at Def Con 8 last July in Las Vegas, recalls: "A
client installed a leading IDS, put in 10 sensors with load balancing? About
2 days after, it was sending 3,000 pages per minute. After 48 to 72 hours,
they shut it down. They got $16K for 3 days' pager bill. I didn't realize
they had systems that tested the pager infrastructure."

It's important to decide what kind of attacks are important enough to
immediately alert the responsible sysadmin. According to Flowers, it should
take several weeks of concerted effort to configure a new IDS so that it
only pages the sysadmin at 2 a.m. for a darn good reason.

The Unique Challenges of the Middleware Environment
For a middleware environment, Gula says, "custom signatures for both network
and log analysis are the answer here. Many times these protocols have clear
text in them and have excellent logging for debugging purposes. HIDS can
read those logs fairly easy. Using a NIDS like Dragon or NFR to read the
clear text stuff can work too, but some analysis is needed upfront by
security folks.

Ranum warns about using public key encryption to guard your transactions:
"Public key crypto is great stuff but is very CPU-intensive because it does
a lot of operations on large numbers. One nasty trick you can play on
PKI-based systems is to get them to bog themselves down signing lots of
stuff or checking signatures. Most designers of crypto protocols are careful
to design around this kind of attack, but there are some cases when it's
unavoidable (for example, e-commerce) because public key is so ingrained in
the system."

Gula advocates a system solution to IDS weaknesses on the sorts of complex
networks you must manage. "With the proper firewall rules and configuration
of the NID, it gets much more difficult," he says. "Adding in host-based
IDS, a honeypot and some firewall correlation raises the bar even more."

"There are many more efficient ways to detect port scans," he continues.
"Things like virtual honeypots, looking for odd TCP flag combinations and
looking for unique ICMP messages (port unreachable and admin prohibited
filters) can identify all sorts of fast, slow and random network scans, even
from many multiple IP addresses."

And Talisker adds: "It may be worth mentioning using an IDS to reconfigure
routers and firewalls to shun attackers, though I would only include it if
you can explain the risks of spoofed IP addresses. I have used one product
that will only shun for certain attacks at certain times and never shun
certain addresses--which reduces the risk significantly. And it may be worth
mentioning that honeypots can alarm when interfered with and that as long as
they are within your borders, they cannot be accused of entrapment (IMHO)."

How to Test Your IDS
Following are some Web sites that will help you test your IDS:


You can test your IDS against targa by obtaining programs from
www.mixter.warrior2k.com.
Robert Graham, CTO of Network Ice, has developed a tool to test IDSs:
"Sidestep" (www.robertgraham.com/tmp/sidestep.html).
For many other exploit programs that will enable you to test your IDS,
source code is available from the Packetstorm site
(www.packetstorm.securify.com).
A less intensive but more user-friendly testing technique is nidsbench, a
network intrusion detection system test suite available from
www.anzen.com/research/nidsbench/. Nessus (www.nessus.org) is also well
regarded as an automated testing program for firewalls and IDS.
One stumbling block to testing hacker exploits is that they often are hard
to compile and run. Tech support and documentation is typically
nonexistent--nmap is a welcome exception. Your best bet is to compile them
on a Linux server. Debian (www.debian.com) is reputed to be the most
compatible with the most exploits. SuSE Linux (www.suse.com) is also
excellent and easier to install. Help on how to install and run hacker
exploits can also be found in this author's book �berhacker!
Conclusion
So how do you keep criminals from defeating your intrusion detection
systems? "Automate understanding your network and its changes with
technology, not people," says Flowers. "The ultimate IDS also determines the
vulnerability of the target." He says Hiverworld will be ready to ship its
"True 100 IDS" as part of its IP 360 security service before the end of the
first quarter of this year.

Ranum, writing in the Computer Security Journal, Vol. XIV #4, agrees. "The
ultimate IDS would not only identify an attack, it would assess the target's
vulnerability. If the target is vulnerable, it would notify the
administrator. And if the vulnerability has a known fix, it would include
directions for applying the fix. This would require huge, detailed
knowledge." Keep an eye out for his newest product, due to be unveiled
around April. How close will it approach his ideal?

Unfortunately, the ultimate IDS as described by Flowers and Ranum isn't on
the market today. For now, keeping criminals out of your network will
continue to be a duel between them and your sysadmins and security people.


Back to the Index