By Tony Lee
Introduction
Disclaimers: This is not a troll. :) Scans were run as root (or sudo) using Nmap version 6.25.
Few
would dispute that the de facto standard scanning tool is Gordon Lyon’s
(aka Fyodor) Nmap. However, if we asked you to thoroughly and
efficiently scan a remote class C (192.168.1.1/24) network, which of the
following would you run?
- nmap 192.168.1.0/24
- nmap -sS 192.168.1.0/24
- nmap -sS 192.168.1.0-255
- nmap -sS -Pn -p 0-65535 192.168.1.0/24
The
answer we hear most often is option a. While this may work for small
networks, it does not scale for larger networks or more thorough
assessments. The astute reader will notice that options a, b, and c,
operate identically. Option a provides the network range in CIDR
notation and since -sS is the default scan type when no options are
supplied--option b is identical to option a. Examining option c,
reveals that it is the same as options a and b, except that the target
is supplied using a network range instead of CIDR notation. The problem
with options a, b, and c is that they will not thoroughly scan the
remote class c network as they will only scan the top 1000 TCP ports.
Option d is close to what we are looking for since it scans all of the
TCP ports; however, it lacks efficiency since we will be scanning all
ports on all hosts, including dead IP space.
The
most important thing to keep in mind is that in this simple example, we
are only asking to scan a class C network. However as security
practitioners, we are often faced with multiple class C’s, class B’s or
even in some cases, class A networks. We will take the rest of this
article to outline a scan strategy for networks of all sizes and some
tips and tricks for carving the output.
Outline
- Host Discovery
- -sn “Ping Scan”
- Throttling Pro-tips
- Generate Live Hosts List
- Port Discovery
- Most Common Ports
- Full Port Scans
- Putting it all Together
- Service and Operating System
- Nmap Scripting Engine (NSE)
- Manipulating Output
- Carving Standard Output
- Convert to HTML
- Processing Script
- Summary
Host Discovery
A
methodical approach to scanning usually involves performing host
discovery first, which eliminates the problem we saw with the above
option d--scanning copious amounts of dead IP space. Fortunately, Nmap
is quite intelligent and flexible for performing host discovery scans.
The common host discovery scan options are listed below:
-sn: Ping Scan - disable port scan
-PS/PA/PU/PY[portlist]: TCP SYN/ACK, UDP or SCTP discovery to given ports
-PE/PP/PM: ICMP echo, timestamp, and netmask request discovery probes
-PO[protocol list]: IP Protocol Ping
|
The option we will focus on in this article is the most popular and the default for host discovery (-sn “Ping Scan”).
-sn Ping Scan
One
thing to note about the -sn “Ping Scan” option (formerly known as -sP)
is that it does much more than just a traditional ICMP echo request. By
default against a remote network the “Ping scan” consists of the
following packets:
- ICMP echo request
- TCP SYN to 443
- TCP ACK to 80
- ICMP timestamp request
Figure 1: Packet capture confirms behavior of a remote "Ping Scan"
When
scanning a local network, Nmap automatically switches to using ARP
requests. This is desirable intelligence because fewer packets are sent
and it is generally more accurate. A system that blocks all ICMP
traffic and filters TCP ports 80 and 443 would be invisible to the four
remote discovery packets sent above. However, hosts typically respond
to ARP requests because layer 2 addresses are necessary to construct
packet headers.
Figure 2: Packet capture confirms behavior of a local "Ping Scan"
For
the first packet capture, the scanning host was placed on a different
subnet than that of the targets--this caused Nmap to use the 4 discovery
packets shown in Figure 1. For the second packet capture, the scanning
host was placed in the same broadcast range and thus ARP was utilized
for discovery as shown in Figure 2. Both of the packet captures were
created with the same discovery scan shown below.
root@DVORAK:~/scans# nmap -sn -T4 -oA Discovery 192.168.1.0/24 Starting Nmap 6.25 ( http://nmap.org ) at 2013-05-13 13:21 EDT
Nmap scan report for Wireless_Broadband_Router.home (192.168.1.1)
Host is up (0.0023s latency).
MAC Address: 00:26:62:XX:XX:XX (Actiontec Electronics)
Nmap scan report for 192.168.1.2
Host is up.
Nmap scan report for android.home (192.168.1.3)
Host is up (0.025s latency).
MAC Address: CC:3A:61:XX:XX:XX (Unknown)
Nmap scan report for computer.home (192.168.1.147)
Host is up (0.00035s latency).
MAC Address: 24:77:03:XX:XX:XX (Intel Corporate)
Nmap done: 256 IP addresses (4 hosts up) scanned in 4.45 seconds
|
Options explained:
-sn = “Ping scan”
-T4 = Throttle to aggressive
-oA <basename>= Output in all three formats (normal, XML, greppable)
Throttling Pro-tips:
Aggressive
(-T4) throttling is substantially faster than the default Normal (-T3)
throttle. We have rarely (if ever) seen Aggressive scanning crash a
host or flood a network. This should be what you start with unless you
know of particular hosts that are sensitive to scanning. With that
said, if hosts are known to crash on simple scans or become easily
flooded, avoid scanning them with -T4 and possibly try Polite (-T2)
throttling. Please realize that -T2 may be up to 10 times slower than
-T3, so be patient and only run it on your most sensitive hosts--not
hundreds at a time. More importantly, avoid any one-off scans such as
version scanning as these are more likely to crash hosts than the speed
of the scan. Some older SCADA components are known to fall over from
simple port scanning, not to mention version scanning. In general, we
do not recommend Insane mode (-T5) as this can negatively affect
accuracy. Lastly, we only recommend -T0 or -T1 when trying to be extra
stealthy (IDS evasion) and only for scanning a few ports on a few hosts
because it will likely be too slow for anything else.
Generate Live Hosts List
Nmap,
like other tools, accepts a file containing a list of IP addresses as
input. Because the remaining scans are more intensive than the
discovery scans, we will want to only feed live hosts to Nmap. We can
use the output files from our discovery scan to generate this list of
live hosts. Three files are generated as a result of the -oA option:
root@DVORAK:~/scans# ls
Discovery.gnmap Discovery.nmap Discovery.xml
|
Files explained:
.nmap = Normal output (what is printed to the screen)
.gnmap = Greppable output
.xml = XML output
We will extract the live hosts from the .gnmap (greppable) file using a command such as the following:
root@DVORAK:~/scans# grep "Status: Up" Discovery.gnmap | cut -f 2 -d ' ' > LiveHosts.txt
root@DVORAK:~/scans# cat LiveHosts.txt
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.147
|
Cut options explained:
-f = field number (in this case, field 2)
-d = delimiter (in this case, a space)
The resulting LiveHosts.txt file is one IP address per line, which can be supplied as input to Nmap and other tools as well.
Port Discovery
Now
that we have generated our LiveHosts.txt file, we can start port
discovery on these hosts. If you have many live hosts to scan and only a
short amount of time, you may want to break it up into two sets of
scans:
- Most common ports
- Full port scans
Most Common Ports
Fortunately,
Fyodor performed a substantial amount of research for his Top Ports
Project which he presented in 2008 at BlackHat and Defcon. He performed
scans on millions of Internet IPs and incorporated internal scans from
large organizations to determine the most commonly open TCP and UDP
ports. The key take-away from this research is that Fyodor made Nmap
default port scans more efficient by targeting the top 1000 TCP and top
1000 UDP ports when no port options are provided. But just how
effective is scanning only the most common ports? Here is an
eye-opening table that summarizes the results of Fyodor’s research:
Table 1: Summary of Fyodor's research
TCP
|
UDP
|
topports 10: 48%
|
topports 10: 50%
|
topports 50: 65%
|
topports 50: 86%
|
topports 100: 73%
|
-topports 100: 90%
|
topports 250: 83%
|
-topports 250: 94%
|
topports 500: 89%
|
topports 500: 97%
|
topports 1000: 93%
|
topports 1017: ~100%
|
topports 2000: 96%
|
|
topports 3674: ~100%
|
As
shown in the table above, according to Fyodor’s research, scanning the
top 1000 TCP ports is 93% effective in identifying all open TCP ports,
while scanning the top 3674 TCP ports is almost 100% effective.
Additionally, scanning the top 1017 UDP ports is nearly 100% effective
at identifying all open UDP ports. Since the default scan is the top
1000 TCP and UDP ports, no port options are needed below. (If scanning
the top 1000 ports is not desirable, use the --top-ports option and specify a new number of ports to scan.)
Most common port scans:
nmap -sS -T4 -Pn -oA TopTCP -iL LiveHosts.txt
nmap -sU -T4 -Pn -oA TopUDP -iL LiveHosts.txt
|
New Options explained:
-sS
= SYN Scan (half-open scanning) - Does not complete the 3-way handshake
so it is faster than -sT (default scan - specifying is optional)
-sU = UDP Scan
-Pn = Do not perform discovery scan (we already know the hosts are alive)
-iL = Input from a file
Full Port Scans
Each
host can have a maximum of 131,072 ports (that is 2^16 = 65536, then
multiply by 2 for TCP and UDP). But why would anyone need to scan all
of those ports? Why isn’t the top 1000 good enough--especially after we
highlighted how effective it was? One surprising port that is not
contained in the top 1000 is TCP/2433 (the alternative default MS-SQL
port, which can be a misconfigured Achilles heel for many networks).
Thus, skipping the full port scan can miss critical services that may
be the entry point for an attacker or a piece of malware.
Assuming
that full port scans are desired, these can be executed when the most
common port scans finish. Full port scans will take substantially
longer to finish than the common port scans. Thus, it is ideal to have
the results of the common port scans as something to review while the
full port scans are running.
Note: Full port UDP scanning is VERY slow, and for extremely large networks may never finish in a reasonable amount of time.
Full port scans:
nmap -sS -T4 -Pn -p 0-65535 -oA FullTCP -iL LiveHosts.txt
nmap -sU -T4 -Pn -p 0-65535 -oA FullUDP -iL LiveHosts.txt
|
New Options explained:
-p <ports> = port list
Putting it all together
So what does the scanning process look like when combined?
Discovery:
nmap -sn -T4 -oA Discovery 192.168.1.0/24
Generate Live Hosts List:
grep "Status: Up" Discovery.gnmap | cut -f 2 -d ' ' > LiveHosts.txt
Common Ports:
nmap -sS -T4 -Pn -oA TopTCP -iL LiveHosts.txt
nmap -sU -T4 -Pn -oA TopUDP -iL LiveHosts.txt
Full Ports:
nmap -sS -T4 -Pn -p 0-65535 -oA FullTCP -iL LiveHosts.txt
nmap -sU -T4 -Pn -p 0-65535 -oA FullUDP -iL LiveHosts.txt
|
Following
this process should perform efficient host discovery, live host file
generation, and full port scans for both TCP and UDP (along with the top
1000 ports, in case the full port scan does not finish in time).
Service and Operating System
After
reviewing the common port scans (or full port scans if available), a
few hosts of concern may emerge. If this is the case, a more intensive
scan may be desired such as a service and/or operating system scan.
These more intensive scans should not be used in discovery on a large
network because it may take too long to finish. If you have many live
hosts, only run these scans on select hosts where more information is
desired.
Service (Version) Scan
The
ports that were discovered in prior scans were quickly labeled with
associated services based on the nmap-services database of about 2,200
well-known services. The example scan below shows ports 22, 23, and 80
all labeled with the expected services ssh, telnet, and http
respectively.
Example SYN scan:
nmap -sS -T4 -PN -n 192.168.1.2
PORT STATE SERVICE
22/tcp open ssh
23/tcp open telnet
80/tcp open http
|
However,
since applications/services can run on any arbitrary port, additional
probing may be desired to ensure that the service matches the associated
port. For this additional probing, version detection (-sV) is useful.
Example version scan:
nmap -sV -T4 -PN -n 192.168.1.2
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 5.3p1 Debian 3ubuntu7 (Ubuntu Linux; protocol 2.0)
23/tcp open telnet Linux telnetd
80/tcp open ssh OpenSSH 5.3p1 Debian 3ubuntu7 (Ubuntu Linux; protocol 2.0)
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel
|
Notice
that while the SYN scan (-sS) labeled port 80/tcp as http (per the
nmap-services information), a deeper version scan (-sV) revealed that
port 80 was really another instance of SSH.
Figure
3: The screenshot above shows a SYN scan and a Version scan. The
version scan revealed that port 80 is actually SSH. The version scan
also correctly guessed the version of SSH.
Version scan syntax:
nmap -sV -T4 -Pn -oA <host>-service.txt <host>
|
New Options explained:
-sV = Service Version Scan
As
you can see, version scanning is quite useful, but as mentioned
earlier, it comes at a cost. It requires more packets and thus takes
longer to scan. Notice in figure 3 that the SYN scan finished in 0.08
seconds, while it took the version scan 10.33 seconds. Multiply the
difference by thousands of machines and it adds up.
Operating System Scan
For
very large networks, the operating system for each scanned host is not
always known. There are two quick checks that almost always work for
high level operating system discovery (Windows vs. Linux)--especially
when the two techniques are combined:
- TTL - Time To Live value (quickly obtained via ping, be sure to subtract the number of hops)
- Windows will generally have a TTL of 128
- Linux will generally have a TTL of 64
- Network devices will generally have a TTL of 255
- RPC (port mapper) port
- Windows hosts will have TCP 135 (msrpc) open
- Linux hosts will have TCP 111 (rpcbind) open
However,
if you need more detail, such as the specific version of Windows or
Linux, this is where Nmap’s operating system scan comes in handy. Nmap
uses TCP/IP stack fingerprinting and an nmap-os-db of more than 2,600
known OS fingerprints in order to accurately identify the operating
system. In most cases it is in the ballpark and if it does list a few
operating systems, the correct OS is usually in the list.
nmap -O -T4 -Pn -oA <host>-service.txt <host>
|
New Options explained:
-O = Operating System Scan
Caution:
Add the necessary ports using the -p option if the ports you want to
use for operating system scans are not in the top 1000 ports. Also be
sure to include at least one open and one closed port to achieve a more
accurate operating system guess.
Figure 4: Comparison between SYN scan and operating system scan.
Some
may want to use both -sV and -O at the same time, which is fine. There
is even a shortcut option for this discussed in the next section;
however, we will provide a word of warning there as well.
NSE
Prior
to the Nmap Scripting Engine (NSE), Nmap had very limited vulnerability
scanning capabilities. However, with the addition of NSE scripts, it
can now seek out (and in some cases exploit) vulnerabilities. This
increases the flexibility and extensibility of Nmap by allowing users to
write their own scripts with just a little bit of LUA scripting
knowledge.
At
the time of this writing, there are 433 NSE scripts provided as part of
the Nmap download. To view the scripts that are available, list the
directories that contain the NSE scripts.
On Windows hosts the scripts are typically in the following location:
c:\Program Files (x86)\Nmap\scripts\
On Linux hosts the scripts are typically in the following location:
/usr/local/share/nmap/scripts/
You can also check this reference for a listing and description: http://nmap.org/nsedoc/
Since
these are interpreted (and not compiled) scripts, it is possible to
view the source to better understand what they do and even tweak them to
behave differently. It is also quite helpful to start from one of
these functioning scripts if LUA is not your forte.
To activate Nmap NSE scripts, use the --script option and supply comma separated script names, categories, or directories:
--script filename|category|directory|expression[,...]
Since
NSE was designed for a variety purposes, including extended version
scanning, vulnerability detection and even exploitation, all scripts
have one or more categories defined. This not only adds organization,
but it can simplify the choices when running multiple scripts by being
able to select a family of scripts based on their categories.
Category
options are: auth, broadcast, brute, default, discovery, dos, exploit,
external, fuzzer, intrusive, malware, safe, version, and vuln.
In
addition to running scripts based on category, there are a couple
really nice shortcuts, but they may not be the best choice in all
circumstances:
- -sC = Script Scanning: Equivalent to --script=default and will execute scripts which have the default category defined
- -A = Aggressive scanning: Performs the following: Operating System (-O), Version (-sV), Script scanning (-sC) and traceroute (--traceroute).
The
reason why caution should be applied when running either of these
options is that default scripts can include scripts in the intrusive
category, which have the potential to crash systems or make printers go
crazy.
Instead,
it may be desirable to exercise more control over which NSE scripts are
executed by using the boolean AND with the “safe” category. For
example:
nmap -sS -T4 -Pn --script “default and safe” <host>
nmap -sS -T4 -Pn --script “http-* and safe” <host>
nmap -sS -T4 -Pn --script “smb-* and safe” <host>
nmap -sS -T4 -Pn --script “safe” <host>
|
Figure 5: Results of an Nmap script scan against a Windows 8 host
For more information, a great resource is Fyodor’s NSE usage page:
Manipulating Output
As
mentioned earlier, the output options for Nmap are normal (.nmap),
greppable (.gnmap), and XML (.xml). We will now explore how we can
manipulate these standard output formats to achieve even greater
insight--which is particularly useful when viewing larger output files.
Carving Standard Output
The
standard output files can be quite useful for generating statistics or
feeding other tools. The nmap and gnmap files are good for pulling
information such as:
- Unique open ports
- Number of unique open ports
- Open ports to feed into other tools
- Hosts that are listening on a particular port
- Number of hosts listening on a particular port
Unique open ports:
grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu
22/tcp
23/tcp
80/tcp
135/tcp
--snip--
|
Number of unique open ports:
grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu | wc -l
16
|
Feed open ports into another tool (space separated):
grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu | cut -f 1 -d '/' | xargs
22 23 80 135 139 443 445 902 912 992 4445 4567 5357 8080 8443 49155
|
Feed open ports into another tool (comma separated):
grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu | cut -f 1 -d '/' | xargs | sed 's/ /,/g'
22,23,80,135,139,443,445,902,912,992,4445,4567,5357,8080,8443,49155
|
Hosts that are listening on a particular port (telnet used below as an example):
grep 23/open/tcp FullTCP.gnmap | cut -f 2 -d ' '
192.168.1.1
192.168.1.2
|
(Replace port 23 with whatever port number you wish to find other ports/hosts of interest)
How many hosts are listening on a particular port (telnet used below as an example):
grep 23/open/tcp FullTCP.gnmap | cut -f 2 -d ' ' | wc -l
2
|
(Replace port 23 with whatever port number you wish to find other ports/hosts of interest)
Convert to HTML
We
just illustrated the usefulness of normal and greppable output files,
however XML files are also useful. Nmap XML files can be converted into
HTML format for easy viewing within a web browser. This is possible
through using Extensible Stylesheet Language Transformations (XSLT).
Detailed information on various options can be found here: http://nmap.org/book/output-formats-output-to-html.html
One very reliable method is using the xsltproc tool (from http://xmlsoft.org/XSLT/). Using it is simple and the result is portable. Use the Nmap XML file for input and use -o to specify the output file.
Syntax is below:
root@DVORAK:~/scans# xsltproc TopTCP.xml -o TopTCP.html
root@DVORAK:~/scans# firefox TopTCP.html
|
Result:
Figure 6: HTML result of using xsltproc
Notice
that the output is very organized and port information is even
displayed in easy to read tables. Navigation is made convenient through
the use of the included hyperlinks and search functionality provided by
the browser.
Summary
With
little-to-no training, anyone can scan a network using Nmap by simply
specifying the host or target range. While Fyodor has done a great job
making Nmap default options as efficient as possible, you should still
be able to improve scanning efficiency by customizing your scans.
Hopefully, by sharing a proven scan methodology and tricks for
manipulating the output, we have provided ideas for you to refine your
Nmap scan strategy and get the most from your efforts.
For this article and others like it, check out the following magazine:
Special Thanks To
Bill Hau
Dan Dumond
Dennis Hanzlik
Jacob Martinson
Ron Nguyen
Rudolph Araujo