Refining Your Nmap Scan Strategy

By Tony Lee

Introduction

Disclaimers:  This is not a troll.  :)  Scans were run as root (or sudo) using Nmap version 6.25.

Few would dispute that the de facto standard scanning tool is Gordon Lyon’s (aka Fyodor) Nmap.  However, if we asked you to thoroughly and efficiently scan a remote class C (192.168.1.1/24) network, which of the following would you run?

1. nmap 192.168.1.0/24
2. nmap -sS 192.168.1.0/24
3. nmap -sS 192.168.1.0-255
4. nmap -sS -Pn -p 0-65535 192.168.1.0/24

The answer we hear most often is option a.  While this may work for small networks, it does not scale for larger networks or more thorough assessments.  The astute reader will notice that options a, b, and c, operate identically.  Option a provides the network range in CIDR notation and since -sS is the default scan type when no options are supplied--option b is identical to option a.  Examining option c, reveals that it is the same as options a and b, except that the target is supplied using a network range instead of CIDR notation.  The problem with options a, b, and c is that they will not thoroughly scan the remote class c network as they will only scan the top 1000 TCP ports.  Option d is close to what we are looking for since it scans all of the TCP ports; however, it lacks efficiency since we will be scanning all ports on all hosts, including dead IP space.

The most important thing to keep in mind is that in this simple example, we are only asking to scan a class C network.  However as security practitioners, we are often faced with multiple class C’s, class B’s or even in some cases, class A networks.  We will take the rest of this article to outline a scan strategy for networks of all sizes and some tips and tricks for carving the output.

Outline

• Host Discovery
• -sn “Ping Scan”
• Throttling Pro-tips
• Generate Live Hosts List
• Port Discovery
• Most Common Ports
• Full Port Scans
• Putting it all Together
• Service and Operating System
• Nmap Scripting Engine (NSE)
• Manipulating Output
• Carving Standard Output
• Convert to HTML
• Processing Script
• Summary

Host Discovery

A methodical approach to scanning usually involves performing host discovery first, which eliminates the problem we saw with the above option d--scanning copious amounts of dead IP space.  Fortunately, Nmap is quite intelligent and flexible for performing host discovery scans.  The common host discovery scan options are listed below:

-sn: Ping Scan - disable port scan  -PS/PA/PU/PY[portlist]: TCP SYN/ACK, UDP or SCTP discovery to given ports  -PE/PP/PM: ICMP echo, timestamp, and netmask request discovery probes  -PO[protocol list]: IP Protocol Ping

The option we will focus on in this article is the most popular and the default for host discovery (-sn “Ping Scan”).

-sn Ping Scan

One thing to note about the -sn “Ping Scan” option (formerly known as -sP) is that it does much more than just a traditional ICMP echo request.  By default against a remote network the “Ping scan” consists of the following packets:

• ICMP echo request
• TCP SYN to 443
• TCP ACK to 80
• ICMP timestamp request

Figure 1:  Packet capture confirms behavior of a remote "Ping Scan"

When scanning a local network, Nmap automatically switches to using ARP requests.  This is desirable intelligence because fewer packets are sent and it is generally more accurate.  A system that blocks all ICMP traffic and filters TCP ports 80 and 443 would be invisible to the four remote discovery packets sent above.  However, hosts typically respond to ARP requests because layer 2 addresses are necessary to construct packet headers.

Figure 2:  Packet capture confirms behavior of a local "Ping Scan"

For the first packet capture, the scanning host was placed on a different subnet than that of the targets--this caused Nmap to use the 4 discovery packets shown in Figure 1.  For the second packet capture, the scanning host was placed in the same broadcast range and thus ARP was utilized for discovery as shown in Figure 2.  Both of the packet captures were created with the same discovery scan shown below.

root@DVORAK:~/scans# nmap -sn -T4 -oA Discovery 192.168.1.0/24 Starting Nmap 6.25 ( http://nmap.org ) at 2013-05-13 13:21 EDT Nmap scan report for Wireless_Broadband_Router.home (192.168.1.1) Host is up (0.0023s latency). MAC Address: 00:26:62:XX:XX:XX (Actiontec Electronics) Nmap scan report for 192.168.1.2 Host is up. Nmap scan report for android.home (192.168.1.3) Host is up (0.025s latency). MAC Address: CC:3A:61:XX:XX:XX (Unknown) Nmap scan report for computer.home (192.168.1.147) Host is up (0.00035s latency). MAC Address: 24:77:03:XX:XX:XX (Intel Corporate) Nmap done: 256 IP addresses (4 hosts up) scanned in 4.45 seconds

Options explained:

-sn = “Ping scan”

-T4 = Throttle to aggressive

-oA <basename>= Output in all three formats (normal, XML, greppable)

Throttling Pro-tips:

Aggressive (-T4) throttling is substantially faster than the default Normal (-T3) throttle.  We have rarely (if ever) seen Aggressive scanning crash a host or flood a network.  This should be what you start with unless you know of particular hosts that are sensitive to scanning.  With that said, if hosts are known to crash on simple scans or become easily flooded, avoid scanning them with -T4 and possibly try Polite (-T2) throttling.  Please realize that -T2 may be up to 10 times slower than -T3, so be patient and only run it on your most sensitive hosts--not hundreds at a time.  More importantly, avoid any one-off scans such as version scanning as these are more likely to crash hosts than the speed of the scan.  Some older SCADA components are known to fall over from simple port scanning, not to mention version scanning.  In general, we do not recommend Insane mode (-T5) as this can negatively affect accuracy.  Lastly, we only recommend -T0 or -T1 when trying to be extra stealthy (IDS evasion) and only for scanning a few ports on a few hosts because it will likely be too slow for anything else.

Generate Live Hosts List

Nmap, like other tools, accepts a file containing a list of IP addresses as input.  Because the remaining scans are more intensive than the discovery scans, we will want to only feed live hosts to Nmap.  We can use the output files from our discovery scan to generate this list of live hosts.  Three files are generated as a result of the -oA option:

root@DVORAK:~/scans# ls Discovery.gnmap  Discovery.nmap  Discovery.xml

Files explained:

.nmap = Normal output (what is printed to the screen)

.gnmap = Greppable output

.xml = XML output

We will extract the live hosts from the .gnmap (greppable) file using a command such as the following:

root@DVORAK:~/scans# grep "Status: Up" Discovery.gnmap | cut -f 2 -d ' ' > LiveHosts.txt root@DVORAK:~/scans# cat LiveHosts.txt 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.147

Cut options explained:

-f = field number (in this case, field 2)

-d = delimiter (in this case, a space)

The resulting LiveHosts.txt file is one IP address per line, which can be supplied as input to Nmap and other tools as well.

Port Discovery

Now that we have generated our LiveHosts.txt file, we can start port discovery on these hosts.  If you have many live hosts to scan and only a short amount of time, you may want to break it up into two sets of scans:

1. Most common ports
2. Full port scans

Most Common Ports

Fortunately, Fyodor performed a substantial amount of research for his Top Ports Project which he presented in 2008 at BlackHat and Defcon.  He performed scans on millions of Internet IPs and incorporated internal scans from large organizations to determine the most commonly open TCP and UDP ports.  The key take-away from this research is that Fyodor made Nmap default port scans more efficient by targeting the top 1000 TCP and top 1000 UDP ports when no port options are provided.  But just how effective is scanning only the most common ports?  Here is an eye-opening table that summarizes the results of Fyodor’s research:

Table 1:  Summary of Fyodor's research

TCP	UDP
­­top­ports 10: 48%	­­top­ports 10: 50%
­­top­ports 50: 65%	­­top­ports 50: 86%
­­top­ports 100: 73%	-­top­ports 100: 90%
­­top­ports 250: 83%	-­top­ports 250: 94%
­­top­ports 500: 89%	­­top­ports 500: 97%
­­top­ports 1000: 93%	­­top­ports 1017: ~100%
­­top­ports 2000: 96%
­­top­ports 3674: ~100%

Source:  http://nmap.org/presentations/BHDC08/bhdc08-slides-fyodor.pdf

As shown in the table above, according to Fyodor’s research, scanning the top 1000 TCP ports is 93% effective in identifying all open TCP ports, while scanning the top 3674 TCP ports is almost 100% effective.  Additionally, scanning the top 1017 UDP ports is nearly 100% effective at identifying all open UDP ports.  Since the default scan is the top 1000 TCP and UDP ports, no port options are needed below.  (If scanning the top 1000 ports is not desirable, use the --top-ports option and specify a new number of ports to scan.)

Most common port scans:

nmap -sS -T4 -Pn -oA TopTCP -iL LiveHosts.txt nmap -sU -T4 -Pn -oA TopUDP -iL LiveHosts.txt

New Options explained:

-sS = SYN Scan (half-open scanning) - Does not complete the 3-way handshake so it is faster than -sT (default scan - specifying is optional)

-sU = UDP Scan

-Pn = Do not perform discovery scan (we already know the hosts are alive)

-iL = Input from a file

Full Port Scans

Each host can have a maximum of 131,072 ports (that is 2^16 = 65536, then multiply by 2 for TCP and UDP).  But why would anyone need to scan all of those ports?  Why isn’t the top 1000 good enough--especially after we highlighted how effective it was?  One surprising port that is not contained in the top 1000 is TCP/2433 (the alternative default MS-SQL port, which can be a misconfigured Achilles heel for many networks).  Thus, skipping the full port scan can miss critical services that may be the entry point for an attacker or a piece of malware.   

Assuming that full port scans are desired, these can be executed when the most common port scans finish.  Full port scans will take substantially longer to finish than the common port scans.  Thus, it is ideal to have the results of the common port scans as something to review while the full port scans are running.  

Note:  Full port UDP scanning is VERY slow, and for extremely large networks may never finish in a reasonable amount of time.

Full port scans:

nmap -sS -T4 -Pn -p 0-65535 -oA FullTCP -iL LiveHosts.txt nmap -sU -T4 -Pn -p 0-65535 -oA FullUDP -iL LiveHosts.txt

New Options explained:

-p <ports> = port list

Putting it all together

So what does the scanning process look like when combined?

Discovery: nmap -sn -T4 -oA Discovery 192.168.1.0/24 Generate Live Hosts List: grep "Status: Up" Discovery.gnmap | cut -f 2 -d ' ' > LiveHosts.txt Common Ports: nmap -sS -T4 -Pn -oA TopTCP -iL LiveHosts.txt nmap -sU -T4 -Pn -oA TopUDP -iL LiveHosts.txt Full Ports: nmap -sS -T4 -Pn -p 0-65535 -oA FullTCP -iL LiveHosts.txt nmap -sU -T4 -Pn -p 0-65535 -oA FullUDP -iL LiveHosts.txt

Following this process should perform efficient host discovery, live host file generation, and full port scans for both TCP and UDP (along with the top 1000 ports, in case the full port scan does not finish in time).

Service and Operating System

After reviewing the common port scans (or full port scans if available), a few hosts of concern may emerge.  If this is the case, a more intensive scan may be desired such as a service and/or operating system scan.  These more intensive scans should not be used in discovery on a large network because it may take too long to finish.  If you have many live hosts, only run these scans on select hosts where more information is desired.

Service (Version) Scan

The ports that were discovered in prior scans were quickly labeled with associated services based on the nmap-services database of about 2,200 well-known services.  The example scan below shows ports 22, 23, and 80 all labeled with the expected services ssh, telnet, and http respectively.

Example SYN scan: nmap -sS -T4 -PN -n 192.168.1.2 PORT   STATE SERVICE 22/tcp open  ssh 23/tcp open  telnet 80/tcp open  http

However, since applications/services can run on any arbitrary port, additional probing may be desired to ensure that the service matches the associated port.  For this additional probing, version detection (-sV) is useful.

Example version scan: nmap -sV -T4 -PN -n 192.168.1.2 PORT   STATE SERVICE VERSION 22/tcp open  ssh     OpenSSH 5.3p1 Debian 3ubuntu7 (Ubuntu Linux; protocol 2.0) 23/tcp open  telnet  Linux telnetd 80/tcp open  ssh     OpenSSH 5.3p1 Debian 3ubuntu7 (Ubuntu Linux; protocol 2.0) Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

Notice that while the SYN scan (-sS) labeled port 80/tcp as http (per the nmap-services information), a deeper version scan (-sV) revealed that port 80 was really another instance of SSH.

Figure 3:  The screenshot above shows a SYN scan and a Version scan.  The version scan revealed that port 80 is actually SSH.  The version scan also correctly guessed the version of SSH.

Version scan syntax:

nmap -sV -T4 -Pn -oA <host>-service.txt <host>

New Options explained:

-sV = Service Version Scan

As you can see, version scanning is quite useful, but as mentioned earlier, it comes at a cost.  It requires more packets and thus takes longer to scan.  Notice in figure 3 that the SYN scan finished in 0.08 seconds, while it took the version scan 10.33 seconds.  Multiply the difference by thousands of machines and it adds up.

Operating System Scan

For very large networks, the operating system for each scanned host is not always known.  There are two quick checks that almost always work for high level operating system discovery (Windows vs. Linux)--especially when the two techniques are combined:

1. TTL - Time To Live value (quickly obtained via ping, be sure to subtract the number of hops)
1. Windows will generally have a TTL of 128
2. Linux will generally have a TTL of 64
3. Network devices will generally have a TTL of 255
2. RPC (port mapper) port
1. Windows hosts will have TCP 135 (msrpc) open
2. Linux hosts will have TCP 111 (rpcbind) open

However, if you need more detail, such as the specific version of Windows or Linux, this is where Nmap’s operating system scan comes in handy.  Nmap uses TCP/IP stack fingerprinting and an nmap-os-db of more than 2,600 known OS fingerprints in order to accurately identify the operating system.  In most cases it is in the ballpark and if it does list a few operating systems, the correct OS is usually in the list.

nmap -O -T4 -Pn -oA <host>-service.txt <host>

New Options explained:

-O = Operating System Scan

Caution:  Add the necessary ports using the -p option if the ports you want to use for operating system scans are not in the top 1000 ports.  Also be sure to include at least one open and one closed port to achieve a more accurate operating system guess.

Figure 4:  Comparison between SYN scan and operating system scan.

Some may want to use both -sV and -O at the same time, which is fine.  There is even a shortcut option for this discussed in the next section; however, we will provide a word of warning there as well.

NSE

Prior to the Nmap Scripting Engine (NSE), Nmap had very limited vulnerability scanning capabilities.  However, with the addition of NSE scripts, it can now seek out (and in some cases exploit) vulnerabilities.  This increases the flexibility and extensibility of Nmap by allowing users to write their own scripts with just a little bit of LUA scripting knowledge.

At the time of this writing, there are 433 NSE scripts provided as part of the Nmap download.  To view the scripts that are available, list the directories that contain the NSE scripts.

On Windows hosts the scripts are typically in the following location:

c:\Program Files (x86)\Nmap\scripts\

On Linux hosts the scripts are typically in the following location:

/usr/local/share/nmap/scripts/

You can also check this reference for a listing and description:  http://nmap.org/nsedoc/

Since these are interpreted (and not compiled) scripts, it is possible to view the source to better understand what they do and even tweak them to behave differently.  It is also quite helpful to start from one of these functioning scripts if LUA is not your forte.

To activate Nmap NSE scripts, use the --script option and supply comma separated script names, categories, or directories:

--script filename|category|directory|expression[,...]

Since NSE was designed for a variety purposes, including extended version scanning, vulnerability detection and even exploitation, all scripts have one or more categories defined.  This not only adds organization, but it can simplify the choices when running multiple scripts by being able to select a family of scripts based on their categories.

Category options are:  auth, broadcast, brute, default, discovery, dos, exploit, external, fuzzer, intrusive, malware, safe, version, and vuln.

In addition to running scripts based on category, there are a couple really nice shortcuts, but they may not be the best choice in all circumstances:

• -sC = Script Scanning:  Equivalent to --script=default and will execute scripts which have the default category defined
• -A = Aggressive scanning:  Performs the following: Operating System (-O), Version (-sV), Script scanning (-sC) and traceroute (--traceroute).

The reason why caution should be applied when running either of these options is that default scripts can include scripts in the intrusive category, which have the potential to crash systems or make printers go crazy.

Instead, it may be desirable to exercise more control over which NSE scripts are executed by using the boolean AND with the “safe” category.  For example:

nmap -sS -T4 -Pn --script “default and safe” <host> nmap -sS -T4 -Pn --script “http-* and safe” <host> nmap -sS -T4 -Pn --script “smb-* and safe” <host> nmap -sS -T4 -Pn --script “safe” <host>

Figure 5:  Results of an Nmap script scan against a Windows 8 host

For more information, a great resource is Fyodor’s NSE usage page:

http://nmap.org/book/nse-usage.html

Manipulating Output

As mentioned earlier, the output options for Nmap are normal (.nmap), greppable (.gnmap), and XML (.xml).  We will now explore how we can manipulate these standard output formats to achieve even greater insight--which is particularly useful when viewing larger output files.

Carving Standard Output

The standard output files can be quite useful for generating statistics or feeding other tools.  The nmap and gnmap files are good for pulling information such as:

• Unique open ports
• Number of unique open ports
• Open ports to feed into other tools
• Hosts that are listening on a particular port
• Number of hosts listening on a particular port

Unique open ports:

grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu 22/tcp 23/tcp 80/tcp 135/tcp --snip--

Number of unique open ports:

grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu | wc -l 16

Feed open ports into another tool (space separated):

grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu | cut -f 1 -d '/' | xargs 22 23 80 135 139 443 445 902 912 992 4445 4567 5357 8080 8443 49155

Feed open ports into another tool (comma separated):

grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu | cut -f 1 -d '/' | xargs | sed 's/ /,/g' 22,23,80,135,139,443,445,902,912,992,4445,4567,5357,8080,8443,49155

Hosts that are listening on a particular port (telnet used below as an example):

grep 23/open/tcp FullTCP.gnmap | cut -f 2 -d ' ' 192.168.1.1 192.168.1.2

(Replace port 23 with whatever port number you wish to find other ports/hosts of interest)

How many hosts are listening on a particular port (telnet used below as an example):

grep 23/open/tcp FullTCP.gnmap | cut -f 2 -d ' ' | wc -l 2

(Replace port 23 with whatever port number you wish to find other ports/hosts of interest)

Convert to HTML

We just illustrated the usefulness of normal and greppable output files, however XML files are also useful.  Nmap XML files can be converted into HTML format for easy viewing within a web browser.  This is possible through using Extensible Stylesheet Language Transformations (XSLT).

Detailed information on various options can be found here:  http://nmap.org/book/output-formats-output-to-html.html

One very reliable method is using the xsltproc tool (from http://xmlsoft.org/XSLT/).  Using it is simple and the result is portable.  Use the Nmap XML file for input and use -o to specify the output file.

Syntax is below:

root@DVORAK:~/scans# xsltproc TopTCP.xml -o TopTCP.html root@DVORAK:~/scans# firefox TopTCP.html

Result:

Figure 6:  HTML result of using xsltproc

Notice that the output is very organized and port information is even displayed in easy to read tables.  Navigation is made convenient through the use of the included hyperlinks and search functionality provided by the browser.

Summary

With little-to-no training, anyone can scan a network using Nmap by simply specifying the host or target range.  While Fyodor has done a great job making Nmap default options as efficient as possible, you should still be able to improve scanning efficiency by customizing your scans.  Hopefully, by sharing a proven scan methodology and tricks for manipulating the output, we have provided ideas for you to refine your Nmap scan strategy and get the most from your efforts.

For this article and others like it, check out the following magazine:

http://hakin9.org/nmap-guide-revisited-hakin9-tutorials/

Special Thanks To

Bill Hau

Dan Dumond

Dennis Hanzlik

Jacob Martinson

Ron Nguyen

Rudolph Araujo