Ethical Hacking: Advanced Nmap Scanning and the Nmap Scripting Engine

Introduction to Nmap Scripting Engine (NSE)

Nmap provides a user with a powerful toolset of features, that allows the user to create scripts to complete custom tasks.

Installing Nmap

To install Nmap on Ubuntu, use the apt package manager by exeucuting:

apt update; apt install -y nmap

The Nmap Scripting Engine is included and is a part of the normal nmap package.

About NSE scripts

Scripts are created using the LUA scripting language and end with the .nse file extension. There are well over 300+ examples to choose from, and many are installed with Nmap.

Envolking NSE

Using the Nmap Scripting Engine is simple! When using the nmap command we append the --script flag, followed by the script name or directory of scripts, and target host or range:

nmap --script script/directory HOST/RANGE

Script Categories

Included Nmap Scripts are listed via different categories.

For example, scripts that are created for the purpose of brute-forcing authentication credentials are placed within the “brute” category.

Full list of NSE Scripts

The Nmap documentation provides a complete comprehensive list of scripts that ship with Nmap when installing. To view a full list of Nmap Scripting Engine example scripts, please visit: https://nmap.org/nsedoc/

At any point, you can download these scripts by clicking on the “Download” link within the documentation.

Nmap Cheat Sheet

A Nmap cheat sheet provided by H4cker.org can be viewed by visiting, or simply using curl and accessing https://h4cker.org/cheat/nmap.

curl https://h4cker.org/cheat/nmap

Docker

The rest of the scenario will use Docker to simulate a vulnerable host on a network. The gravemind Docker container is obtained from Docker Hub. Multiple hosts will be set up for you using Docker.

Please verify that two gravemind instances are running using the following command:

docker ps

If no containers running, a setup script is provided in your home directory and can be executed by running: bash ~/gravemind_docker_network_setup.sh

These two containers have statically assigned IP Addresses:

  • Container 1: 10.0.0.5
  • Container 2: 10.0.0.6

FTP Anonymous Login

After observing that a machine on the network has an open port 21, commonly used for FTP, we can employ the help of Nmap to anonymously login on available FTP servers across the network, and check what available files are being served. For this, we will be using the Nmap Scripting Engine to carry out the task for us.

The FTP-anon.nse script can perform such action.

Executing the script on a single host

To perform a scan on a single host, pass the FTP-anon.nse script name as an argument and append a target host. For example, the command structure would be as follows:

nmap --script FTP-anon.nse HOST

To execute the command against our Gravemind container, execute the following command:

nmap --script ftp-anon.nse 10.0.0.5

The following output can be seen after executing the command:

Starting Nmap 7.80 ( https://nmap.org ) at 2021-08-13 20:24 UTC
Nmap scan report for 10.0.0.5
Host is up (0.000097s latency).
Not shown: 994 closed ports
PORT    STATE SERVICE
21/tcp  open  ftp
| ftp-anon: Anonymous FTP login allowed (FTP code 230)
| -rw-rw-r--    1 0        0              16 Aug 13 19:46 file1.txt
| -rw-rw-r--    1 0        0              16 Aug 13 19:46 file2.txt
| -rw-rw-r--    1 0        0              29 Aug 13 19:46 file3.txt
|_-rw-rw-r--    1 0        0              26 Aug 13 19:45 supersecretfile.txt
22/tcp  open  ssh
53/tcp  open  domain
80/tcp  open  http
139/tcp open  netbios-ssn
445/tcp open  microsoft-ds

Nmap done: 1 IP address (1 host up) scanned in 0.22 seconds

Here we see that an FTP server was found, and accessibnle through the anonymous user. The directory structure is listed, including:

read-write-executeFile or DirectoryUser IDGroup IDSize in BytesDateFilename
-rw-rw-r–10016 Aug 13 19:4616file1.txt
-rw-rw-r–10016 Aug 13 19:4616file2.txt
-rw-rw-r–10029 Aug 13 19:4629file3.txt
-rw-rw-r–10026 Aug 13 19:4529supersecretfile.txt

Executing the script on a network range

To perform the scan on a complete range of hosts, append the CIDR formatted range to the command:

nmap --script ftp-anon.nse RANGE/CIDR

For example to scan our docker network:

nmap --script ftp-anon.nse 10.0.0.0/24

Below we can see that hosts 10.0.0.5 and 10.0.0.6 have been found and are listing the available files on their respective hosts.

Starting Nmap 7.80 ( https://nmap.org ) at 2021-08-13 20:27 UTC
Nmap scan report for katacoda (10.0.0.1)
Host is up (0.00018s latency).
All 1000 scanned ports on katacoda (10.0.0.1) are closed

Nmap scan report for 10.0.0.5
Host is up (0.00019s latency).
Not shown: 994 closed ports
PORT    STATE SERVICE
21/tcp  open  ftp
| ftp-anon: Anonymous FTP login allowed (FTP code 230)
| -rw-rw-r--    1 0        0              16 Aug 13 19:46 file1.txt
| -rw-rw-r--    1 0        0              16 Aug 13 19:46 file2.txt
| -rw-rw-r--    1 0        0              29 Aug 13 19:46 file3.txt
|_-rw-rw-r--    1 0        0              26 Aug 13 19:45 supersecretfile.txt
22/tcp  open  ssh
53/tcp  open  domain
80/tcp  open  http
139/tcp open  netbios-ssn
445/tcp open  microsoft-ds

Nmap scan report for 10.0.0.6
Host is up (0.00020s latency).
Not shown: 994 closed ports
PORT    STATE SERVICE
21/tcp  open  ftp
| ftp-anon: Anonymous FTP login allowed (FTP code 230)
| -rw-rw-r--    1 0        0              16 Aug 13 19:46 file1.txt
| -rw-rw-r--    1 0        0              16 Aug 13 19:46 file2.txt
| -rw-rw-r--    1 0        0              29 Aug 13 19:46 file3.txt
|_-rw-rw-r--    1 0        0              26 Aug 13 19:45 supersecretfile.txt
22/tcp  open  ssh
53/tcp  open  domain
80/tcp  open  http
139/tcp open  netbios-ssn
445/tcp open  microsoft-ds

Nmap done: 256 IP addresses (3 hosts up) scanned in 3.17 seconds

Documentation

The complete documentation can be found on the Nmap Scripting Engine Documentation

Network Share Enumeration

SMB is another popular file transfer protocol, which is very common within Microsoft Windows environments as the default file share application. Gravemind, our example container, is running an installation of Samba serving files through the SMB protocol.

Executing the script on a single host

The common port for the SMB protocol is port 445. We can query this port using the smb-enum-share.nse script to receive a list of available SMB shares on a single host.

To perform the scan, use the --script smb-enum-shares.nse argument when invoking the nmap command.

Note: For simplicity, we added the -p 445 argument to the command for faster scans, but we can omit the argument to scan the top 1000 ports.

nmap --script smb-enum-shares.nse -p 445 HOST

For example, to scan all shares on our first Gravemind container:

nmap --script smb-enum-shares.nse -p 445 10.0.0.5

Below we can see the 10.0.0.5\workfiles share, along with some print services shared by our Gravemind host.

Starting Nmap 7.80 ( https://nmap.org ) at 2021-08-13 21:21 UTC
Nmap scan report for 10.0.0.5
Host is up (0.00017s latency).

PORT    STATE SERVICE
445/tcp open  microsoft-ds

Host script results:
| smb-enum-shares:
|   account_used: <blank>
|   \\10.0.0.5\IPC$:
|     Type: STYPE_IPC_HIDDEN
|     Comment: IPC Service (Samba 4.9.5-Debian)
|     Users: 1
|     Max Users: <unlimited>
|     Path: C:\tmp
|     Anonymous access: READ/WRITE
|   \\10.0.0.5\print$:
|     Type: STYPE_DISKTREE
|     Comment: Printer Drivers
|     Users: 0
|     Max Users: <unlimited>
|     Path: C:\var\lib\samba\printers
|     Anonymous access: READ/WRITE
|   \\10.0.0.5\workfiles:
|     Type: STYPE_DISKTREE
|     Comment: Confidential Workfiles
|     Users: 0
|     Max Users: <unlimited>
|     Path: C:\var\spool\samba
|_    Anonymous access: READ/WRITE

Nmap done: 1 IP address (1 host up) scanned in 7.57 seconds

Executing the script on a network range

To receive a list of available SMB mounts on a range of hosts, append the CIDR to the command:

nmap --script smb-enum-shares.nse -p 445 RANGE/CIDR

An example to scan both Gravemind containers can be found below:

nmap --script smb-enum-shares.nse -p 445 10.0.0.0/24

Starting Nmap 7.80 ( https://nmap.org ) at 2021-08-13 21:38 UTC
Nmap scan report for katacoda (10.0.0.1)
Host is up (0.00033s latency).

PORT    STATE  SERVICE
445/tcp closed microsoft-ds

Nmap scan report for 10.0.0.5
Host is up (0.00037s latency).

PORT    STATE SERVICE
445/tcp open  microsoft-ds

Host script results:
| smb-enum-shares:
|   account_used: <blank>
|   \\10.0.0.5\IPC$:
|     Type: STYPE_IPC_HIDDEN
|     Comment: IPC Service (Samba 4.9.5-Debian)
|     Users: 1
|     Max Users: <unlimited>
|     Path: C:\tmp
|     Anonymous access: READ/WRITE
|   \\10.0.0.5\print$:
|     Type: STYPE_DISKTREE
|     Comment: Printer Drivers
|     Users: 0
|     Max Users: <unlimited>
|     Path: C:\var\lib\samba\printers
|     Anonymous access: READ/WRITE
|   \\10.0.0.5\workfiles:
|     Type: STYPE_DISKTREE
|     Comment: Confidential Workfiles
|     Users: 0
|     Max Users: <unlimited>
|     Path: C:\var\spool\samba
|_    Anonymous access: READ/WRITE

Nmap scan report for 10.0.0.6
Host is up (0.00032s latency).

PORT    STATE SERVICE
445/tcp open  microsoft-ds

Host script results:
| smb-enum-shares:
|   account_used: <blank>
|   \\10.0.0.6\IPC$:
|     Type: STYPE_IPC_HIDDEN
|     Comment: IPC Service (Samba 4.9.5-Debian)
|     Users: 1
|     Max Users: <unlimited>
|     Path: C:\tmp
|     Anonymous access: READ/WRITE
|   \\10.0.0.6\print$:
|     Type: STYPE_DISKTREE
|     Comment: Printer Drivers
|     Users: 0
|     Max Users: <unlimited>
|     Path: C:\var\lib\samba\printers
|     Anonymous access: READ/WRITE
|   \\10.0.0.6\workfiles:
|     Type: STYPE_DISKTREE
|     Comment: Confidential Workfiles
|     Users: 0
|     Max Users: <unlimited>
|     Path: C:\var\spool\samba
|_    Anonymous access: READ/WRITE

Nmap done: 256 IP addresses (3 hosts up) scanned in 12.70 seconds

Documentation

The complete documentation can be found on the Nmap Scripting Engine Documentation

Web Application Enumeration

Nmap’s scripting engine can ease the process of finding commonly used webpage paths on a web server. It does so by digging through a fingerprint file and applying a pattern match across the webserver. It can also identify specific versions of the web server and display them within the output.

Executing the script on a single host

The http-enum.nse file can be passed to the script argument, as well as -sV which provides a more verbose service and version information. The script can be executed by using the following format:

nmap -sV --script=http-enum.nse HOST

To scan our Gravemind container, execute the following:

nmap -sV --script=http-enum.nse 10.0.0.5

Below we can see the scan found /admin//admin/index.html, and /s/ as interesting routes to access. The webserver information can also be seen as being Nginx followed by the version number.

Starting Nmap 7.80 ( https://nmap.org ) at 2021-08-13 22:34 UTC
Nmap scan report for 10.0.0.5
Host is up (0.000097s latency).
Not shown: 994 closed ports
PORT    STATE SERVICE     VERSION
21/tcp  open  ftp         vsftpd 3.0.3
22/tcp  open  ssh         OpenSSH 7.9p1 Debian 10+deb10u2 (protocol 2.0)
53/tcp  open  domain      ISC BIND 9.11.5-P4-5.1+deb10u5 (Debian Linux)
80/tcp  open  http        nginx 1.14.2
| http-enum:
|   /admin/: Possible admin folder
|   /admin/index.html: Possible admin folder
|_  /s/: Potentially interesting folder
|_http-server-header: nginx/1.14.2
139/tcp open  netbios-ssn Samba smbd 3.X - 4.X (workgroup: WORKGROUP)
445/tcp open  netbios-ssn Samba smbd 3.X - 4.X (workgroup: WORKGROUP)
Service Info: Host: 4EDD64A64E32; OSs: Unix, Linux; CPE: cpe:/o:linux:linux_kernel

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 11.77 seconds

Executing the script on a range

To execute the http enumeration script on a range of IPs, use the following format:

nmap -sV --script=http-enum RANGE/CIDR

Using this format, we can scan our Gravemind container by executing:

nmap -sV --script=http-enum 10.0.0.0/24

Starting Nmap 7.80 ( https://nmap.org ) at 2021-08-13 22:42 UTC
Nmap scan report for katacoda (10.0.0.1)
Host is up (0.00018s latency).
All 1000 scanned ports on katacoda (10.0.0.1) are closed

Nmap scan report for 10.0.0.5
Host is up (0.00021s latency).
Not shown: 994 closed ports
PORT    STATE SERVICE     VERSION
21/tcp  open  ftp         vsftpd 3.0.3
22/tcp  open  ssh         OpenSSH 7.9p1 Debian 10+deb10u2 (protocol 2.0)
53/tcp  open  domain      ISC BIND 9.11.5-P4-5.1+deb10u5 (Debian Linux)
80/tcp  open  http        nginx 1.14.2
| http-enum:
|   /admin/: Possible admin folder
|   /admin/index.html: Possible admin folder
|_  /s/: Potentially interesting folder
|_http-server-header: nginx/1.14.2
139/tcp open  netbios-ssn Samba smbd 3.X - 4.X (workgroup: WORKGROUP)
445/tcp open  netbios-ssn Samba smbd 3.X - 4.X (workgroup: WORKGROUP)
Service Info: Host: 4EDD64A64E32; OSs: Unix, Linux; CPE: cpe:/o:linux:linux_kernel

Nmap scan report for 10.0.0.6
Host is up (0.00022s latency).
Not shown: 994 closed ports
PORT    STATE SERVICE     VERSION
21/tcp  open  ftp         vsftpd 3.0.3
22/tcp  open  ssh         OpenSSH 7.9p1 Debian 10+deb10u2 (protocol 2.0)
53/tcp  open  domain      ISC BIND 9.11.5-P4-5.1+deb10u5 (Debian Linux)
80/tcp  open  http        nginx 1.14.2
| http-enum:
|   /admin/: Possible admin folder
|   /admin/index.html: Possible admin folder
|_  /s/: Potentially interesting folder
|_http-server-header: nginx/1.14.2
139/tcp open  netbios-ssn Samba smbd 3.X - 4.X (workgroup: WORKGROUP)
445/tcp open  netbios-ssn Samba smbd 3.X - 4.X (workgroup: WORKGROUP)
Service Info: Host: FA2E2E48B5A4; OSs: Unix, Linux; CPE: cpe:/o:linux:linux_kernel

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 256 IP addresses (3 hosts up) scanned in 14.84 seconds

Documentation

The complete documentation can be found on the Nmap Scripting Engine Documentation

Exporting Nmap Output

The output of any scan can be output into various formats, including XML and a grep-friendly format. Each format has a respective command argument associated and expects a basename.

Exporting Scan Output in the Generic Nmap Format (-oN)

To save the”normally outputted” scan as shown on screen to a file, the -oN FILENAME argument can be appended to any scan.

The format of the complete command is shown below for a TCP SYN Scan:

nmap -sS -oN FILENAME HOST/RANGE

To run this command against the Gravemind container, execute the following:

nmap -sS -oN nmap_scan.nmap 10.0.0.5

If you ls in the current directory, you will see a file named nmap_scan.nmap.

You can view the file by using the command cat nmap_scan.nmap.

Exporting the Scan Output in XML (-oX)

To save the scan as an .xml file, append the -oX FILENAME option to a scan.

The format of the complete command is shown below for a TCP SYN Scan:

nmap -sS -oX FILENAME HOST/RANGE

To run this command against our Gravemind container, execute the following:

nmap -sS -oX nmap_scan.xml 10.0.0.5

If you ls in the current directory, you will see a file named nmap_scan.xml.

You can view the file by using the command cat nmap_scan.xml. Alternatively, you can parse the data using any software that is capable of displaying .xml files.

Exporting the Scan Output in a File that be Parsed with Grep (-oG)

The grep command is a powerful tool to search and filter data within a file. Normal usage includes passing in a pattern to find and a path to the data file.

To save the scan in a format that is easier to use the grep command in, append the -oX FILENAME option to a scan.

The format of the complete command is shown below for a TCP SYN Scan:

nmap -sS -oG FILENAME HOST/RANGE

To run this command against our Gravemind container, execute the following:

nmap -sS -oG nmap_scan.gnmap 10.0.0.5

If you ls in the current directory, you will see a file named nmap_scan.gnmap.

You can view the file by using the command cat nmap_scan.gnmap.

Exporting the Scan Output in All Formats (-oA)

To save the scan in all the aforementioned formats, use the -oA FILENAME argument when performing a scan.

The format of the complete command is shown below for a TCP SYN Scan:

nmap -sS -oA BASE_FILENAME HOST/RANGE

To run this command against our Gravemind container, execute the following:

nmap -sS -oA nmap_scan 10.0.0.5

If you ls in the current directory, you will see a file named nmap_scan.gnmapnmap_scan.xml, and nmap_scan.nmap.

Creating Nmap Python Scripts

Python can be used to programmatically manipulate Nmap scans and return formatted output back to the user.

Installing Python 3 & Pip3

Installing Python 3 and the pip3 package manager can be done using apt.

Execute the following command to install the python3 and pip3 packages:

apt update; apt install -y python3 python3-pip

Execute the following command to make sure that pip3 is updated to the latest version: python3 -m pip install --upgrade pip

Installing the Python Nmap module

pip3 install python3-nmap

Viewing the supplied Python file

For this lab, a file called nmap.py has been placed within your home directory. To view the contents of the file execute cat nmap.py.

Importing the Nmap3 module

Near the top of the file, we must specify to use the nmap3 module.

To do this, we can write the statement import nmap3 to the top of our file.

Next, we must create a nmap object to be able to execute different scans. We can create the object by assigning a variable to nmap3.Nmap(). Below we can see both the import and object instantiation at play.

# Import the Nmap Module
import nmap3

# Create an nmap Object
nmap = nmap3.Nmap()

Now we can start our scanning using the nmap object.

Nmap Helper Functions and Globals

Within the code, a function called print_output has been written to minimize repeated code within the script. The function simply takes a message string and result dictionary to display correctly on the screen.

The Gravemind container IP is also assigned to the HOST variable near the middle of the Python file. This is to add more flexibility to the scans. The string can be set to any target host within the network.

The SUBNET global is similar to HOST but instead of targeting one host, the target is a complete range of hosts.

Scanning Top 1000 ports

To scan the top 1000 ports of a host, use the .scan_top_ports(hostname: str) -> dict method. This method takes a hostname string as a parameter and returns a dictionary object as a response.

The following code scans our Gravemind Container:

...

# TCP SYN scan example
results = nmap.scan_top_ports(HOST)
print_output(f"Top ports of {HOST}", results)

...

Below we can see that the 10.0.0.5 host has been found and subsequent metadata of the host has been placed into different keys/value pairs.

...

'Top ports of 10.0.0.5'
{'10.0.0.5': {'hostname': [],
              'macaddress': None,
              'osmatch': {},
              'ports': [{'portid': '21',
                         'protocol': 'tcp',
                         'reason': 'syn-ack',
                         'reason_ttl': '0',
                         'scripts': [],
                         'service': {'conf': '3',
                                     'method': 'table',
                                     'name': 'ftp'},
                         'state': 'open'},
                        {'portid': '22',
                         'protocol': 'tcp',
                         'reason': 'syn-ack',
                         'reason_ttl': '0',
                         'scripts': [],
                         'service': {'conf': '3',
                                     'method': 'table',
                                     'name': 'ssh'},
                         'state': 'open'},

...

Nmap OS Detection

To detect the operating system of a host, we can use the .nmap_os_detection(hostname: str) -> dict method of the Nmap class to perform the scan. The function takes a hostname as a parameter and returns a dict of all hosts found.

...

# Nmap OS Detection
results = nmap.nmap_os_detection(HOST)
print_output(f"OS detection for {HOST}", results)

...

The host OS details for our Gravemind container, 10.0.0.5, can be found below.

{'10.0.0.5': {'hostname': [],
              'macaddress': {'addr': '02:42:0A:00:00:05', 'addrtype': 'mac'},
              'osmatch': [{'accuracy': '96',
                           'cpe': 'cpe:/o:linux:linux_kernel:2.6.32',
                           'line': '55173',
                           'name': 'Linux 2.6.32',
                           'osclass': {'accuracy': '96',
                                       'osfamily': 'Linux',
                                       'osgen': '2.6.X',
                                       'type': 'general purpose',
                                       'vendor': 'Linux'}},
                          {'accuracy': '96',
                           'cpe': 'cpe:/o:linux:linux_kernel:4',
                           'line': '65105',
                           'name': 'Linux 3.2 - 4.9',
                           'osclass': {'accuracy': '96',
                                       'osfamily': 'Linux',
                                       'osgen': '4.X',
                                       'type': 'general purpose',
                                       'vendor': 'Linux'}},

...

Nmap Subnet Scan

To scan an entire subnet of hosts, the .nmap_subnet_scan(subnet: str) -> dict method can be used. This method will return all open ports for hosts found on the network.

...

# NSE subnet scan
results = nmap.nmap_subnet_scan(SUBNET)
print_output(f"HTTP Enumeration for {SUBNET}", results)

...

Below is a snippet of the returned output for each host.

...

 '10.0.0.5': {'hostname': [],
              'macaddress': {'addr': '02:42:0A:00:00:05', 'addrtype': 'mac'},
              'osmatch': {},
              'ports': [{'portid': '21',
                         'protocol': 'tcp',
                         'reason': 'syn-ack',
                         'reason_ttl': '64',
                         'scripts': [],
                         'service': {'conf': '3',
                                     'method': 'table',
                                     'name': 'ftp'},
                         'state': 'open'},
                        {'portid': '22',
                         'protocol': 'tcp',
                         'reason': 'syn-ack',
                         'reason_ttl': '64',
                         'scripts': [],
                         'service': {'conf': '3',
                                     'method': 'table',
                                     'name': 'ssh'},
                         'state': 'open'},
...

 '10.0.0.6': {'hostname': [],
              'macaddress': {'addr': '02:42:0A:00:00:06', 'addrtype': 'mac'},
              'osmatch': {},
              'ports': [{'portid': '21',
                         'protocol': 'tcp',
                         'reason': 'syn-ack',
                         'reason_ttl': '64',
                         'scripts': [],
                         'service': {'conf': '3',
                                     'method': 'table',
                                     'name': 'ftp'},
                         'state': 'open'},
                        {'portid': '22',
                         'protocol': 'tcp',
                         'reason': 'syn-ack',
                         'reason_ttl': '64',
                         'scripts': [],
                         'service': {'conf': '3',
                                     'method': 'table',
                                     'name': 'ssh'},
                         'state': 'open'},

...

Running the python script

To run the python script, use the python command followed by the script name. In our case, the command to run the provided nmap.py file would be:

python3 nmap.py

Full example

# Import Pretty Printing
from pprint import pprint

# Import the Nmap Module
import nmap3

# Create an nmap Object
nmap = nmap3.Nmap()

# Helper functions

def print_output(message: str, results: str) -> None:
    """Handle output message and printing results"""
    pprint("")
    pprint('-'*20)
    pprint(message)
    pprint(results)


# Define target host
HOST = "10.0.0.5"
SUBNET = "10.0.0.0/24"

# TCP SYN scan example
results = nmap.scan_top_ports(HOST)
print_output(f"Top ports of {HOST}", results)

# Nmap OS Detection
results = nmap.nmap_os_detection(HOST)
print_output(f"OS detection for {HOST}", results)

# NSE subnet scan
results = nmap.nmap_subnet_scan(SUBNET)
print_output(f"Subnet scan for {SUBNET}", results)

Documentation

To view the full documentation of python3-nmap please visit: https://github.com/nmmapper/python3-nmap

Python3-nmap is under the GNU General Public License 3.0 and the license can be found here: https://github.com/nmmapper/python3-nmap/blob/master/LICENSE