Ethical Hacking: Active Recon – Enumeration and Fuzzing with Feroxbuster

https://websploit.org/

https://github.com/The-Art-of-Hacking/h4cker

https://hackingscenarios.com/

Introducing Feroxbuster

Feroxbuster is an ethical hacking tool written in Rust that is designed to perform Forced Browsing. You can use it to enumerate and access resources that are not referenced by a web application, but can still be accessible to the attacker.

Feroxbuster Source Code and GitHub Repository

Feroxbuster is an open-source tool and its source code can be accessed at this GitHub repository. The following link provides a comparison with some of the tools we discussed in previous scenarios:

https://github.com/epi052/feroxbuster#-comparison-w-similar-tools

Feroxbuster and Wordlists

In previous lessons part of these Ethical Hacking Labs, you learned that a wordlist is a text file containing a collection of words to use in different attacks including, but not limited to password cracking, recon, and fuzzing. feroxbuster uses wordlists to search for unlinked content in target directories. Some of these resources and content may store sensitive information about the web application and the underlying operating systems. You may often find additional information such as source code, credentials, sensitive logs, and more.

You can use feroxbuster to perform:

  • Predictable resource location attacks
  • File enumeration
  • Directory enumeration
  • Resource enumeration

In the next section, you will learn how to install feroxbuster

Installing Feroxbuster

You can install feroxbuster in Windows, Linux, and macOS.

Installing Feroxbuster in Windows

You can download the latest release of feroxbuster from the release section of its the GitHub repository. Use the x86-windows-feroxbuster.exe.zip binary file to install feroxbuster in Windows.

Installing Feroxbuster Using the apt Package Manager

Kali Linux:

If you are using Kali Linux, you can install feroxbuster using the following command:

sudo apt update && sudo apt install -y feroxbuster

Other Debian-based Linux Distributions:

Complete the following steps to install feroxbuster in any other Debian-based Linux distributions (such as Ubuntu and Parrot Security OS):

  1. Download feroxbuster_amd64.deb from the GitHub repository Releases section, as demonstrated below:

curl -sLO https://github.com/epi052/feroxbuster/releases/latest/download/feroxbuster_amd64.deb.zip

  1. Unzip the file:

unzip feroxbuster_amd64.deb.zip

  1. Install feroxbuster using apt:

sudo apt install -y ./feroxbuster_*_amd64.deb

Once feroxbuster is installed, you can verify the version using the following command:

feroxbuster -V

$ feroxbuster -V
feroxbuster 2.3.3

Installing Feroxbuster Using snap

Snap is a software packaging and deployment system developed by Canonical. In this lab, you are using Ubuntu:

lsb_release -a

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

You can install feroxbuster using snap, as demonstrated below:

sudo snap install feroxbuster

Note: The snap package can only read wordlists from a very specific locations. If the wordlist is on the same partition as your home directory, you can hard-link it into ~/snap/feroxbuster/common, as demonstrated below:

ln /path/to/the/wordlist ~/snap/feroxbuster/common

If the wordlist is on a separate partition, hard-linking won’t work. You’ll need to copy it into the snap directory:

cp /path/to/the/wordlist ~/snap/feroxbuster/common

Installing Feroxbuster Using cargo

Feroxbuster is written in the Rust programming language and it’s available in crates.io. If you have the Rust programming language installed in your system, you can install feroxbuster using cargo, as demonstrated below:

cargo install feroxbuster

Intalling Feroxbuster in macOS

You can install feroxbuster in macOS using Homebrew with the following two commands:

  1. brew tap tgotwig/feroxbuster
  2. brew install feroxbuster

Intalling Feroxbuster in BlackArch

You can install feroxbuster in BlackArch Linux) using the following command: pacman -S feroxbuster

Running the Feroxbuster Docker Container

You can run feroxbuster using Docker, as shown below: sudo docker run --init -it epi052/feroxbuster -u http://10.6.6.8 -x js,html

NOTE: Additional details about the feroxbuster Docker image can be obtained from the GitHub repository.

In the following section, you will learn how to customize the feroxbuster configuration

Feroxbuster Configuration Details

The feroxbuster configuration file is ferox-config.toml. The feroxbuster tool searches for the ferox-config.toml configuration file in the following locations (in the order shown):

  1. /etc/feroxbuster/ (global)
  2. CONFIG_DIR/feroxbuster/ (per-user)
  3. The same directory as the feroxbuster executable (per-user)
  4. Your current working directory (per-target)

You can override the default configuration by modifying the following location of the CONFIG_DIR (configuration directory) on each supported operating system:

  • Linux: $XDG_CONFIG_HOME or $HOME/.config
    • For example: /home/bob/.config
  • MacOS: $HOME/Library/Application Support
    • For example: /Users/bob/Library/Application Support
  • Windows: {FOLDERID_RoamingAppData}
    • For example: C:\Users\Bob\AppData\Roaming

A good example of a use case where you would like to override the default configuration is when you prefer to use a different wordlist setting than the default/built-in wordlist.

You are using Ubuntu in this lab. You can display the contents of the default configuration file with the following command:

cat /etc/feroxbuster/ferox-config.toml

The contents should be similar to the following output:

$ cat ferox-config.toml 
# Example configuration for feroxbuster
#
# If you wish to provide persistent settings to feroxbuster, rename this file to ferox-config.toml and make sure
# it resides in the same directory as the feroxbuster binary.
#
# After that, uncomment any line to override the default value provided by the binary itself.
#
# Any setting used here can be overridden by the corresponding command line option/argument
#
# wordlist = "/wordlists/seclists/Discovery/Web-Content/raft-medium-directories.txt"
# status_codes = [200, 500]
# filter_status = [301]
# threads = 1
# timeout = 5
# proxy = "http://127.0.0.1:8080"
# replay_proxy = "http://127.0.0.1:8081"
# replay_codes = [200, 302]
# verbosity = 1
# parallel = 8
# scan_limit = 6
# rate_limit = 250
# quiet = true
# silent = true
# auto_tune = true
# auto_bail = true
# json = true
# output = "/targets/ellingson_mineral_company/gibson.txt"
# debug_log = "/var/log/find-the-derp.log"
# user_agent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"
# redirects = true
# insecure = true
# extensions = ["php", "html"]
# url_denylist = ["http://dont-scan.me", "https://also-not.me"]
# no_recursion = true
# add_slash = true
# stdin = true
# dont_filter = true
# extract_links = true
# depth = 1
# filter_size = [5174]
# filter_regex = ["^ignore me$"]
# filter_similar = ["https://somesite.com/soft404"]
# filter_word_count = [993]
# filter_line_count = [35, 36]
# queries = [["name","value"], ["rick", "astley"]]
# save_state = false
# time_limit = "10m"

# headers can be specified on multiple lines or as an inline table
#
# inline example
# headers = {"stuff" = "things"}
#
# multi-line example
#   note: if multi-line is used, all key/value pairs under it belong to the headers table until the next table
#         is found or the end of the file is reached
#
# [headers]
# stuff = "things"
# more = "headers"

You can display the feroxbuster command usage/help information by executing the following command:

feroxbuster -h

The output should be similar to the following:

$ feroxbuster -h
feroxbuster 2.3.3
Ben 'epi' Risher (@epi052)
A fast, simple, recursive content discovery tool written in Rust

USAGE:
    feroxbuster [FLAGS] [OPTIONS] --url ...

FLAGS:
    -f, --add-slash        
            Append / to each request

        --auto-bail        
            Automatically stop scanning when an excessive amount of errors are encountered

        --auto-tune        
            Automatically lower scan rate when an excessive amount of errors are encountered

    -D, --dont-filter      
            Don't auto-filter wildcard responses

    -e, --extract-links    
            Extract links from response body (html, javascript, etc...); make new requests based on findings (default:
            false)
    -h, --help             
            Prints help information

    -k, --insecure         
            Disables TLS certificate validation

        --json             
            Emit JSON logs to --output and --debug-log instead of normal text

    -n, --no-recursion     
            Do not scan recursively

    -q, --quiet            
            Hide progress bars and banner (good for tmux windows w/ notifications)

    -r, --redirects        
            Follow redirects

        --silent           
            Only print URLs + turn off logging (good for piping a list of urls to other commands)

        --stdin            
            Read url(s) from STDIN

    -V, --version          
            Prints version information

    -v, --verbosity        
            Increase verbosity level (use -vv or more for greater effect. [CAUTION] 4 -v's is probably too much)


OPTIONS:
        --debug-log                         
            Output file to write log entries (use w/ --json for JSON entries)

    -d, --depth 
            Maximum recursion depth, a depth of 0 is infinite recursion (default: 4)

    -x, --extensions ...          
            File extension(s) to search for (ex: -x php -x pdf js)

    -N, --filter-lines ...                 
            Filter out messages of a particular line count (ex: -N 20 -N 31,30)

    -X, --filter-regex ...
            Filter out messages via regular expression matching on the response's body (ex: -X '^ignore me$')

        --filter-similar-to ...
            Filter out pages that are similar to the given page (ex. --filter-similar-to http://site.xyz/soft404)

    -S, --filter-size ...                   
            Filter out messages of a particular size (ex: -S 5120 -S 4927,1970)

    -C, --filter-status ...          
            Filter out status codes (deny list) (ex: -C 200 -C 401)

    -W, --filter-words ...                 
            Filter out messages of a particular word count (ex: -W 312 -W 91,82)

    -H, --headers ...                     
            Specify HTTP headers (ex: -H Header:val 'stuff: things')

    -o, --output                            
            Output file to write results to (use w/ --json for JSON entries)

        --parallel 
            Run parallel feroxbuster instances (one child process per url passed via stdin)

    -p, --proxy 
            Proxy to use for requests (ex: http(s)://host:port, socks5(h)://host:port)

    -Q, --query ...                        
            Specify URL query parameters (ex: -Q token=stuff -Q secret=key)

        --rate-limit 
            Limit number of requests per second (per directory) (default: 0, i.e. no limit)

    -R, --replay-codes ...
            Status Codes to send through a Replay Proxy when found (default: --status-codes value)

    -P, --replay-proxy 
            Send only unfiltered requests through a Replay Proxy, instead of all requests

        --resume-from 
            State file from which to resume a partially complete scan (ex. --resume-from ferox-1606586780.state)

    -L, --scan-limit                  
            Limit total number of concurrent scans (default: 0, i.e. no limit)

    -s, --status-codes ...
            Status Codes to include (allow list) (default: 200 204 301 302 307 308 401 403 405)

    -t, --threads                        
            Number of concurrent threads (default: 50)

        --time-limit                   
            Limit total run time of all scans (ex: --time-limit 10m)

    -T, --timeout                        
            Number of seconds before a request times out (default: 7)

    -u, --url ...                            
            The target URL(s) (required, unless --stdin used)

        --dont-scan ...                      
            URL(s) to exclude from recursion/scans

    -a, --user-agent                  
            Sets the User-Agent (default: feroxbuster/VERSION)

    -w, --wordlist                          
            Path to the wordlist


NOTE:
    Options that take multiple values are very flexible.  Consider the following ways of specifying
    extensions:
        ./feroxbuster -u http://127.1 -x pdf -x js,html -x php txt json,docx

    The command above adds .pdf, .js, .html, .php, .txt, .json, and .docx to each url

    All of the methods above (multiple flags, space separated, comma separated, etc...) are valid
    and interchangeable.  The same goes for urls, headers, status codes, queries, and size filters.

EXAMPLES:
    Multiple headers:
        ./feroxbuster -u http://127.1 -H Accept:application/json "Authorization: Bearer {token}"

    IPv6, non-recursive scan with INFO-level logging enabled:
        ./feroxbuster -u http://[::1] --no-recursion -vv

    Read urls from STDIN; pipe only resulting urls out to another tool
        cat targets | ./feroxbuster --stdin --silent -s 200 301 302 --redirects -x js | fff -s 200 -o js-files

    Proxy traffic through Burp
        ./feroxbuster -u http://127.1 --insecure --proxy http://127.0.0.1:8080

    Proxy traffic through a SOCKS proxy
        ./feroxbuster -u http://127.1 --proxy socks5://127.0.0.1:9050

    Pass auth token via query parameter
        ./feroxbuster -u http://127.1 --query token=0123456789ABCDEF

    Find links in javascript/html and make additional requests based on results
        ./feroxbuster -u http://127.1 --extract-links

    Ludicrous speed... go!
        ./feroxbuster -u http://127.1 -t 200

In the next section, you will learn how to enumerate files and directories using Feroxbuster.

Enumerating Directories and Files using Feroxbuster

Let’s start enumerating files and directories in a web application.

Accessing the Vulnerable Application

There is a vulnerable web application running in a container called unsc_infinity. Verify that the container is running by using the following command:

docker ps

If the container is not running, execute the following command:

bash ~/setup.sh

The vulnerable web application should be running on 10.6.6.8. Verify that you can reach the container by:

  • Using pingping -c 3 10.6.6.8
  • Using curlcurl http://10.6.6.8

Download a Wordlist

Let’s download a common wordlist from hackingscenarios.com to enumerate directories in the target web application.

wget https://hackingscenarios.com/wordlists/dir_enum

File and Directory Enumeration

The following is a simple test using the wordlist that you downloaded in the previous step:

feroxbuster -w dir_enum -u http://10.6.6.8

  • the -w is used to specify the wordlist
  • the -u is used to specify the URL of the web application

The feroxbuster tool starts enumerating directories. The tool uses 50 threads by default. You should see the additional details of the current test in the banner, as shown below:

 🎯  Target Url            │ http://10.6.6.8
 🚀  Threads               │ 50
 📖  Wordlist              │ dir_enum
 👌  Status Codes          │ [200, 204, 301, 302, 307, 308, 401, 403, 405, 500]
 💥  Timeout (secs)        │ 7
 🦡  User-Agent            │ feroxbuster/2.3.3
 💉  Config File           │ /etc/feroxbuster/ferox-config.toml
 🔃  Recursion Depth       │ 4

During the scan, you should see the following columns:

  • column 1status code – can be filtered with -C|--filter-status
  • column 2number of lines – can be filtered with -N|--filter-lines
  • column 3number of words – can be filtered with -W|--filter-words
  • column 4number of bytes (overall size) – can be filtered with -S|--filter-size
  • column 5url to discovered web application resource

NOTE: You should see a large number of errors. Do not be alarmed. Some of these errors are because the tool is not able to find the specific resources.

The wordlist that you downloaded earlier includes thousands of words. Even scanning or fuzzing a small application could take several minutes to complete. You can press Enter to access the Scan Cancel Menu. Once press Enter, you can enter a comma-separated list of indexes or ranges to cancel (for example: 1-4,8,9-13). You can also use -f to skip confirmation (for example: 3-5 -f).

If the scan is taking too long, feel free to skip several tests or press Ctrl+C to terminate the enumeration. If you press Ctrl+C, a file with the extension .state is created. This file includes the terminated test “state” in JSON format. You can open the file with the following command:

cat ferox-*.state | python3 -m json.tool

Extracting Links from the Response Body

You can extract links and look for additional endpoints to scan from the response body. You can simply use the --extract-links option to extract links from the response body, as shown below:

feroxbuster -u http://10.6.6.8 --extract-links

You can also limit the number of scans allowed to run at any given time by using the --scan-limit option, as shown below:

feroxbuster -w dir_enum -u http://10.6.6.8 --scan-limit 2

NOTE: The recursion feature of feroxbuster will still identify new directories. However, newly discovered directories can only begin scanning when the total number of active scans drops below the specified number of scans. The scan limit is set to 2 in the example above.

In the next section, you will learn how to send results to a web proxy.

Sending Results to a Proxy

One of the best ways to understand how a web application actually works is to observe it. A sniffer (packet capture software) is one possible choice. However, a web proxy is another available tool that can make the job a little easier. The following are two of the most popular web proxies among penetration testers and bug hunters:

Web proxies allow the penetration tester and bug hunters to attack and debug web applications. These tools allow you to intercept, inspect, and modify the raw contents of the traffic, as follows:

  • Intercept: Enables you to see under the hood and watch the traffic move back and forth between the client and the server.
  • Inspect: Enables you to enumerate how applications work and see the mechanisms they use.
  • Modify: Enables you to modify the data in an attempt to see how the application will respond (for example, injection attacks).

These tools help you perform SQL injection, cookies subversion, session hijacking, command injection, cross-site scripting (XSS), cross-site request forgery (CSRF), and many other types of attacks.

Sending the Results to a Web Proxy

Feroxbuster allows you to send the results to a web proxy such as Burp Suite or the OWASP Zed Attack Proxy (ZAP). You can use the –proxy option (as demonstrated below) to send the results to a web proxy:

feroxbuster -w dir_enum -u http://10.6.6.8 --insecure --proxy http://127.0.0.1:8080

Burp Suite and the OWASP ZAP use port 8080 by default.

NOTE: The --insecure flag is set because the proxy is not using HTTPS.

Sending the Results to a SOCKS Proxy

You can send the results to a SOCKS proxy (including DNS resolution lookups) by using the following command:

feroxbuster -w dir_enum -u http://10.6.6.8 --proxy socks5h://127.0.0.1:9050

In the following section, you will learn about additional advanced enumeration options.

Advanced Enumeration Options

The following are a few additional advanced enumeration options.

Specifying File Extensions to Scan Specific Files

You can specify multiple file types (extensions) to be scanned with the -x option. The following example adds .pdf.js.html.php.txt.json, and .docx extensions to each scanned URL.

feroxbuster -w dir_enum -u http://10.6.6.8 -x pdf,js,html,php,txt,json,docx

You can specify each extension using comma-separated or space-separated arguments. After executing the command above, you should be able to see the following extensions in the feroxbuster tool banner.

 💲  Extensions            │ [pdf, js, html, php, txt, json, docx]

Including HTTP Header Elements

You can include HTTP header elements using the -H option, as shown below:

feroxbuster -u http://10.6.6.8 -w dir_enum -H Accept:application/json "Authorization: Bearer {token}"

This is particularly useful when testing application programming interfaces (APIs).

Passing an Authorization Token via the --query Parameter

Token-based authentication is a protocol which allows web applications to verify their user’s identity or the identity of another web application by using a unique access token. During the life of the token, users or other applications (often via APIs) access the application resources instead of having to re-enter credentials each time they go back to the same web application resource protected with that same token. You can pass an authorization token using feroxbuster’s --query parameter, as demonstrated below:

feroxbuster -u http://10.6.6.8 -w dir_enum --query token=0987654321DEADBEEF

Auto-Tune and Auto-Bail

The --auto-tune and --auto-bail options allow you to configure different “policies” for “tunning” or “bailing” a scan. These auto-tune and auto-bail policies are only enforced after at least 50 requests have been made by the feroxbuster tool. Tunning and bailing actions are triggered by the following criteria:

  • number of general errors (for example: timeouts) is higher than half the number of threads (or at least 25 if threads are lower)
  • 90% of responses are 403 (Forbidden) error messages
  • 30% of requests are 429 (Too Many Requests) error messages

NOTE: All the criteria above is on a per-directory-scanned basis.

The --auto-tune policy enforces a rate limit on individual directory scans when one of the aforementioned criteria is met. The rate limit self-adjusts every (timeout / 2) seconds. If the number of errors have increased during that time, the allowed rate of requests is lowered. On the other hand, if the number of errors hasn’t moved, the allowed rate of requests is increased. The rate limit will be removed completely if no additional errors are found after a certain number of checks.

The --auto-bail policy cancels individual directory scans when one of the aforementioned criteria is met. They just stop getting scanned, no muss, no fuss.

Running Multiple Scans in Parallel

You can run multiple scans in parallel using the –parallel option. This allows you to scan a large number of hosts or web applications at the same time. You can put all your target applications URLs in a file and then use that file as shown in the following example:

cat my_targets | feroxbuster --stdin --parallel 10 --extract-links --auto-bail

The target file used in the example above is my_targets and 10 scans are run in parallel.

Rate Limiting

You can limit the number of requests per second using the --rate-limit option, as shown below:

feroxbuster -u http://10.6.6.8 -w dir_enum --rate-limit 50

You can also limit the number of directories that can be processed or scanned at the same time using the --scan-limit option, as demonstrated below:

feroxbuster -u http://10.6.6.8 -w dir_enum --rate-limit 50 --scan-limit 1