Detection Lab: Threat hunting and automated detection

Detection Lab: Threat hunting and automated detection

Finally had time to build my own Detection Lab, the idea is to build a complete environment for Threat Hunting and Detection, using the latest best practices to create secure environments. Eventually building it via Infrastructure as a Code in the Cloud using DevOps Tool chain – (Git, Docker, Vagrant, Terraform, Powershell, Python, WMI and shell bash script.), . I am adding tools for PenTesting, Threat Detection, Threat Intelligence and DFIR .

Final goal is to expose it to the wild using honeypots and do some real world threat detection. But, that is long time away. There really isn’t many options like this to get access to a complete lab environment. Few options like OSCP and SANs training provide very specific lab environments. (e.g. SANs NetWars.) Each of these would cost circa $2,000 for training and lab access.



Lab configuration

This is my home lab configuration to build Detection Lab based on Chris Long, I will migrate this to AWS or GCP using Terraform in the near future, after testing the automation scripts.

My Lab Setup

My home lab (aka The Best) is based on a HP DL350 running Windows 2012R (MSDN licensing.) I was able to buy this at HP Cost pice via my contacts! I can add another CPU and more storage if required, but this is enough power to run multiple VM up without any issues. I was running Hyper-V but changed to VirtualBox (due to issues with Vagrant. But,VirtualBox is extremely unstable, compared to Hyper-V which is more stand and has higher performance.). I use TeamViewer to connect to it remotely and over the internet. Of course the name of my server is TRON. I installed a Nvidia GTX 980 so I can also play games and get at least 30-60 FPS on Squad.

TRON 2020-02-22 13-13-58

Lab Hardware

  • HP ML350e Gen8 E5-2407
  • Windows Server 2012 R2 Datacenter
  • 36 GB RAM
  • Nvidia GTX 980 (So I can play Squad while working and in-between lab builds.)
  • Intel Xeon CPU E5-24070 @2.20 Ghz
  • Multiple Hard Drives in RAID 1, etc. 2 TBs+ storage
  • Hyper-V or VirtualBox
    • I had allot of problems trying to using Vagrant and Hyper-V, I am sure I can get it to work, but wasting time troubleshoot. So i decided to convert existing VMs into VirtualBox as the scripts all work.


  1. AD
    1. Domain Controllers, DNS, KMS
  2. DB
    1. SQL
  3. Windows 10
    1. Shift + Flare
  4. Offensive-RedTeam
    1. KALI
    2. CommandoVM
  5. Web Server
    1.  Honeypot
    2.  NGNIX
    3. ModSecurity
  6. Pfsense
  7. Defencive-Blue-Team
    1.  Logger
    2. Docker
  8. Azure
    1. Honeypot


  1. Add my Tool Set into Lab using Automation
  2. Build on AWS or GCP with strip down script and export of VMs into S3 or Git or another file location.
  3. Create honey pot and do threat detection.
  4. Add more tools for DFIR, PenTest, Technologies, etc..
  5. Create Honeypot
  6. Create a Secure OS and Web Server with WAF, etc.
  7. Install Splunk in Docker – Done
  8. Install LogRythm – Done
  9. Install ArcSight Logger
  10. Install Ansible
  11. Install Python and Tensor Flow, etc.
  12. Install GNS Simulator
  13. UTM, Router and Event Data
  14. Setup a Real time, Isolated Honeypot with with full monitoring.

Building Splunk lab for spelunking, Get a license for Splunk.

Splunk offers few options for Labs, Developer license, Free Version and NFT license for partners.

To generate your NFR keys, do the following:

  1. Login to the partner portal (if you don’t have a login, you need to go to and create a Splunk account name first) or Splunk Dev license –
  2. Scroll over the person icon
  3. Go to “Licenses”
  4. Click on “Generate License” next to “Splunk Enterprise – Splunk Partner+ NFR License”

By selecting this option, ES will be delivered to you too

  1. Be on a look out for 2 emails – one for your license key and one from splunkbase with instructions on how to download ES and other Apps

Install a license for a standalone instance

To install a license for a standalone instance of Splunk Enterprise:

  1. On the instance, navigate to Settings > Licensing.
  2. Click Add license.
  3. Do one of the following:

    1. Click Choose file and browse for your license file and select it, or
    2. Click copy & paste the license XML directly… and paste the text of your license file into the provided field.
  4. Click Install.
  5. If this is the first Enterprise license that you are installing on the instance, you must restart Splunk Enterprise.

*Kindly note that as per NFR program, we only allow 1 instance of 50GB NFR per account per year.

Threat Datasets

here is what I have in my notes. Not tested, but recommended by good people: generate data in AD build an AD with 0 efforts, other reco: or in several steps: create: populate: add misconfigs: other reco to build an AD with little effort:

Breach Simulation

  • Caldera –
  • Safebreach –
  • Cymulate –
  • Verodin –
  • Infection Monkey –
  • Atomic Red Team –
  • CobaltStrike –
  • Leonidas –


Detection Lab Build

  • Enable or Disable Hypver-v on Windows –
  • [sourcecode]
    Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All
    Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All
    remove-windowsfeature -name Hyper-V
    dism.exe /Online /Enable-Feature:Microsoft-Hyper-V /All
    dism.exe /Online /Disable-Feature:Microsoft-Hyper-V
    bcdedit /set hypervisorlaunchtype off
    bcdedit /set hypervisorlaunchtype auto
  • Install X2Go Remote control for Linux
  • Install Windows Package manager –
  • [sourcecode]@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString(''))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"[/sourcecode]
  • @powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((new-object net.webclient).DownloadString(''))" && SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin
  • Install Git, packer, Cygwi (for future packing) and Vagrant
    • cinst git.install -y choco install git -params '"/GitAndUnixToolsOnPath"'
      cinst vagrant -y
      choco install vagrant
      choco install packer
      choco install cygwin
      choco install putty
      setx path "c:\tools\cygwin\bin"
      choco install vagrant --upgrade
      choco install vagrant --force
      shutdown /r /t 00 
      vagrant global-status
  • Vagrant Usage
    • @"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "clear"
      vagrant init
      vagrant up --provider=hyperv
      vagrant box add {name-of-box} {}
      vagrant box list
      //Example Box
      //This will add box from
      vagrant box add hashicorp/precise64 --provider=hyperv
      vagrant init
      vagrant up --provider=hyperv
      vagrant ssh
      // Enable Vagrant Debug Logs
      set VAGRANT_LOG=info
      vagrant up
      vagrant up --debug 2>&1 | Tee-Object -FilePath ".\vagrant.log"
      vagrant up --debug 2>&1 | Tee-Object -FilePath ".\vagrant.log"
      //Set Default Provider
      [Environment]::SetEnvironmentVariable("VAGRANT_DEFAULT_PROVIDER", "hyperv", "User")
      //Install Plugin
      vagrant plugin update
      vagrant plugin install vagrant-reload
      //Remove all Boxes
      vagrant box list | cut -f 1 -d ' ' | xargs -L 1 vagrant box remove -f
      //Uninstall Vagrant
  • Create Hyperv Virtual Switch and setup permissions
  • You can also do this with PowerShell
  • Select the NIC connected to the internet / Ethernet Properties / Sharing / Enable Internet Sharing with the VirtualBox or HyperV External Network or Birdge.
  • Vagrant Configuration file for Hyperv
  • [sourcecode]# -*- mode: ruby -*-
    # vi: set ft=ruby :
    # All Vagrant configuration is done below. The "2" in Vagrant.configure
    # configures the configuration version (we support older styles for
    # backwards compatibility). Please don't change it unless you know what
    # you're doing.
    Vagrant.configure("2") do |config| = "hashicorp/precise64"
    config.vm.provider "hyperv" "public_network"
    config.vm.synced_folder ".", "/vagrant", disabled: true
    config.vm.provider "hyperv" do |h|
    h.enable_virtualization_extensions = true
    h.linked_clone = true
  • Convert from HyperV to VirtualBox (VT-x is not available (VERR_VMX_NO_VMX).)
    • VBoxManage clonehd input.vhdx output.vdi –format VDI
      Added this new disk as an IDE drive (“PIIX4”)
      Enabled System/EFI Setting for this VM.
      VT-x is not available (VERR_VMX_NO_VMX).
  • Vagrant Hyper-V 2012 R2 issue

 Detection Lab Reference

Other Technology Deployment

Install Splunk in Docker


// Install SSH Server

sudo iptables -L | grep ssh
sudo netstat -anp | grep sshd

sudo ufw disable
sudo ufw verbose

sudo ufw allow ssh
sudo ufw allow 22

sudo apt-get update
sudo apt-get install openssh-server

sudo apt update
sudo apt install openssh-server

sudo apt-get remove openssh-client openssh-server
sudo apt-get install openssh-client openssh-server

sudo systemctl status ssh


$ sudo apt-get remove docker docker-engine containerd runc

//Upgrade linux to latest edition
sudo apt-get update
sudo apt-get upgrade
sudo do-release-upgrade
uname -r
lsb_release -a

// Fun stuff
sudo apt-get install asciinema
asciinema auth
asciinema rec demo.cas
asciinema upload

// Install Docker
sudo apt install
sudo systemctl start docker
sudo systemctl enable docker
docker --version

// Install Docker from Official Repository  
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL | sudo apt-key add –

sudo add-apt-repository "deb [arch=amd64]  $(lsb_release -cs)  stable"
sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg2 \ software-properties-common

$ curl -fsSL | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository \ "deb [arch=amd64] \ $(lsb_release -cs) \ stable"
$ sudo apt-get install docker-ce docker-ce-cli
sudo docker run hello-world

//Install Splunk in Docker
docker pull splunk/splunk:latest
docker run -d -p 8000:8000 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=' splunk/splunk:latest 
docker ps -aq
docker ps -l
docker container rm
docker ps -a -f id=
docker container start
docker stop
//docker exec -it bash …license" -v ~/Desktop/local_app:/opt/splunk/etc/apps/container_app splunk/s… 




Install Microfocus ArcSight Logger on Centos

Kernel Panic on Hyper-V The latest versions of Ubuntu and CentOS both get Kernel panic errors on Hyper-V, and the only work around is to use a previous version and then upgrade. (very frustrating.) Capture

Install ArcSight Logger


// Setup Static IP Address
nmtui edit eth0

// Install
sudo yum install epel-release
sudo yum install asciinema
asciinema auth
asciinema record cast.cast
//Update hostname
sudo nano /etc/sysconfig/network
nano /etc/host
hostnamectl set-hostname
/etc/init.d/network restart
sudo systemctl restart NetworkManager.service   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 hostname

// Install wget, unzip, tmux,tar and nano
yum install wget unzip nano tmux tar -y
sudo yum install -y nano
// Hyper-V Optimization
sudo nano /etc/udev/rules.d/100-balloon.rules
SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
sudo yum install -y hyperv-daemons
sudo yum list installed | grep hyperv
//Disable IP 6
// Change the Disk I/O Scheduler
su root
echo noop > /sys/block/sda/queue/scheduler
// Defaults
yum install - unzip
yum install -y fontconfig \ dejavu-sans-fonts

// Make sure that you have the latestsupported tzdata rpm, tzdata2019c, installed on your OS before
installing Logger.

yum install tzdata
yum install tzdata

wget http://some_website/sample_file.rpm
sudo rpm –i sample_file.rpm
sudo yum localinstall sample_file.rpm
sudo rpm –i sample_file.rpm
sudo dnf localinstall sample_file.rpm
// A non-root user account must exist on the system on which you are installing Logger, or the installer will
ask you to provide one. Even if you install as root, a non-root user account is still required. The userid
and its primary groupid should be the same for this account. The UID for the non-root user should be
1500 and the GID should be 750. For example, to create the non-root user, run these commands as

groupadd –g 750 arcsight
useradd –m –g arcsight –u 1500 arcsight
chown -R arcsight:arcsight /opt/arcsight/

// Increase default user process limit
nano /etc/security/limits.d/20-nproc.conf
// delete all connets in file and add the follwing
* soft nproc 10240
* hard nproc 10240
* soft nofile 65536
* hard nofile 65536
// then log out and verify open files 65536 / max user processes 10240

ulimit -a

// modify login.conf .. RemoveIPC=no

nano /etc/systemd/logind.conf
cd /etc/systemd
systemctl restart systemd-logind.service

// Map CIFS share to software
yum install samba-client samba-common cifs-utils
mkdir /mnt/win
nano /etc/fstab
\\winbox\getme /mnt/win cifs user,uid=500,rw,suid,username=sushi,password=yummy 0 0 mount.cifs \\\\.....\\share /mnt/software cifs user,uid=500,rw,suid,username=,password= 0 0
mkdir /opt/arcsight
mkdir /mnt/software
mount.cifs \\\\\\synology /mnt/software cifs user,uid=500,rw,suid,username=,password= 0 0
cd /mnt/software/ISOLibary/Software/ArcSightLogger
cp ArcSight-logger- /opt/arcsight
chmod u+x ArcSight-logger-7.0.[8280].0.bin
./ArcSight-logger-7.0.[8280].0.bin -i console

// Gnome Desktop
yum groupinstall “GNOME Desktop” “Graphical Administration Tools”
sudo yum groupinstall -y "KDE Plasma Workspaces"

/etc/init.d/arcsight_logger status {start|stop|restart|status|quit}
/etc/init.d/arcsight_logger start all

systemctl disable firewalld
systemctl stop firewalld
systemctl status firewalld

// Access Logger Dashboard
https://....1.121/platform-ui/ admin|password


Upgrade CentOS


//upgrade CentOS - log in as root

cat /etc/centos-release
sudo yum check-update
sudo yum clean all
sudo yum update





Vmware KALI Mount shared Folder


ls /mnt/hgfs

sudo mount -t fuse.vmhgfs-fuse .host:/ /mnt/hgfs -o allow_other

ln -s /mnt/hgfs// /root/Desktop/



Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.