Introduction to the Incident

Recently we where notified by a large international company of an incident in one of its sites in the US. The incident resulted in the spread of the computer worm QakBot.T onto an estimate of 160 workstations and laptops inside one of the network sites of the client.

The computer-worm created a domain user account lockout, which resulted in employees being unable to work and the client implemented an auto-unlock script that ran every 5 minutes, unlocking all users that have been locked.

Although the client had worked together with an anti-malware vendor in order to create new virus definitions, to find and remove QakBot.T from infected machines, it had not led to the expected result and machines kept locking-out users – a nasty side-effect of the worm.

The client initiated the formal Incident Response a week after it was identified by the client, after which the formal Incident Response procedure began. An immediate conference call was initiated between us and the client, in which initial analysis of the events had been performed and relevant recommendations have been given in regards to containing the incident.

Incident Scope

We have been requested to contain and eradicate the computer worm that was infecting the clients network and at this stage forensics, to identify the origin of the infection, had not been conducted yet. This whitepaper contains additional details about the attacking-worm itself, as there are clear indicators that data had been siphoned towards the attackers.

This document is the result of that analyses and contains a graphical summary as well as a technical overview of the incident, including behavioral analysis, performed by us, of the malware.

The Activity in Numbers

We deployed the Cleanup Application (CCA) with a special module developed for the QakBot.T worm. The CCA was deployed through SCCM as well as manually.

The cleanup statistics and coverage from a few days after the deployment of CAA are as below.

The ‘Total Executions’ count is the amount of times CCA had ran on all machines in the network. 

The ‘Unique IP’ count the unique count of local IP address, which are present inside the log files. This gives an indication of the SCCM coverage.

The ‘Total Locations’ count are the number of external (public) IP addresses submitted to the server, this indicates where the client employees are using their computers.

The ‘Total Current User AppData Microsoft Folder’ count shows the number of files and folders that have been removed by the CCA. This cleanup is performed based on a whitelist, and can include some valid files.

The ‘Total All Users AppData Microsoft Folder’ count shows the cleanup performed as a high privileged user, cleaning out other user folders who are not currently logged on in on the workstation.

The ‘Total All Users windows XP’ count is the total removal count of files and folders on Windows XP machines, if they were present in the network.

The ‘Total Current User AppData *.LNK files’ count represents the amount of *.lnk files removed for the current user.

The ‘Total Windows XP AppData *.LNK files’ count represents the amount of *.lnk files removed on Windows XP machines.

The ‘Total All Users AppData *.LNK files’ count shows how many *.lnk files have been removed for all existing users folders on the system.

The ‘Total *.exe files in PRINT$’ count is an important metric. This shows how many persistent infections are removed. This number is stable since the deployment of CCA and should not increase over time.

The ‘Total *.exe.cfg files in PRINT$’ count is similar to the above metric. The file belongs to the .exe file itself and was used by the worm to drop the executable itself.

The First Numbers

-----------------STATS---------------------
Total Executions   : 50862
Total Unique IP    : 305
Total Locations    : 67
-------------------------------------------
Total Current User AppData Microsoft Folder       : 3403
Total All Users AppData Microsoft Folder          : 578
Total All Users windows XP                        : 0
Total Current User AppData *.LNK files            : 1
Total Windows XP AppData *.LNK files              : 0
Total All Users AppData *.LNK files               : 14
Total *.exe files in PRINT$                       : 46
Total *.exe.cfg files in PRINT$                   : 42

-------------------------------------------

The latest summary report (numbers on the right side of the graph) of the top 1000 AD lockouts for the past 24 hours shows only 9 lockouts. These 9 lockouts were unrelated to the infection outbreak. This is compared to the 4005 lockouts at the left side of the graph, which happened in one hour when we just arrived on site.

The first red line shows the start of manual removal of the worm through the CAA. The second line represents the start of the SCCM deployment of CCA on all workstations. 

Lessons Learned

It is important to reflect on the incident and identify which lessons have been learned during and after the incident. It is recommended that these lessons learned are embedded into the organization, to prevent similar incidents from reoccurring.

During the identification phase, it was noticed that the origin and exact start date / time could not have been identified easily due to the absence of network traffic logs on firewall / switch level. This prevented the possibility on forensic analysis and resulted in a reactive hunt for the worm instead of a proactive chase.

Hourly reports on computer lockouts were provided to identify active infections. Some of the employees had network traffic capture software running on their own workstations. This enabled us to obtain valuable malware samples that led to the positive identification of the QakBot.T worm.

Network Traffic intercepted during the malware analysis shows that the worm connects and submits encrypted data to several different web servers. It also uploads an encrypted zip file to an FTP server. The contents of the data submitted is unknown as the data is transferred in an encrypted manner. No attempts on decryption have been done during the incident response, as the main objectives at this stage were to contain and eradicate the computer worm that was infecting the clients network.

The worm appeared to be restricted only to one specific site of the client, while infected workstations in other regions were identified and proved to be infected while they were connected to the network in this specific infected site. It has not been possible to confirm the spread of the worm onto other client locations. The underlying reason for why the infection does not seem to spread around has not been fully investigated and it might be related to the way Internet access is arranged in other client locations. It seemed that the infected site was the only site with direct, non-proxies, Internet access. It however has not been confirmed this was the reason.

We implemented and deployed its cleanup application onto the client workstations. This cleanup application includes a specific module that has been tailored-made to target the QakBot.T infection with the client specific settings.

The Cleanup Application (CCA) was deployed through an SCCM package onto the client workstations as well as manual implementations.

All in all, the CCA showed a stable cleanup rate for the last 48 hours (having cleaned up 46 unique workstations without any reinfection).

We continued to monitor the infection status for additional two weeks and provided additional recommendations, including when to remove the CCA from the workstations. The removal of the CCA is best to be performed by CCA itself, where we issued a “cleanup call” to the application that will remove itself from the workstations that are connected to CCA.

Recommendations

Ø After (cyber) security incidents, it is always recommended to have a lessons learned / post mortem meeting with all involved entities, to walk through the incident and ensure that follow-ups with concrete actions are in place and prevent similar incidents from reoccurring.

Ø Additionally, it is strongly recommended to implement a centralized network logging & monitoring solution that identifies malicious threats and activities within the network, to prevent malicious outbreaks like the one contained and eradicated. Having such monitoring solution also helps with forensic analysis and origin-tracing.

Ø It is recommended to review the formal incident response process and enhance it with a globally operating CERT and experienced CIRT in accordance with the ISO27035:2016, to create a governed process in regards to incidents. This will enable better coping with incidence and alliance with the upcoming GDPR as well.

Technical Analysis

How the threat was identified

The first initiated steps were to get a clear understanding of the problem and to verify its root cause, so that the business impact could be significantly reduced and eventually eliminated. A plan of action, which would be followed and implemented over the following days, was created on Thursday.

Upon arrival in the office of the client the C$ and the ADMIN$ shares have been disabled by the network team, trying to prevent further spreading of the worm inside the network. PRINT$ and IPC$ where still enabled.

The client had sent several samples of the worm; however, the zip files became corrupted during transport. Therefore, the CIRT obtained several new samples from infected computers. The worm appears to be running with local system rights, meaning that the worm is installed with local administrative rights, otherwise the Scheduled Tasks that were installed by the worm could not be created.

The samples revealed 4 files with random names, the executable was a 35mb file with a random file name inside the user’s %APPDATA%/Microsoft folder. First analysis of this file indicated it as a malicious one, which was not detected by many Antimalware programs – only 10 out of 61 tested programs.

The file is installed into windows as a service from %APPDATA% and PRINT$, running automatically in the background on the infected computers. This shows the persistent angle of the worm.

A second file location was found inside the PRINT$ share inside the network. This file is smaller in size and is spread by the worm itself throughout the network.

When performing behavioral analysis on the file, it is confirmed this file is the core worm and is responsible for downloading and creating other files inside the computer and registry. It also establishes several RPC Pipes, which are used to write files. The images below are two examples out of the behavioral analysis performed on the second file.

This file is tagged by multiple Antivirus programs (36/61) at the time of analysis and is part of the QakBot.T infection. This was the evidence that was needed to positively identify the worm and its hiding locations.

A behavioral analysis on the file shows the execution flow of the malicious application. This shows how the worm installs itself onto the computer via several different steps (including creating scheduled tasks to check on its existence).

Network traffic during the analysis of the payload shows that the worm connects to several websites, posting data to them. The contents of the sent data are unknown, as it is sent to the web-server in an encrypted fashion. Initial attempts to break into the web-server to get access to the data have been unsuccessful. No attempts have been made to decrypt the encryption mechanism.

Besides posting data to webservers, the worm also connects to an FTP server, uploading to it an encrypted zip file. The contents of the zip file are unknown, as the encryption key was not researched.

How the threat was contained and eradicated

By identifying the locations of the malware and how it persists itself, a cleanup application could be created by us.

The CCA for the client was deployed to remove the QakBot.T worm from the network. Due to the nature of the client network, several test runs were made, to ensure that the removal application works on all types of machines, 32bit and 64 bit. The application has been tested on Windows XP, 7, 8.1 and 10. Both 32 and 64 bit environments, with and without internet access.

The application has been deployed through SCCM and some manual installations on workstations and runs every 15 minutes on the machines through a scheduled task created by the Cleanup Application.

A statistical analysis from our side a few days after initial deployment is shown below

As can be seen in the above image, there are 260 unique machines connected to the removal tool (these machines have been identified with internal IP addresses and all have an internet connection). In total, the cleanup was executed 24.135 times from 37 unique locations (which are Internet IPv4 addresses).

The difference between the SCCM numbers and the Cleanup Application (CCA) numbers could not be explained. It is 100% certain however that the CCA numbers are unique connections and include a few test machines and manual installations. It would seem likely that not all the client machines are connected to SCCM.

There have been found 46 instances where the PRINT$ share contained malicious files that have been cleaned up. This was a stable number over the last 48 hours, which indicated that there were no new infections showing up in the network.

The cleanup in AppData_Microsoft is done based on a whitelist, removing only folders that are not on a whitelist. It can therefore be that this cleanup number included valid folders. The best KPI for cleanup is the PRINT$ (printer_exe) metric.

The statistics below are 5 days after the initial deployment:

-----------------STATS---------------------
Total Executions    : 50862
Total Unique IP     : 305
Total Locations     : 67
-------------------------------------------
Total Current User AppData Microsoft Folder        : 3403
Total All Users AppData Microsoft Folder           : 578
Total All Users windows XP                         : 0
Total Current User AppData *.LNK files             : 1
Total Windows XP AppData *.LNK files               : 0
Total All Users AppData *.LNK files                : 14
Total *.exe files in PRINT$                        : 46
Total *.exe.cfg files in PRINT$                    : 42

-------------------------------------------

Follow up during recovery

The CCA had ran at the client locations for several weeks, updating when needed.

It is essential that all infected machines will be cleaned, as missing a single one can result in a relapse and reinfection of the entire network.

If your company faces malware infections or if you require assistance during major incidents please let us know. We are here to help you and to ensure that you will have a good night rest once again. We will protect your valuable assets.

Obviously, feel free to reach out to me via LinkedIn as well 🙂