On Monday, August 19, Information Technology Services discovered and remediated a malware installation that impacted up to 1900 computers on campus. A successful team effort between ITS Help Desk Support, Security, Networking and distributed IT staff using a variety of technical tools lessened the impact of this intrusion. The following is a summary of lessons learned In the interest of transparency and to prepare all campus staff for the possibility of future intrusion attempts.
Not the average malware
Bitcoin is a digital monetary payment system. (http://www.youtube.com/watch?v=Um63OQz3bjo ) It uses technology that harnesses distributed processing power to facilitate for-hire computing tasks. Bitcoin processing uses legitimate Windows protocols that are neither detected by intrusion prevention systems (IPS) nor by antivirus software. While there is debate as to Bitcoin's legitimacy in the marketplace, the unauthorized use of UNL staff credentials to distribute Bitcoin mining software on University-owned computers via the UNL network, is a violation of University Computer Use Policy ( http://www.unl.edu/ucomm/compuse/ ) and defined this activity as an intrusion and the installation as malware.
Discovery
A user reported a slow responding computer; not a scenario that immediately sets off alarm. This user was technically astute however, and relayed to a Help Center support technician that virtually 100% of the computer's processing power was being utilized for an unknown process. The support technician was able to remote into the computer and noted the unusual activity. Using UNL's client device management tools, the technician observed the same process occurring on multiple systems across campus. The security team analyzed network traffic flow to look for unusual activity and an originating source. This pinpointed a computer from a completely different department that was acting as a distribution point for the malware. The tools further helped to review system administrator login activity. It was then discovered that the malware was installed via a system administrator's compromised credentials.
48 hours
The following actions were taken on Monday, August 19 within six hours of discovery. The infected distribution computer was blocked, removed from the network and stored for further forensics. The compromised credentials were disabled. All ITS system staff changed passwords. A script to stop and remove the malware was written by ITS staff and quickly distributed to infected computers via the device management tool. Further monitoring later in the day revealed that Bitcoin activity halted on a majority of the machines.
By Wednesday afternoon, ITS staff had worked with techs in affected departments to reach other non-managed computers. The remaining handful of machines running Bitcoin ceased activity. All identified Bitcoin outbound traffic was blocked. An additional script was sent to scan infected machines for Personally Identifiable Information (PII) that could have been compromised through the malware. Results are being analyzed but it is generally agreed that Bitcoin impacted no data. Forensics are ongoing to identify how the credentials were compromised and to further monitor and search for any hidden malicious activity related to the Bitcoin intrusion.
Lessons learned
There are a number of takeaways not only for Information Technology Services, but also for departmental IT techs and the campus. The response was swift, decisive and there was excellent cooperation between ITS and distributed IT staff on campus. It is important to note that UNL was able to respond and mitigate the issues effectively as a result of the security and client device management tools, specifically KACE, that we now have available.
The following are key points for the future:
• ITS System administrators will be using temporary/throwaway credentials when logging into machines outside of their immediate control
• Two-factor authentication is strongly recommended for any account with system administration privileges
• Limit firewall access to as small of scope as possible, instead of all of campus.
• Passwords should be changed regularly to limit the possibility of compromised credentials
• All system administrators must keep computers and servers updated and patched
• Everyone has a part in ensuring security--Always be alert for suspicious activity
• People, just like computers, store, process and transfer highly valuable information. Take advantage of UNL's SANs Securing the Human series at http://security.unl.edu/video-training
If you have any security questions or concerns, contact the ITS Security team, security@unl.edu