On August 11 at approximately 8:15pm, network monitors indicated a network distribution hub failure, resulting in a loss of connectivity to numerous buildings on city campus.** ITS networking personnel were called on site and determined the scope of the outage pointed to a distribution point in one of the city campus buildings. The distribution switch was examined and rebooted. At the moment it was powered down, network traffic was reassigned to its corresponding standby distribution point in the data center. Network service to the impacted areas was restored by approximately 10:15pm. All campus fire alarms remained operational through backup telephone lines.
“What this incident demonstrated was that our network architecture—all of our distribution redundancy and dual links to buildings, performed as designed.” says Jim Hlava, ITS Network Manager. “The issue remaining is determining why the failover from the assigned primary distribution to backup did not occur automatically. We have collected data from the router and other systems to gain a clear understanding of what happened, so this does not happen again.”
UNL has opened a ticket with the Cisco Technical Assistance Center (TAC) to isolate the issue. Cisco will help determine whether a hardware replacement will be issued or provide a software update to remediate the problem. Currently, network traffic for this sector is running through the data center switch while the issue is being investigated by Cisco. If a failure would occur in the meantime, the switch in the city campus building would be reassigned as the primary again.
“We have been using this type of of Virtual Switching System (VSS) distribution since 2009 and this is the first issue of this kind that we’ve experienced,” added Hlava. “This particular hardware was new this summer. We are confident Cisco will determine the cause. If it is a software issue, it will be a positive step as we have future installations that will benefit from remediating this issue.”
Amy Metzger, Data Center & Operations Manager is coordinating ITS Disaster Recovery and echoed Hlava’s statement. “An incident like this is important for us to use as an opportunity to learn and discover points of vulnerability—hardware, software and processes. It’s another step to becoming more resilient as an institution.”
For additional questions, contact the UNL Computer Help Center at 472-3970 or Toll Free (866) 472-3970 or email at mysupport@unl.edu
**Buildings impacted during the 8/12/2014 outage include the following:
501 Building
Alexander
Anderson
Andrews
Architecture Hall
Avery
Baseball
Behlen Lab
Bessey
Burnett
Canfield Admin.
CBA
Champions Center
East Stadium
Hamilton Hall
Haymarket Park
Hewit
Kimball
Lied
Love North & South
Manter
Morrill
Multicultural Center
North Stadium
Oldfather Hall
Richards
Ross
Sheldon
South Stadium
Stadium Parking
Temple
West Stadium
Westbrook
Wick
Woods