Bank Battles Virus

[See the timeline for an hour-by-hour account.] How would your operation be affected if a major virus disabled your network? That's what happened recently at one of the largest banks in the Midwest, National City Corporation (NCC). Here's how NCC worked with IBM to successfully stop the virus and keep the bank's operations running, from a recent interview with NCC Executive Vice President Jon Gorney by the editors of Application/ Enabling World. National City Corporation is a $50 billion commercial bank holding company headquartered in Cleveland, with about 900 retail offices in Ohio, Kentucky, Indiana, and Pennsylvania. NCC's computing network of about 300 servers and 8,000 workstations handles both mission-critical and supporting applications. Like many organizations, NCC had taken prudent steps to prevent viruses. Virus detection software was in place on workstations throughout the network. All new equipment was checked for viruses by the PC dealer before delivery, then rechecked by NCC before installation. Employees were prohibited from bringing in software from home. Despite these precautions, a virus infected NCC's network last spring. With ingenuity and the help of IBM's Virus Emergency Response Team and AntiVirus software, NCC was able to solve the crises with minimal disruption to bank operations and absolutely no impact on customers.
The first signs of a problem
Late one Tuesday afternoon, users trying to log on to the NCC network were suddenly being denied access. NCC's IS staff soon realized this was more than a set of isolated incidents. It was the start of a serious virus infection and it was spreading through the network at an alarming rate. Jon Gorney, NCC Executive Vice President, was at a hotel in Pittsburgh when he was alerted to the problem. "I got a call at 2:00 a.m.," he recalls. "A virus of some kind was totally locking out the workstations when you tried to sign on. It was clear that when people came in the next morning and signed on, it would spread even more rapidly." Gorney quickly assembled other NCC staff members traveling with him and secured a hotel conference room for planning and communications. Meanwhile, NCC executives in Cleveland called their IBM Client Executive Don Parker at 2:00 a.m. to ask for help.
IBM's local team and virus experts respond at once
Parker went into action immediately, marshaling local IBM resources and IBM's expert virus-fighters in New York. Before daybreak, the local team had captured a sample of the virus and transmitted it to Ed Hahn, manager of the IBM Virus Emergency Response Project. Hahn and his staff went to work to identify the virus and find an antidote.
"By 5 or 6 in the morning, we were able to have our technicians talking to IBM specialists and transmitting software with the virus embedded so that some analysis could begin," Gorney explains. "By early morning, IBM was able to come back and say, 'Yes, it's a virus and we've identified it as MAJO-1644.' That was the good news. The bad news was that a disinfectant did not exist."
NCC executes its communications plans
Meanwhile, National City's staff was rushing to keep the virus from spreading and trying to develop workarounds to accomplish the bank's day-to-day functions without affecting customers. Effective communications was critical. "As part of our disaster and recovery contingency planning process, [we had] a very good understanding of the call matrix -- who to call and how to contact them," Gorney comments. "That worked extremely well. There were signs posted: 'Do not sign on to your PC.' There were broadcasts over the internal PA system. Our support teams at the local level were all aware of it so we had assigned a person per floor or department to be the spokesman. It was a combination of talking to individuals who would then communicate to our contingency planning teams [and] business units."
The real challenge: Deploying the disinfectant
By 11:00 a.m., IBM technicians had developed an antidote and transmitted it to NCC. While IBM was working on a solution, the supplier of NCC's installed anti-virus software was also trying to find an antidote. NCC evaluated both options and elected to try the installed vendor's solution. But problems with the other disinfectant emerged almost immediately. NCC decided to deploy the IBM solution and asked Ed Hahn to come to Cleveland to help. The NCC corporate jet, which had already retrieved Gorney from Pittsburgh, was dispatched to New York. By 6:30 p.m., Hahn was on his way.
"Now the good new was, we had a cure," Gorney remembers. "As it turns out, the disinfectant was the easy piece. Orchestrating how you go about getting this rolled out...became the challenge. "It took some senior people within my organization and within IBM to get in one room and say, 'What do we need to do? We need a strategy to roll this out.' It was clear there was not going to be a quick fix." The team moved several high-powered workstations to a conference room to create a data center as the central point for distributing the antidote and cleansing servers. But inventory and asset management issues hampered the cleansing process. "To be perfectly blunt, we did not have a good inventory and asset management system in place," Gorney states. "You couldn't easily [say], 'Here's where every server is physically located.' Nor could you say, 'Here's the exact configuration of every workstation.'...The process was very complex and very difficult and required rotating teams in and out about every five or six hours for 48 hours." Ed Hahn also recalls the complexity of the deployment. "As we came into contact with servers they had cleaned up with the other solution, the virus was gone but the programs wouldn't execute and we couldn't get the servers up," he explains. "The easiest way was to restore the infected servers and let IBM AntiVirus clean those infected programs." AntiVirus detects more than 7,500 viruses. During the deployment, other viruses were discovered and resolved.
NCC staff works to keep the bank running smoothly
Meanwhile, another NCC team was ensuring the bank operations could continue. "While all that was going on, we had another team dealing with business disruption," Gorney recalls. "[They] went application by application, city by city, business unit by business unit, and we had contact points in all our various cities to try to define, 'Okay, what can't we run? What kind of workarounds do we have? "That team did an outstanding job. They were able to...develop a workaround for every single application. There was no disruption to customers that we're aware of. It didn't have any direct customer impact, through a lot of hard work."
The final test: Bringing the network up
When all servers were cleansed, it was time to allow users back on the network. The plan was to have IBM AntiVirus cleanse each workstation during the logon process. "Then the most difficult task was getting people comfortable with a different sign-on process," Gorney recalls. "A good part of Thursday night was [spent deciding] 'How do we communicate with 8,000 users about what they're going to experience when they sign on?" This was complicated by multiple PC configurations. "We pretty much know how it would work for a 486...but we couldn't go out and hand [users] a set procedure," explains Gorney. "We had to go back to our whole call tree again and have people back out on each floor helping with the sign-on process." By 10 a.m., the network was up and running and the expected flood of calls to the help desk had not materialized. NCC was back in business.
How the virus got in
National City later determined that the virus came in on a new PC installed at its distribution center the day before the outbreak began. Though the distribution center went through multiple steps to detect viruses in the new equipment, the virus got in anyway. "It was a real wake-up call for us," Gorney acknowledges. "We had a procedure that made sense, we were following that process, yet it still got through. We've challenged our team: 'If 1645 shows up tomorrow, given everything we've just done, what would our experience be?' My belief is that based on what we've put in, we may not be able to resolve it right away, but we'd recognize it, we'd isolate it, and we'd shut it down."
How the crisis strengthened NCC
NCC had done many disaster drills in the past to prepare for unexpected crises, but ironically, a virus infection had never been tested. "Every year, we do a simulation of a disaster in our major cities. Generally, it's a physical disaster, like a plane crashing into the data center," Gorney notes. "The interesting thing is that as much as we had tested physical disaster, the likelihood of that is probably remote compared to this." Gorney believes many benefits resulted from the virus outbreak and calls it "an absolute blessing. I'd never want to do it again and I wouldn't wish it on anybody, but the reality is that when it was done, we're smarter and we understand it better... We're more secure than we were.
"There was a point when I wasn't sure we could get out of it," he acknowledges. "But when I look back, it was the best experience we could have had. "We learned a hell of a lot. It challenged us to have tighter standards. The more we had out there, the more complex it became. It highlights the need to have greater central management of this asset and a true asset management system. "We had been far too accommodating, giving people far too many choices versus saying, 'If your job is a financial analysis, you need this kind of PC.' We are migrating to a much more rigid and tighter standard to try to eliminate the variability."
Gorney lauds IBM for its rapid response and customer service. "Their performance was outstanding," he states. "There were teams around the clock made up of National City people and IBM people and they would change from one shift to another and I honestly couldn't tell you, when I went into the room, who was who. That may be the best way to describe it." Many colleagues in the banking industry who heard about the incident have called Gorney for advice. He suggests organizations plan a human communications strategy and evaluate the strength of their anti-virus software. "You've really got to have a very strong communications plan already built," he observes. "I've also suggested they really need to take a look at what their anti-virus software can do. I was very open. I said I don't know how we would have gotten out of this situation if IBM had not been there."

Timeline

| Tuesday, May 14
| Sporadic logon problems are beginning to appear on workstations in the National City Corporation network. NCC technicians start working on the problem.
| The problem is now widespread. National City realizes it has a serious situation that is probably virus-related. An alert/escalation process begins.
| Wednesday, May 15
| NCC Executive Vice President Jon Gorney is reached at a hotel in Pittsburgh. An NCC manager calls IBM Client Executive Don Parker for assistance. Parker begins rallying others at IBM.
| A conference call to discuss the situation is held between NCC and IBM. Tony Martinez, IBM Manager of Security Services, calls the virus emergency team in New York, inlcuding Ed Hahn, Virus Emergency Response Project Manager.
| David Jesse, IBM Software Account Manager, is on site at National City with the latest version of IBM AntiVirus. Jesse takes samples of the virus and sends them electronically to the lab in New York.
| IBM indentifies the virus as MAJO-1644. The search begins for a disinfectant.
| NCC employees begin arriving for work. Security guards and signs alert everyone: "Do not touch a PC." National City's business units devise workarounds to continue normal bank operations.
| The full local IBM client service team is on site. IBM virus researchers in New York are working on a solution with an expected completion of 11:15 a.m.
| IBM delivers a disinfectant, as does the vendor who had supplied NCC's existing antivirus software. NCC opts to try its current vendor's solution.
| The other solution has removed the virus from several servers but is crippling executable files in the process. Many servers now must be rebuilt. NCC decides to deploy the IBM disinfectant.
| IBM and National City staffers create a lab in the bank's operations center for testing and deployment. The NCC corporate jet is sent to New York to bring Ed Hahn to Cleveland.
| A plan for cleansing servers is developed and cleansing commences. After completing 38 servers, the team realizes the virus has infected the lab. The lab is disinfected and the process of cleansing servers starts over.
| Thursday, May 16
| Progress is slow. Network bandwidth is a problem. Staff exhaustion is becoming a factor. To reduce errors, two technicians are assigned to each terminal.
| Cleansing continues all day. The bank's network is still unavailable to employees.
| Cleansing is completed and all servers are protected. Everything is double-checked for problems and errors.
| Friday, May 17
| Technicians bring up a pilot workstation for the first time in a controlled environment. Problems are addressed, testing progresses, and the process becomes smoother.
| NCC executives decide it's a "go" -- the network will come up that morning. The team has two hours to get ready.
| Anticipating problems and confusion when users sign on, extra help desk coverage and technical support are put in place at all the bank's major locations. An IBM team in New York is standing by with a back-up help desk.
| Users are logging on in large numbers. During logon, workstations are scanned for the virus. Most users experience a delay of only two to three minutes. Virus penetration data starts coming in. Cleveland and Columbus workstations are heavily infected with other locations infected to varying degrees. Two other viruses are detected in the process and fixed by the software.
| The NCC and IBM team is amazed at how smoothly the network has come up. Problems are few and readily addressed. Team members begin to head home for much-needed rest. Ed Hahn and others remain to help NCC plan system back-up.
| Ed Hahn leaves for New York. The rest of the team is asleep. NCC is back in business and the virus is cured.