By Sam Elliott
An unprecedented loss of city data has Lawrence officials looking for what caused the crash and what the city can do now to recover and prevent further future damage.
Lawrence’s system running Logos software, made by New World Systems, experienced a crash Nov. 7, and the city lost nearly three years’ worth of data. Digital forensics investigator Robert Olson, of Input Output Solutions in McCordsville, presented his findings to the Lawrence Common Council during a special meeting Tuesday, Dec. 29.
The New World Systems was the victim of several things,” Olson said. “It was the victim of poor maintenance, people not reading the server logs, not understanding the maintenance routines and maintenance logs, not being able to diagnose and anticipate the needs of the City of Lawrence, it had not been properly maintained and the updates had not been applied.”
Lawrence’s New World System software had not been updated since November of 2014, and was running the 9.0 version of the software. Two updates had been released since then, but not installed by city IT staff.
“On Nov. 7, for some unknown reason, the contention is that they went back to install the 9.2 release, which was obsolete by two releases at the time they chose to install it,” Olson said. “What’s strange about that also is on the 6th of November they had actually requested from New World Systems the latest version. It arrived here on the 10th. So if you’re going to install a software version update and you order the latest version to be delivered on the 10th, why would you go in on the 7th and install one that’s two generations old? It doesn’t seem to make sense to me.”
Olson told the council that in the history of New World Systems software, a crash of Lawrence’s magnitude has never happened and crashes that have occurred have been able to be remedied within two hours.
“At this point in time, what happened on the 7th of November is to me a bit of a mystery because I don’t believe it had anything to do with the New World Systems2 update, and that leaves a lot of room for speculation,” he said.
Had Lawrence IT staff waited until the following Monday morning after the Saturday crash, New World Systems could have had the city’s system back up and running in two hours. Instead, an attempt to install a backup — later found to be one of 150 corrupt backup files dating back to December of 2012 — resulted in the permanent loss of the city’s data.
“’Catastrophic’ was the word I heard used most of all,” Olson said. “The likelihood of 150 backups being corrupted out of 150 is extremely small, especially since the procedures should be that you periodically verify the backups to make sure you are in fact getting good data.
“The one thing that actually saved you from having no utility data, no permits, no information at all — and the only thing that saved you — was there was one copy sitting on a New World Software server that they had used for testing purposes that was from the 8th of September,” he added.
Olson recommended additional training for Lawrence IT staff or outsourcing the city’s system off-site entirely in addition to creating disaster recovery plans, and will send his full final report to city officials in early January.
