In December 2022 Arnold Clark was the subject of a cyber attack which nearly brought the company to its knees. This month at the NFDA Driving Automotive conference former CEO Eddie Hawthorne gave a blow-by-blow account to delegates of the actions the company took to survive and the lessons learnt
This month Eddie Hawthorne spoke at the NFDA annual conference Driving Automotive on his experience of a cyber attack in December 2022 which crippled operations and took six months to get back up to speed.
“We had a devastating cyber-attack that brought out business to a shuddering halt in December 2022. At that time, we used about 1,400 servers, a mix of on frame and in the cloud,” he told delegates at the conference.
“Our digital department has about 220 people in it, 12 of them were cyber-security experts. We spent about £22m on the wage bill for that department and our annual capital spending on IT infrastructure was £10m to £15m. So, as a CEO, tick that box, got that covered. That’s not a problem. How wrong could I be?”
Industry veteran Hawthorne told delegates that despite having systems and investment in place, it still happened.
“What I learnt – and about 150 companies that day learned too – is you must be lucky all the time. These people only must be lucky once. It is not if you are going to be cyber attacked, it is when. It will happen and it is how you respond to that. So, 23 December 2022 is a day that will be in my memory. It happened about half past six at night. I got a phone call from the CIO saying “Boss, we have a wee bit of a problem right here.”
The CIO had been alerted to the fact that a number of files had been deleted on the system. The deleted files alert was the people putting ransomware on the system and deleting the last file they used to gain entry. Arnold Clark took the system offline.
“We could not find it because we took the servers offline, so we knew something was going to happen, but we had to wait and see. It was man against machine. Out team was there, fighting it, trying to disable VPN access trying to disable a lot of things but it was man against machine.
“Once we shut something down, something else appeared.”
About 2.50 am they [IT staff] told Hawthorne they had just about lost control of the system which would have meant waking up with a big blue screen in the morning. But they persevered and won.
“Well, we managed to pull the plug, everything stopped, five to 10% of our servers were encrypted but the attack stopped,” said Hawthorne.
On Christmas Eve Hawthorne was told the system could not be plugged in because of the risk of Trojan software on the system.
“We genuinely could not plug in anything. We split our team into two to find out what happened, how they got in and start to bring back our servers. Bringing back servers is not a quick job, it takes a long time. So, we were going to be down for a while.”
Hawthorne said the company had contingency plans in place but had not tested for everything going down at the same time leaving the company powerless.
“No email, no systems, no method of communicating with customers, no access to the internet, no diagnostic machines, no finance, nothing. And we were opening the next day with 700 people coming in to pick up their cars, 2,000 people coming to get their car serviced and 1,500 people coming to hand in their rental vehicles. And that was just the tip of the iceberg.
“How did we survive? Well, again the motor industry we work in small markets we are resilient, we are adaptable and that is what you have to take out of this because everybody mucked together. There was a mixture of teams, mobile laptops and paper and pencil and calculators. Old school because we needed that. We had no phones for two days and we had no system at all. Our systems were re-built in 10 days and got back to about 30% utilisation,” said Hawthorne
According to Hawthorne, one of the most difficult challenges was staff management.
“If you have 12,000 that come to work that have no idea of what is happening, no way of telling them this and you have got to keep them busy. Our staff did not know what to do because we could not communicate with them. That really was a big challenge for us because you cannot bring back your systems all at the one time. It comes back gradually. Picking what systems came back was a real challenge for us. Now, obviously as a businessman you are going to being back those that make the most money
“We had all our systems back after about six months. Some of our staff did not get their systems back for six months and a lot of people forget the human impact on your staff. We lost a lot of good people who were stressed. Speak to the people at Marks & Spencer, they went through the same thing, there are a lot of people who worked really hard here. If you are every unfortunate enough to join this unique club, remember your staff,” he said.
By mid-January Hawthorne got another jolt with the Russian ransomware group telling him they had had stolen some data, too.
“This was data that was on head office servers, and I thought it was encrypted but as a CEO I now know what encrypted at risk means. As long as you do not load the file it is encrypted but as soon as you load it, the encryption falls off. So, they do have a wee bit of information, but what these people do – and they are very good at what they do – they copy information they don’t take information. They leave it where it is and then delete the last file they used to get in,” he said.
“The police told me it is a bit like someone breaking into your house, hiding in your loft photocopy all your documents and then leave and change the locks as they are going out. So, you must piece it together.”
So, what to do. On the one hand you do not negotiate with criminals, on the other you must find out information they have.
I am not going to pre-judge anyone who goes through this. People who have paid a ransom usually do so for a reason, just to get their business back. The thing about this you are funding crime.
“So, we negotiated with them because we had no idea of what they had. And we had to find out. They were sending us pictures and videos. They are IT specialists and they are not negotiator, so we managed to spin this out for about two months and managed to piece together what they had.
“We then told the customers, negated the blackmail threat and protected our customers but that took a lot of time and effort and in the meantime we were trying to get the business up and running and our year end audit was going on at the same time and the auditors were asking why they could not access the systems.”
Hawthorne said the company learnt from the experience.
“What did we learn, we learnt an awful lot. Cyber security is a journey. If you think it is won and done, then don’t because it is ever developing, it is something that needs to be on your agenda, not necessarily at the top but very near the top,” he said.
The other thing he learnt was the important of speed of response,
“How quickly you can get to a cyber incident is how you defeat it. When 2022 came along, it was probably 12 to 18 hours it took us to do a response but now today we are down to minutes.
Hawthorne said keeping the exterior clean was a must.
“They look at reconnaissance. So, if you can keep the exterior of your systems to a minimum so people cannot see how to connect to what you have got, that reduces the risk.”
But once they had done reconnaissance they then land with credentials, whatever that is that is how they get in and expand through your systems with accelerated credentials.
“We have now developed a kill chain methodology with our partners and that allows us when somebody gets in and beaches one barrier, we shut it down. We now have security to take priority over operations. So, if there is something that is not right, we shut it down. And then we look at it, make sure it is OK before we turn it back on again. That is difficult to do in a fast-growing business, but I would urge you to take that approach. If you see something that should not be there, it should not be there for a reason.”
Eddie Hawthorne said here too employees were a critical element of the fight against cyber-crime.
“The other thing was we brought our staff on that journey. So, going back to that phishing exploitation, bringing your staff on that journey is a must because they need to understand what happened.”
Hawthorne said phishing email training only went so far. “People switch off,” he said,
“You need to bring them with you, so we have a red button on our email so if our staff think this email does not look right, they press it. Last year, they pressed that button 49,000 times. 960 of those times were malicious emails with software aimed at doing us damage.”
He said emails are masquerading as suppliers and partners which he would expect to receive emails from.
Hawthorne said the company is not getting so many phishing emails alerts, but they are now more targeted masquerading as suppliers and partners.
“So last year we had 30 emails from suppliers and partners, last month we had 150. So, it was 150 from people you were expecting to get emails from that are not actually the people you think they are. The worrying thing for us is that five or those partners don’t even know their systems is compromised until we tell them.”
Hawthorne wrapped up with a warning to dealers. Cyber security is a must; you must look at it. It is something that is real, it is a business risk. If you cannot afford to do it [tackle cyber- crime] you probably cannot afford not to do it.
