Stefan Würtemberger is Vice President Information Technology at Marabu GmbH & Co KG and member of the CyberCompare Advisory Board. In his more than 20 years of career in IT of industrial companies and their protection, he has experienced many different IT situations. He talks straightforwardly about the two ransomware attacks on Marabu and how the company was able to cope with them. He has also created a best-practice action guide for affected parties based on his experience.
In the following, an interview with Stefan Würtemberger is published, which was specially made for Bosch ConnectedWorld (BCW) 2022.
In the interview, Simeon Mussler, COO Bosch CyberCompare, talks to Stefan Würtemberger about his experience with two ransomware attacks as well as his lessons learned. What happened? How does a cyberattack take place? How was the IT infrastructure rebuilt? How did IT and the entire company handle the attacks? In case of ransomware attacks, pay – or not?
Stefan Würtemberger answers these and many other questions in detail and provides tips and advice during the interview.
Below you can find the official interview (option to play subtitles and read 🇬🇧 English translation):
Dear participants,
we welcome you to our virtual stand at Bosch ConnectedWorld. We are pleased that we can present a very interesting interview partner, with which we will cover some topics in the area of cybersecurity, who will especially take us along on the events he had to / was able to live through, in a ransomware attack, or in the meantime even two ransomware attacks, and he will tell you a bit about how that happened, how he rebuilt the IT and what lessons he learned that might be interesting for you too.
Perhaps allow me to briefly introduce ourselves as Bosch CyberCompare. In contrast to the other Bosch locations and other exhibitors here, for us it’s less about selling a product and more about being a purchasing platform for cybersecurity solutions and products.
We have simply noticed that IT departments in the subject breadth of digitalization and cloud migration also have to work on the topic of cybersecurity or information security and are of course relatively overloaded. So why should every company start from scratch when it comes to IT security and make up their own opinion and try to see through the supplier market?
On the other hand, we have a very broad provider landscape, strong developments, new tools, new abbreviations, through which advertising materials fly, from EDR, then it becomes an XDR, then there is also an NDR, and if I then also choose a service, I have an MDR. And to see through this whole range is simply a big challenge and is also a bit inefficient if every IT department does it for itself. That is why we are there to accompany these tenders. We do two things in the process:
- One can be that we make an audit with you, where we say, well, based on our experiences, what should you do next.
- But if we start a specific product/project together, we would go through the entire tendering process with you, which means we start with standardized specifications, why should everyone think about what they are doing and create a Word document themselves. Thanks to our knowledge of the market, we can already say quite well which providers are suitable for this, but as independent platforms we are always ready to take in or let out other providers.
We work entirely on behalf of our clients and have no financial ties to the providers themselves. We bring our market knowledge into play, because that’s what we do every day with almost 250 customers in the DACH region. The tendering process is anonymous, and the final result is actually a decision document, where we say we have surveyed the market and we can recommend which provider you should take. Of course, the decision lies with you, your management together with the purchasing department.
That’s us. You can find more information in our online stand and our colleagues are also happy to answer any questions you may have. You are also welcomed to use the chat function.
Enough about that. Now let me introduce our guest for today: Stefan Würtemberger. I’m really looking forward to talking to you. Maybe you want to say two sentences about yourself first.
Yes, thank you for the invitation to participate virtually here at Bosch ConnectedWorld. My name is Stefan Würtemberger. I am responsible for IT at Marabu. We are a paint manufacturer, active worldwide and yes, we were hit twice with ransomware and we just want to talk a little bit about it today.
Thank you, Stefan, and first of all it’s great that you’re talking about it. It’s not often the case. Maybe you want to start looking at the first case, what happened, what has occurred, how did you find out about it?
Yes, the first case started on 29 Nov 2019, a Black Friday. It was really “black” for us. After six and a half hours, nothing worked at all. Almost 95/96% of all IT systems, from office IT, cloud to OT, were encrypted and, yes, it was really a brutal intervention in operations.
By chance I found out about the whole thing in Spain, when shortly after six in the morning the production department called and said, “Something’s not right here. Weird production papers with hieroglyphs come out of the printer.” We then started to check out what the situation was.
At first, we didn’t even know it was a ransomware attack. We just thought a server system was broken and over the six hours it turned out that we had a serious ransomware attack and then the ransomware was spreading more and more in the countries. From that point of view, it hit us brutally.
And what were the first steps, so what do you do when you realize that everything is ultimately blocked? How did you start there?
Well, at the beginning you don’t even know, ransomware, yes/no. You can’t see that partly, at least we didn’t see that. What have we done?
At least we did the first things of IT, we took a look, shut down the server, changed VMware, until we realized “Oh, it’s a cyberattack.”
Then very clear procedure. Crisis team convened, inform all department heads, if that was still possible, because the telephone was gone. Then there was the fallback to cell phones, who can you reach, who can’t you each? And then it actually started in IT, shutting everything down, that could be done. A bit fatal in some countries, the colleagues then simply ripped out the power plug out of sheer desperation, and it wasn’t so optimal, but of course shutting everything down. Review the situation, crisis team meetings, forensics, calling in IT companies, estimate the extent.
These were the first steps of the first days. Yeah, wasn’t the easiest of times.
I believe that immediately. And now looking beyond IT, how is the entire organization? How did she deal with it? I believe that this is also a dramatic experience, not only for the IT department, but also for the other employees.
Yes, a cyberattack really is a corporate event. Depending on how heavy it is, and in our case, it was really global, there’s nothing working. And the IT department clearly tries to solve it technically, but all other departments are also faced with huge problems.
Starting with the gate does not open anymore, trucks can’t drive in anymore, you can’t unload anything. That was hell for all the departments, of course, because you can’t get anything moved around the house. Production stop. Although production actually continued at our plant. We have a very analog manufacturing, but that continues into storage. You can’t get anything in/out of the high rack. You can’t find the goods. You can no longer communicate either because there is no e-mail, no telephone, no EDP.
That’s why the departments have to implement their emergency measures themselves, which was a bit of chaos for us at the beginning until it sorted itself out and today, you’re just very dependent on IT, but it’s not just a pure IT problem, it’s really an organizational problem.
And how did the first three/four days look like? So from, “I know it happened” to at least having some clarity and a plan forward.
Yes, well, the first few hours looked bad, because what are you doing? You can’t inform anyone.
Well, what did our boss do? He wrote handwritten notes and for the time being ordered a shutdown.
Then on Friday we sort of made the first assessment of the situation. Of course, you are also asked: “Yes, how long do we estimate until we can work again?” Then you’re very optimistic and say: “Yes, Monday/Tuesday everything will be fine again.”
Unfortunately, it wasn’t the case that things went so quickly and then you first have to inform all these employees, form telephone chains – in our case we didn’t even have phones anymore, just smartphones. But all the contact details were no longer available because they were on Outlook. You have to create chat groups from your private contacts and bring all employees up to date. And that was actually the first challenge on Friday.
On Saturday, only the IT was actually in operation. Then the police were there, the forensic experts were there. Then you have to find out what actually happened. How did that get into the company? Because you have to know that first before you can then initiate protective measures and recovery. Takes a long time and then we tried to get backups working again. When we then noticed that they were also compromised, we had to find other solutions to get IT running again.
The first six days until careful ERP operation was possible were relatively exhausting. Everything is sorted out afterwards and you then have a better overview, but then you also have to set priorities. What system is needed to be able to do what? That’s the order of priority of restoring, of recovery. Then, of course, it also goes into the countries. What needs to be done, which country, when? That has to be organized in the crisis team and then communication rules have to be set up. The first 24 hours are decisive in the cyberattack, which regulates the basic framework. After that, processing the incident is not quite as stressful and difficult. But the first week is hell.
And a question that of course many companies then have is: “To pay or not to pay?” Which can also be an existential question. What was it like with you? How did you proceed? Was that clear from the beginning or how did you approach this decision?
From our side, it was actually quite clear: We don’t pay. The company can only answer that for itself, do I pay or not. Paying ransom in cyber extortion may not always be the right decision. The hackers may not be able to decrypt everything either, it depends on the professionalism of the hackers. It’s a business case and of course they’re trying, but the question of paying is a very difficult one.
For us it was clear, “Yes, we don’t pay.” But over time, the longer that doesn’t work, the more concerned you naturally become. Do you maybe pay for it to stop faster? Of course, you always have to keep your options open. Our Compliance says no ransom will be paid, but you still have the option of passing compliance and saying, “okay, if it then becomes so critical for the company that it can’t be done without payment, maybe you have to jump over your shadow and then pay.”
But at the same time, if you leave this option open, you have to prepare everything in order to pay. It’s all in crypto. “Do I have the right account? Do I know how to convert real money into crypto? Do I know how to pay then?” That should also be on the screen, even if our credo was always “we don’t pay”, which we ultimately managed to do without paying a ransom.
What is also often the case: paying the ransom does not protect you from being attacked by the next group of criminals, because even the hacked accounts, are then resold on the darknet to make maximum profit.
Well, it’s a very political decision by the company, but we stand for it – better not pay. It is better to prepare yourself preventively for the whole topic. But as I said: Every company has to decide for itself.
Yeah, no use wagging your finger. How does communication with such hackers work, so do I then have a chatbot or an address or how do I even get in touch with them or were you in contact with them at all?
During the first cyberattack, we were never really in contact with them. We only did that very, very late together with the police.
Normally, it is chat rooms or e-mail addresses where you send an e-mail or where you have a darknet link to a web server. Looks a bit like WhatsApp, where you simply negotiate with them, where you then send files/patterns/examples, which they then decrypt and then it’s all about anonymous chats or in the second case it was email communication, where you then send e-mails back and forth. It’s very different.
Yes, and next maybe: once this incident has been dealt with as far as possible, how did you and the organization finally approach the reconstruction? What have you done? Have you thrown everything away and made new from scratch? Or how did you come to rebuild IT and IT security?
Well, with us the first cyberattack was almost a greenfield approach, if almost 95/98 percent of the IT no longer works, you naturally have a lot of chances. We then also took advantage of the opportunities for ourselves, then redesigned the security foundation, outsourced many services to the cloud, and immediately replaced legacy software with new applications. What you really can’t do in normal operation.
However, you have to look at it that way, after a cyberattack is before a cyberattack. You have a compromised network, you know all the security gaps through forensics. Then I have to rebuild it while the operation was compromised, back to a secure IT infrastructure and there were an awful lot of sub-projects afterwards. Of course, you do state-of-the-art security, which we have already done together with you. You then have SOC, you have SIEM systems, you have segmentation and so on.
Of course, all the technical things then come up that you can find on the market today: MDR and so on. But then it’s a hell of a way to go because, of course, you then have a running operation again and then build the IT on a future-proof security foundation around.
And if you look back on it from today’s perspective, now on this serious attack, before we come to the second, what did you learn from it, which might also help people who of course may slip into similar situations, so how can you prepare yourself? What did you learn, what should/could you have done differently or the organization? What you can now give to other participants.
The most important insight we have drawn is that companies should be aware of the topic of cyberattacks. Completely different processes are necessary during a cyberattack or another crisis.
If you take as an example a fire in the company, then I simply know: “Yeah, well, there’s a fire. I’ll call the fire brigade, an organization will come and extinguish it, then I’ll put it back together.”
This is not the case with a cyberattack. After all, it starts when a cyberattack hits. Who am I even calling? Where can I get the IT fire brigade to help me? A lot of people don’t even know that, and these are just things that you can clarify in advance, so when it’s needed, I know: “Okay, I’ll call him now, I have this contingent, I have the people who then also know the infrastructure.” And that’s one part.
The second part is to play through the whole thing and practice. That I just don’t walk into this situation completely naked, not knowing at all how I’m going to react now, who do I need, where does the crisis team meet, what are the tasks, who has which role? I can practice everything in advance. Of course, I can then do a lot of technology around it, and what is also very important is simply raising the awareness of the employees that cyberattacks are present, that cyberattacks are happening more often than ever, that they can have bad effects on the company and that’s just the whole issue of awareness. Just talk about cyberattacks in the company, be aware, no matter what size, no matter what industry, it can hit you at any time and those are actually the most important things you can learn from it.
And maybe about your role or the role of an IT manager in such a situation. In a situation like this, you are somehow the contact person for your IT department, which somehow wants to know what to do. You are of course the main contact person for other departments. “I can’t work, when can’t I start again?” The management will certainly ask once or twice when it will be possible again or what the next step is. How did you perceive this role, how did you manage it and where did you also take away or learn something for yourself?
Yes, the role is actually relatively fast. You become head of the crisis team because you have to provide information for everything. It’s an IT impact, yes, and the IT manager is the only one who can oversee the issue.
What is important, in the managerial role, you also have to manage, you have to organize. You shouldn’t fall into the topic, “Oh, now I’ll start installing it myself.” I think that is rather negative in the situation. I have been a spokesperson, the point of contact for everyone and it will also affect everyone else, because of course a lot of information then converged centrally around me. And then you have to process them, you have to distribute them, and many wrong decisions will be made. They must then be revised if necessary. You also have to document everything a bit and that’s sort of the organizer, the one who takes care of everything that’s going on around it. Of course, there are also many meetings where you always have questions and answers. You have to stay calm.
We also had the case that colleagues then asked, from their point of view already justified at the time, “when will my e-mail signature work again, for example?” If you are now in an inner stress situation, then you explode. Then they go at each other’s throats. That’s useless. You have to do it calmly, prudently.
But you also have to give yourself some space, so the first six/seven days are really the worst. You have to switch off sometimes, leave the whole topic behind, so that you can calm down a bit, relax and then start the new day fit. That doesn’t change anything about the situation, it’s still there, so from that point of view, those are actually the key points that you should pay attention to: keep calm, organize, don’t work, take care, listen, collect ideas. Those were actually the most important things that I learned.
For the organization is actually exactly the same. Dealing with a lot of fears, there is psychological pressure on all sides. You just have to keep calm and talk very openly within the company so that the employees have an understanding of what is happening now, what comes next, am I safe at work, how is everything going to continue in the company? You just have to radiate that a bit.
Yes, thank you for sharing. If we now draw the bow to the second attack. Now you had rebuilt everything, also invested a lot of time and probably money to protect yourself better. Apparently, it wasn’t enough now. What happened?
Well, yes, it hit us a second time, but it still results a bit from the restructuring measures of the first cyberattack. As I said before, we had a cyberattack that happened in the first nine weeks. After the nine weeks we haven’t rebuilt everything yet. That takes a very long time and we were in the final stages of the first cyberattack when an old legacy system was hit that we still wanted to exchange.
You can already see the differences, yes, we didn’t do it well and consistently, but then we knew with the second attack “oh, it got us again” and it was very fast and then we had the overview straight away. “Okay, it’s a cyberattack, it’s ransomware.” Then the processing of the incident actually started like the first time. But much more orderly, much faster, much more structured and you can see that in the time.
The first lasted 9 weeks until we were able to work 95% again. With the second, everything was over after 48 hours in the company. And you can see that even when it’s practiced, when the routine is there, when the contacts are there, when the people know, “Oh, that and that happened.” Then it actually goes down according to the checklist, also with the total number: in the first cyberattack there were 23 consultants plus my department over the first six weeks, in the second cyberattack there was even an admin on vacation. And we hardly had any external support apart from forensics, and that’s actually the best way to see the difference. Also, in terms of money. In the beginning millions. Now, not quite 100,000 euros anymore. And you can see that – as I said, the difference when you know what hit you.
Of course, it is better to know in advance that it will be tried, so prevention is even better. Unfortunately, it wasn’t like that, but the better you practice it, the easier and faster it goes.
Yes, thank you. There are many companies that are of a similar size, typical medium-sized companies, I would call it, like you. Are there any organizational measures or tools or even expensive tools where you say you should have that nowadays, that should have been introduced, implemented, whatever. Do you have your top 3 / top 5 of what to do?
Well, the most important thing and that is actually an organizational measure, I need good documentation and a crisis manual that is available offline. If the IT doesn’t work anymore, I can pull it out of the closet and then work it off. But this must also be updated regularly. It’s also no use to me if it’s been in the cupboard for three years now and has a layer of dust. It’s just part of the cybersecurity strategy.
A second topic, which doesn’t cost that much money now, it’s a bit more stressful for IT now, just remove the admin accounts on laptops, do a bit of domain tearing, that does not cost an insane amount of money, but increases protection.
The more expensive measures that bring a lot, are then SIEM systems and then maybe also an SOC that someone keeps an eye on your network and on unwanted activities 24 hours a day. But setting that up is a lot of work, a lot of stress, and I have to do some preliminary work. I should have a good firewall infrastructure, good network segmentation. I need a strategy to separate to OT, to Office, to implement all this. And the worse prepared it is, the more impact it has on the team, because then simply far too many full positives come, and it also does not work that way.
And as I said, SOC is great because someone looks at it professionally and you can’t have the resources. But it is one of the most expensive measures that one should invest in today.
Yes, thank you. And with tenders like this, how do you think we fit in? So, what can CyberCompare do as additional value, as a platform? What’s your view on that?
The additional value is great for me, also because to create this tender, all the requirement analyses, looking for the manufacturers, comparing them, is not easy and we simply don’t have the time for that.
We did our SOC tender together with you (CyberCompare). You do a questionnaire once, you get super-profound results, you have contacts with the manufacturers or suppliers, you know who you are talking to and they already know who on the other side wants what, because it’s already well prepared. I think doing it with you instead of doing it alone will definitely save months of work.
We love to hear that, and as closing: Do you have any tips or advice, you would like to pass on to all your colleagues in the CIO or IT manager chair? Where you say, these are the one/two things – remember them and think of me.
Well, the two most important things during a cyberattack, if it hits you:
Never lose your sense of humor. It’s the worst situation that an IT professional can imagine, but the drop is sucked anyway. That’s why you can still laugh during a cyberattack.
And the second, that’s actually a personal statement: There is no such thing as 100% IT security. Be prepared, software is programmed by people, configured by people, and used by people who shamelessly exploit each other’s gaps because they want to make a lot of money and you just have to be prepared. Those are the two most important statements on my side.
Thank you. Great, then we come to an end.
Thank you so much Stefan for sharing your experience with us.
I think there’s certainly something there for the audience, for the participants here at BCW, that they can take away with them. Everyone can decide for themselves whether they remember it or not. If you are also working on topics as a listener/viewer, would like to have an independent view of your cybersecurity landscape or are looking for support with tenders and comparisons of offers, please do not hesitate to contact us.
If you have any questions based on this video, if you also have a question for Stefan Würtemberger, let us know. We’re happy to process them, we’re happy to help. The colleagues are online. Stefan and I are here at the fair directly, so if you should be here in person, feel free to stop by the Bosch CyberCompare booth and we look forward to working together, to the feedback and for the moment.
Thank you once again, Stefan, and have a nice day!
Please remember: This article is based our knowledge at the time it was written – but we learn more every day. Do you think important points are missing or do you see the topic from a different perspective? We would be happy to discuss current developments in greater detail with you and your company’s other experts and welcome your feedback and thoughts.
And one more thing: the fact that an article mentions (or does not mention) a provider does not represent a recommendation from CyberCompare. Recommendations always depend on the customer’s individual situation.
