The biggest crisis of modern times, which fortunately never became reality

On the second weekend of February the Baltic states disconnected from the Russian electricity grid. The Estonian, Latvian and Lithuanian power systems were successfully joined to Continental Europe. Although for the ordinary person nothing happened or changed, months of preparation went on behind the scenes.

At a time when technology surrounds and accompanies us everywhere, we asked how the landmark weekend passed at the Baltics' largest data center and what a large-scale power outage would really mean for Estonia. Toomas Kell, head of operations and technology and board member at Greenergy Data Centers, gives the background in this interview.

While ordinary people were asked to stock up on food, water and other essentials in case of a crisis, how did you prepare?

For us it was important to check that the electrical infrastructure was working and that the backup systems that have to kick in if the power fails were functioning. That was the main task on the technical side.

On the other hand we reviewed the procedures related to crisis management. All institutions providing important services have a crisis plan, a plan for coping with a crisis. An inseparable part of that is the recovery plan: if something should happen, how we behave, what we do and how we resolve the situation. We reviewed the plans and updated them according to the circumstances.

What was the most difficult part of that?

You had to look very clearly into all the small routines and details and understand them so that the ball would not be dropped anywhere, because a small mistake can escalate into a big blunder fairly quickly.

To give an illustrative small detail: one potential scenario was that the power is out across Estonia and all communication is cut. How does the implementation of the recovery plan continue? Our internal agreement is that if the crisis management group can no longer reach each other, we gather at the data center within an hour and a half at the latest.

How did you check the backup systems?

We have UPS units and batteries in the building that take the first hit if the power fails and supply the critical systems with electricity until the generators start. That means from the customer's perspective the service is not interrupted; power is provided from the backup systems.

To review that backup infrastructure we checked the batteries, looked at how they hold up, how the UPS units work. Second was testing the generators, which we do regularly anyway. We started the generators and watched how they run. Generators need fuel, so we did fuel analyses.

Finally we did a so-called half-blackout test. We cut one of the two power legs and watched how all the different nodes responded to that event. That is, whether the power supply switches from the UPS units to the batteries, whether the generators start after that, and so on. While we do short tests weekly in which the generators run for up to 15 minutes, this time the whole building was on generator power for more than an hour. We did the same test on the other leg.

How long could you have operated on your fuel reserve?

Our reserves are calculated according to the standard: when the building runs at full load, we have to ensure power supply for 72 hours in a row. That is our absolute minimum. Derived from the current load, we could manage autonomously for more than a month.

In addition, for fuel delivery we have a supply assurance contract with two different suppliers. That means more fuel is brought to us on the next working day, for as long as fuel is available at all. So in the event of a larger act of sabotage whose effects lasted weeks or months, we would have done our part nicely from that perspective.

Did any shortcomings come to light during the checks and tests? Or did everything work flawlessly?

The technical side worked flawlessly. Since we run preventive tests in the building regularly, faults are as a rule eliminated before the system tests. It was rather the procedures that needed reviewing. The plans existed, but they had to be thought through and the roles reviewed: who has to do what and where if a given scenario materialises, all the way to crisis communication. In the worst-case scenario our first priority is to restore normality and then communicate with customers, not the other way round. There are a great many small nuances.

Have you had to think about such nuances before?

Once a year we hold larger exercises in which we go through the whole theory in practical terms. We take one real situation from our crisis action plan, for example power can no longer be obtained from the grid or someone gets hurt, and go through the process. Communication is part of all that too.

In the end everything went well, but you had to be prepared for the worst. What scenarios did you prepare for?

The main keywords are loss of power and power quality. If power quality is compromised, it affects the equipment. Since our customers' hardware is fairly expensive, any deviations can become very, very costly.

And physical security?

Regardless of the fact that our complex is already secured, we gained a so-called extra layer of protection. Although we are not yet on the state's list of critical infrastructure, next to us is the Harku substation, which was guarded with might and main. Reinforced concrete barriers were brought in, barbed wire was set up. The Defence League and the police were present. Where before we were theoretically aware that our neighbour is protected, now we saw it for real.

That in turn created interesting challenges. It happens quite often that customers come to us at night too, when something of theirs has broken. When the police stop you at one in the morning with automatic weapons, emotions can vary quite a bit. Some feel great pride that the matter is being taken seriously. Others ask why they are being harassed: they just wanted to hurry off as usual to solve their problem.

What did that weekend actually look like? Were there situations that had to be resolved on the fly?

We gathered our "forces" in the building. Together with the operations center we monitored both the disconnection from the Russian grid and the synchronisation with Continental Europe. We kept a keen eye on the important events.

Regardless of the fact that we went into mentally heightened readiness, the aim was not to sow panic among ourselves. My message to the operations center was likewise to stay calm and operate in normal mode. Fortunately it turned out that we had not a single technical hiccup. Everything went smoothly.

Were glasses clinked in the operations center when the connection to Continental Europe succeeded?

We had no such ceremonial moments. There was no clinking of champagne glasses, but there were warm pastries. It was simply a relief that the thing went well both for us and at the national level.

Let's be honest, the media managed to raise the level of anxiety across Estonia quite considerably. To give an example from personal experience: I go to one large shop every Friday, and on that Friday, the day before desynchronisation, the shop was packed with people and there were no shopping trolleys. It felt as if there would be no tomorrow.

On the other hand, it may have been good training and preparation for people and society in a practical sense too. The threat of war is still theoretical, but a large-scale power outage was a tangible event. Everyone thought through what they would do if some crisis came. As a result we are much better prepared, as organisations, businesses and as a state.

So what was the main lesson?

For institutions and organisations that were going through a crisis or recovery plan for the first time, it was actually extremely difficult. They had to leave their comfort zone, their normal routine was disrupted and people behaved rather shakily. If you do it with a certain regularity, the routines are clear: you do not rush, you move at a normal pace, you are not in a flap. Sometimes it is wiser to stand still than to run, because without thinking you may also run in the wrong direction.

Where previously we talked about our backup systems, the underground fuel tanks, backup generators and the rest, in theoretical terms, now that a threat could actually be felt in the air, people started to understand why we do things the way we do. The fact that we are built to be crisis-proof was probably one of the reasons that weekend passed so uneventfully for us.

When might it happen that this preparation is needed next time?

I would like to say never, but that is probably not true. On the other hand, I stress again that the role of a proper data center is to be ready for crises. Our facility is built so that it withstands crises. Our team is put together so that we withstand crises.

"There is always some uncertainty in the air"

Ergo Tars, director of the State Information and Communication Technology Center (RIT), describes the agency's preparation:

Preparing for desynchronisation gave us the opportunity to understand even better how important RIT's services are and how high our customers' expectations are. The process highlighted how important it is for public sector institutions and providers of vital services to have up-to-date crisis and service recovery plans. Since the world and the security situation change quickly, testing and improving such plans must be on the work plan every year.

We prepared in two directions: ensuring RIT's own continuity and supplying customers with computer workstations should the risks materialise. More broadly, we learned how to secure the state's critical IT services in a power and communications outage. Many details had to be thought through: what computer workstations and access to provide for people coming from the private sector to help the state, how to ensure access to data, how to solve the need for printing, how to ensure communication among ourselves by satellite link, for example, and much more.

The greatest risk is ensuring that all RIT customers whose tasks are critical to the functioning of the state receive the necessary IT support in an unexpected situation. We have customers across Estonia, but resources, both money and people, are limited. The challenge is finding the balance: how to prioritise the most critical needs and use the available resources as effectively as possible.

By mapping risks, streamlining processes and developing alternative solutions we can raise readiness, but there is always some uncertainty in the air, because every crisis situation is different and requires flexible solutions. So the challenge is to ensure flexibility alongside the rules, so that we can support customers regardless of the nature of the crisis.

Alongside the continuity of RIT's services, during the preparations we paid great attention to employees' awareness and readiness, so that our people would know how to act in crisis situations and, if necessary, also support their families and loved ones. At the same time we saw that such awareness and practice must be developed continuously so that the response is smooth and confident.

We highly value customers' initiative in thinking through not only synchronisation but more broadly the impact of crises on their services: how they depend on IT solutions, electricity and communications. We invite customers to systematically map their critical dependencies, and we are always ready to support them in exercises and provide input for playing through different scenarios, to jointly make the functioning of the state even better.

Planning should have started earlier

Tarmo Tulva, head of IT infrastructure at the energy company Enefit, shares the company's experience:

More thorough preparation for the desynchronisation project began about four months before the deadline. From a group-wide perspective we reviewed both the information security measures already implemented and the additional measures planned specifically for the desynchronisation period. We communicated with our partners on the telecom operator side and agreed possible crisis contacts and methods of exchanging information. We also asked for additional support from the partners who supply our network equipment.

The primary prerequisite for the success of such projects is still sufficient planning, and the usual lesson is that one could and should always start planning a little earlier than actually happens. In this project too we should have started planning somewhat earlier, to analyse all the risks and possible mitigation measures more thoroughly.

The bigger risks were certainly related to information security. Unfortunately one can never be quite one hundred percent sure that the bad actors did not start their project years ago and are simply waiting for the right moment to activate their activities. That is why we thoroughly analysed different possible scenarios and mapped both preventive actions and possible responses in case something should happen on the day. One of the risk mitigation measures was also setting up a so-called reserve command post at the GDC data center, to be as close as possible to our own physical infrastructure and, on the other hand, to be able to count on the availability of power and data connectivity. In reality that command post was not needed, but for playing through the exercise scenario, so to speak, it was certainly useful for us. We now have precise knowledge of how to move our command post to another location if needed and what actions have to be taken beforehand.

The desynchronisation itself passed unnoticed for us and there was not a single cyber incident. The lesson for the future is rather longer planning. This project also helped draw attention to additional information security measures that need implementing.