Lesson 07
NESO
NESO is the UK company responsible for keeping power flowing to 67 million Britons. It’s walking a tightrope between a carbon-intensive past and the clean energy future - but that rope is getting wobblier every day. Locked in its vaults, NESO may have the data it needs to train AI to keep its increasingly complex processes running smoothly. But is the data good enough?
Mist swirls in the floodlights
mounted on the side of the long, low-slung office block in the Berkshire countryside. It settles over the cars parked in the surrounding lot - more than you’d expect at six a.m. on a cold October morning - and wraps the thick forest that hems in this business park. At the gatehouse, the guard’s breath steams in the freezing air as he raises the barrier.
Inside, the building is bright and warm, but an autumnal hush seems to have slipped in through the air vents and settled even in the innermost sanctum. This is a high-ceilinged room at the heart of the building - white walls, blonde wood - a room whose very nondescriptness seems designed to force your attention onto the screens that glow with information everywhere you look. There are literally hundreds of them, monitors crammed shoulder to shoulder across the rows of desks that all face towards the front of a room: monitors on stands, monitors on arms, monitors squeezed so tight they crowd each other out. Shirtsleeved workers sit behind them, studying the data and occasionally tapping commands into their keyboards.
Gazing back at them, mounted high on the front wall, is the master display: a two metre-high screen that stretches across the whole width of the room. It’s covered in an intricate schematic of intersecting lines and interlocking rectangles, labelled where they cross with cryptic white numbers. As the operators tap their machines, the lines change colour: blue to purple, purple to red. The bar-graphs at the edges rise and fall.
This is the control room of NESO, the National Energy System Operator. The network on the screen is a high-voltage circuit diagram of the whole of the UK turned 90 degrees, with Aberdeen in one corner and Cornwall diagonally opposite. In real time, it shows the energy coursing through the country, from power stations and wind farms and solar arrays, across the grid, and out to the local networks that step down the voltage and feed it into homes and offices.
The control room, with its cathedral ceiling and hushed atmosphere, is a temple to power. But it’s also a shrine to data, the millions of bits of information that feed back from the grid’s nervous system to inform the decisions that NESO operators make every day to keep the lights on.
If they’re going to keep those screens glowing, everything is going to have to change.
Keeping the lights on in the post-fossil era
NESO is probably the most important company in the UK that most people have never heard of. It’s certainly the newest. In the control room, banners on the side walls sport the company’s new corporate branding - a soothing mauve-and-plum colour scheme - covering up places formerly painted with the blue National Grid logo.
NESO was spun out in October 2024 to be a freestanding entity that manages the UK’s energy system. Power plants’ giant cooling towers, dense wind farms spinning above the North Sea, steel pylons marching across the landscape: these are all integral parts of NESO’s business. But it doesn’t own or operate any of them. Instead, its remit is to manage the energy that flows from and through that infrastructure.
It does that by sending instructions to different generators to provide just the right amount of power at the right time: turn it on, turn it up, turn it down or turn it off. It’s a delicate balancing act that they have to get right every minute of every day. If the amount of energy going into the grid isn’t the same as the amount of energy coming out of it, things fall apart very quickly.
NESO’s overriding mission, to the exclusion of all else, is to keep the lights on in the UK. It’s a job their engineers and engineers have done with quiet efficiency and enviable stability. In 2014, the network boasted 99.99995% reliability, the best in Europe. Nine years later, that last digit had ticked up to a nine. NESO’s goal is to add another nine on the end for good measure.
But NESO now has another mission. The government has charged it with devising the roadmap to shift Britain’s energy sector fully away from fossil fuels by 2030. That’s five years ahead of the US goal, and ten years ahead of the EU’s. The New York Times, not given to hyperbole, describes it as ‘the most ambitious target of any major industrialised economy.’
Other commentators are more pointed and use words like ‘fantastical’ and ‘unachievable’. Fintan Slye, the guitar-playing Irishman who’s in the hot seat as NESO’s CEO, can see their point. ‘We’re not saying that the target is achievable with the current energy industry processes and systems and ways of working,’ he told the Guardian. ‘In fact, it’s not achievable in those circumstances. But if you can make the required changes, then it can be delivered.’
Evolving to become ‘world class’ in AI
What needs to change? In his interview, Fintan mentioned the planning regime, the regulatory system, the grid connections process… to say nothing of the physical infrastructure itself. The last big buildout of Britain’s power grid happened in the 1960s, when fossil fuel plants were largely centralised in the industrial heartlands of the Midlands, close to the coalfields that supplied them. Managing the grid with a limited number of large suppliers, their output predictable and reliable, was never easy; but it was a fairly straightforward proposition. Now things are changing.
The day before NESO came into being, Britain’s last coal-fired power-plant, at Ratcliffe-on-Soar in Nottinghamshire, closed. Its icon will vanish from the circuit map on NESO’s big screen, marking the end of 140 years of coal-powered electricity generation in the UK.
There’s no comparable big replacement plant: instead, the capacity will be supplied by a galaxy of different renewable projects of every shape and size. It captures the broader trends that are reshaping the nation’s electricity mix: a huge opportunity for decarbonising the economy, but a huge challenge for the company charged with overseeing that transition while making sure the system never skips a beat.
NESO needs AI. Without it, the grid won’t be able to manage all the wind farms, solar panels, batteries and electric vehicles coming online in the next few years. And nobody needs the consequences spelling out if it fails.
The company’s executives are ahead of the game and fully signed up to the need for AI. ‘NESO’s leadership is very ambitious at becoming world class in AI,’ says Niko, who’s worked firsthand with the management team to help them shape their AI strategy. ‘Very few other organisations in the energy sector are as ambitious as these guys are with AI.’
If ambition and commitment were all that was needed, NESO would already be well on the way towards solving its AI requirements. But there’s a challenge - one that for many organisations would be insuperable. It’s the data.
Banishing the myth of perfect data
‘Data is the oil of the 21st century,’ goes the famous saying. While the analogy isn’t perfect, it’s got a particular poetic resonance in the energy industry, and it captures a core truth. Without data to feed the models, the AI machines won’t run. Early AI systems built using hand-crafted rules turned out to be brittle creations that struggled with the complexity of the world they were modelling. It was impossible to codify all the rules for a system to follow. But if you have enough data (and enough compute to process it), then deep learning algorithms can learn those rules for themselves, with far more subtlety and nuance than hard-coded versions.
That development has been at the heart of the AI revolution of the last few years. It’s also changed the way we think about data, and elevated this once arcane technical concept to the top table of business discourse. Chief Data or Information officers roam the C-suite, with fat budgets to invest in programmes to collect or manage data. Executed well (see for example Inspired Education’s data platform in chapter 3), these sorts of programmes provide the foundations needed to successfully adopt AI into an organisation. But there are traps that need to be navigated, both by organisations with little data, and by those with lots.
NESO definitely falls into the latter category. Managing 20 terawatt hours of electricity a month (which NESO helpfully quantifies as 20 billion washing machine cycles), sending it the length and breadth of the country - and overseas, via interconnectors - generates mind-boggling quantities of data. Other companies profiled in this series, like DRIFT or Cera, had to wait months or years to accumulate enough data before they could train the algorithms they wanted to build. NESO is positively drowning in data. Just one data set, for example, is receiving 90 million updates every day. Multiply this by the hundreds of data sets they keep, and you get a picture of the scale of information they need to crunch.
But although NESO has the volume, there’s a catch. AI algorithms want their data in tabular form, racked and stacked in orderly columns and rows. Some of NESO’s data fits the bill, but other parts of their sprawling data portfolio can range from PDFs of ancient manuals, to Word documents, to tangled schematic and process diagrams that any algorithm would struggle to digest. Some of the data is held in different software applications of a certain age that don’t like to talk to each other. Other data changes as it goes through different business processes, without a clear record of the path it’s taken. It’s… complicated.
To be clear, this isn’t a problem unique to NESO. While digital native companies have the freedom to design their data architecture from scratch and build their institutions around it, any long-established organisation with complex legacy systems is going to have patchy, convoluted, or hard-to-reach areas of their data estate.
And when those companies face significant operating challenges that might be solved by better use of data - or simply when they hear calls from investors to ‘do AI’ - many of them will succumb to the temptation to embark on a big data infrastructure exercise, getting everything scrubbed and tidy before unleashing the algorithms on it. Which makes sense. After all: no data, no AI.
But the desire to ‘fix’ the data before it can be put to good use is ultimately built on a pair of myths: the myth of ‘perfect’ data, and the myth of ‘complete’ infrastructure. The truth is, data is never perfect; and data infrastructure is never done. Both exist, like the world they catalogue, in permanent states of imperfection and constant evolution. Trying to pursue some idealised end state isn’t just futile, it’s costly and time-consuming. The modern corporate mausoleum is filled with the corpses of large-scale data transformations that either took a lot longer than planned, cost a lot more, or failed entirely.
NESO is taking a different approach.
Planning the ‘right time’ for an outage
Faculty’s Energy Transition and Environment team is the newest business unit in the company. It started in 2021 with a single hire, and it’s been growing ever since. Niko Louvranos, the unit’s energetic Business Development Director, gets even more animated than usual when he describes its mission. ‘We’re the youngest team, taking on the biggest challenges, for the most significant players, in the industry that the world is relying on to get to net zero.’ The data scientists, engineers, and consultants who work on the team are young, super smart, and serious about making real change.
When Faculty started working with NESO, the grid company already had a comprehensive digitalisation programme in play: a well-designed, well-run project that any CIO would be proud of. But crucially, rather than focussing all of their energy on getting the infrastructure right, NESO also looked for opportunities to address some of their most urgent challenges using data they had available already, even if it was imperfect. They decided they could build the tools and infrastructure they needed as they worked out the use cases it would serve.
The first project in the pipeline looked at planned outages on the network. These are decisions to turn off particular pieces of infrastructure - anything from a whole power plant to an individual transmission line - in order to perform planned maintenance. This might be for something as simple as a bird’s nest on a pylon, or as complex as a major refit on a power plant. Some outages are so significant that they can be scheduled up to six years in advance.
One common reason for an outage is to connect a new power source to the grid. With ever more renewable assets and supporting infrastructure being added to meet the UK’s net zero targets, these are becoming more and more frequent. The backlog is growing, and if vital maintenance gets delayed too long then suddenly a planned outage can become an unplanned, emergency outage. Which is a whole new world of pain.
Shutting down any part of a network as complex and critical as the national grid is a daunting proposition, and a whole team at NESO - the Network Access Planning Team - is dedicated to evaluating all the factors that have to be taken into account to find the best time, and the best mitigations.
If a transmission cable is out, for example, is there enough headroom on the other available cables to safely carry the extra power that will be coming their way? If generating capacity goes offline, what will replace it, and how much more will it cost if that happens at peak times?
To perform this analysis, gaming out all the different scenarios to find the best solution, is a monumental task for the team. ‘At the moment, they will do hundreds and hundreds of very heavy duty simulations to work out exactly when looks like the right time for the outage,’ explains Katrina Soderquest, the senior Faculty data scientist who worked on the project. ‘It takes a lot of time, and a lot of people.’ Although there are optimisation tools to help, they rely on trial and error, manual runs and human intuition.
You don’t start with the data you have and then think about what to do with it. You start with the problem you want to solve, then think about the technology that will help address it, and then what data you need to feed into the technology.
NESO were keen to see how machine learning and AI could help them speed up the process to handle the growing wave of demands. When the team started working with Faculty, both sides knew that the data and supporting infrastructure would be imperfect. But Niko and his team were determined to push the art of the possible. They knew that a specific model, with a specific objective, only needs specific data to achieve its aim. More data is only useful if it directly relates to the objective and adds more signal to the model, rather than noise. In other words: you don’t need all the data, you just need the right data.
And you don’t start with the data you have and then think about what to do with it. You start with the problem you want to solve, then think about the technology that will help address it, and then what data you need to feed into the technology. Data is the essential enabler, but it comes last in the logic, not first.
NESO were wise to that. They resisted the temptation to try to ‘fix’ their data before starting; they didn’t have time to chase that endlessly receding horizon.
‘The outer limits of what’s achievable’
Thanks to that leap of faith, the pilot project that Faculty worked on proved that AI could manage with the data NESO had, imperfections and all, and significantly speed up the outage planning cycle. ‘It’s about doing the easier stuff quickly,’ says Katrina. ‘You’re never going to replace the experts in those fields, but this project showed we can make planning more robust and quicker at the same time.’ There’s also a cost implication. NESO spends billions of pounds a year buying energy to balance its system. Even a small increase in efficiency translates to huge savings.
Other projects, run along similar lines, have demonstrated AI’s promise in the tricky tasks of balancing voltage on the grid (a physics lesson in itself), and in gas transmission network planning. More are in the works. Because of the complexity of the grid, and NESO’s management’s desire to make sure that these changes happen in a coherent way, the new algorithms haven’t been implemented operationally yet. ‘They want to link the different AI solutions to their wider processes,’ says Niko. ‘They don’t want to just fix things in isolation. They want to do it holistically.’ But the course has been set and the direction is clear.
In his Guardian interview, Fintan Slye described the challenge of getting to a clean power system as ‘at the outer limits of what’s achievable. But,’ he went on, ‘if you’re prepared to do things differently, and to take difficult decisions early on, then yes, absolutely it is doable.’
So long as you don’t wait for perfect data.
The lesson in summary
There’s no such thing as complete data.
- Data is essential to modern AI. No data, no AI.
- But AI is a precision game. A specific model, with a specific objective, will need specific data to achieve it. More data is only useful if it directly relates to the objective at hand and adds more signal to the model.
- The right starting point is always to be clear about the problem you’re trying to solve. Then what technology can help you solve that problem. Then what data you need to power the technology. In that order. Data comes last, not first.
- It is easy to be held back by getting this thinking the wrong way round. By starting with the data; trying to bring it all into one place, to organise it in the perfect data lake, to ‘fix’ it in some way. In fact, data is never perfect. And data infrastructure is never finished. Both exist in permanent states of imperfection, and constant evolution.
- Waiting until you have complete data to deploy AI will mean waiting forever. Instead, you need to focus on making sure you have the specific set of data you need for specific applications you want to build.
Did you enjoy this story? There are nine others just like it, told from the perspectives of nine of our other inspiring customers, in the full book 'Ten Lessons From Ten Years of Applied AI'. Just leave your details below to get instant access to your copy of the book.
With contributions from:
Ten Lessons From Ten Years of Applied AI
Download the eBook
Get instant access to ten examples of AI solving the world's biggest challenges, told through the stories of ten of our most brilliant customers.