After four attacks in the UK, Theresa May had had enough. In front of the eyes of the world, she demanded that tech companies make a paradigm shift in their ability to stop terrorist propaganda. If they could not identify it and take it down within two hours of it being posted - the crucial window of opportunity - then her government would legislate to force their hand.

The tech companies said it wasn’t possible. The sheer volume of content Daesh pumped out would overwhelm any human moderators, while automated solutions were out of the question. The social media giants had the best AI and software engineering teams in the world, the largest-scale digital operations ever built, and they could confidently say that the technology May was demanding didn’t and couldn’t exist yet. 

But the British Prime Minister knew otherwise.

The terror of Daesh

The people within the Office for Security and Counter-Terrorism (OSCT) are one of the smartest and most impressive groups in the British government. Few among the general public know the team exists, but they’re lucky it does. Located in a highly secure area of the Home Office headquarters in Westminster, the staff who work there are deeply expert in analysing and understanding terrorist threats. A typical example is Tom Drew OBE, who worked there for seven years (and later joined Faculty). A softly-spoken thirty-something, with a dark brown beard and a penetrating gaze, Tom has dedicated his career to keeping the public safe.

In 2017, he and his team were troubled by a new emerging threat. In the Middle East, Daesh’s self-styled ‘Caliphate’ was in retreat: it had suffered significant territorial losses and been driven out of its de facto capitals in Mosul and Raqqah. In response, the group changed its tactics. Instead of encouraging supporters in the West to come to Syria and Iraq, they urged them to stay at home and carry out ‘single-actor’ attacks against unprotected civilian targets. Acting alone, outside of existing networks and often with no previous history of extremist activity, they would be almost impossible to stop.

The result was dramatic. An attack on the Houses of Parliament ended with five people killed; a bomber at an Arianna Grande performance in Manchester killed 22 concertgoers; a mass stabbing in London killed eight more.

The resulting coroner inquests showed that almost all the attackers were radicalised by the propaganda they encountered online. Daesh was using the West’s own social media networks against it, recruiting its killers in plain sight on places like Facebook, Twitter and YouTube. The platforms had teams of human moderators trying to find and remove the content, but they were far too few and way too slow to have any meaningful impact.

Key distinctions between analytics & operations

At the time, Faculty was a tiny startup operating out of an Edwardian townhouse in Marylebone. ‘Even the building sort of made a statement,’ recalls Angie Ma, one of the co-founders. ‘It didn’t look like a classic tech office. We wanted to make the point that we were a different sort of company, that tech didn’t have to be in your face.’ Still using its original name, Advanced Skills Initiative (ASI), the company had barely 20 employees. But Tom felt they had potential.

‘Their approach was (and is) to be collegiate problem-solvers,’ he recalls. ‘They treated it as an experiment to develop a new model, not an opportunity to just resell an existing process or product.’ Matt Collins concurs. ‘There was a can-do attitude when we presented the problem, an enthusiasm to just get stuck in and really see what the art of the possible was.’

Of course, the nature of an experiment is you don’t know ahead of time if it’ll succeed. John Gibson, who at the time led ASI’s consulting business, remembers fielding the first call from Tom. ‘We thought it could be done,’ says John, looking back, ‘but the only way we could prove that it would work was to actually build the technology.’

The heart of the challenge was one that comes up surprisingly often in the AI world: the distinction between analytics and operations. In many organisations, Data Science and AI teams have typically been siloed off in Analytics or Business Intelligence functions, well away from the messy business of the shop floor. Analytical teams provide insight for understanding the world; operational teams act on it and make things happen.

The danger, of course, is that a gap develops so that insights never actually get put into practice. And in this case, that gap was a chasm. The analysts were at the OSCT, while the people who could implement their recommendations were at the tech companies. Not only were they not on the same page, they didn’t even believe what the other was saying was possible.

Enter AI. The power of the technology is that it can take insights and weave them into actual workflows - but it’s not a quick fix. When you’re building technology for operational processes, there are always requirements you need to account for. They might be user needs, existing workflows, infrastructure requirements, policies or regulations - all the existing rules and constraints of a workplace. Even if the technology works flawlessly, it will never be implemented if it can’t deal with these real-life issues.

In this case, there were three key operational requirements for the solution: it needed to be 1) accurate, 2) quick, and 3) discerning. And these requirements were exacting: not just reasonably accurate or fairly quick, but orders of magnitude more accurate than the 99% that Mark Zuckerberg had dismissed, and fast enough to apply that accuracy at web speed. 

The tech companies may have been overstating the case when they said a technical solution was impossible, but they weren’t wrong about the scale of the challenge.

Operational requirements

The first requirement was for speed. Research produced by the UK Home Office and the European Union indicated that the first two hours after the release of new propaganda was the crucial window for disruption. In that time, videos were amassing millions of views, and more than 90% of all the links to that content that would ever exist had already been created. If you couldn’t find it straight away, you were already too late.

Some types of entirely legitimate content resembled terrorist propaganda in specific ways that might trip up an algorithm. Worse, the most difficult to classify was also the content that Tom’s team least wanted to remove. Islamic prayer videos and news reports of events in the Middle East, for example, might have certain similarities to jihadist content: censoring them would be not only controversial, but also counterproductive, because much of it actually served to highlight the flaws and hypocrisies in Daesh’s messaging.

So the system Faculty were being asked to build had to be able to discriminate between superficially similar ‘good’ and ‘bad’ content; with a false positive rate of less than five in 100,000; and all within two hours of the content being posted.

And the team could never forget that the clock was ticking. In October, a man drove a pickup truck into a group of pedestrians on a bike path in Lower Manhattan. In Marseille, a man stabbed two women at the train station. Every month brought more grim reminders of the stakes involved.

Extracting multi-modal signals from video files

John’s team at Faculty had a hunch that the reason the social media companies had failed to crack the problem was because they weren’t looking at the content broadly enough. Rather than examining any one single aspect of the videos, Faculty wanted to build an ‘ensemble classifier’, a model that would incorporate not only the relationships between particular attributes, but the relationships between the relationships. To do that, they’d have to wring every scrap of data they could out of the jihadist content.

But only from the content itself. The social media companies had a trove of data on individual users and who they were connected to, their networks and how they behaved online: all invaluable information that would help establish whether the content was terrorist-related. But the companies wouldn’t share that data, and even if they would, Tom didn’t want Faculty’s algorithm using it. ‘We wanted a solution that made a classification purely on the content itself,’ he says, ‘to prove that this could be done with the bare minimum data a government or third-party could capture - the media files themselves.’ It also avoided any issues with accessing users’ personal data, which would have tripped all sorts of ethical and regulatory safeguards. So all that Faculty had to work with was what any YouTube or Facebook user anywhere in the world could access: the videos themselves.

‘But if your model bases its classification on what the terrorists are actually saying in their propaganda - not just the words but the underlying tenets of their ideology and call to action - then it’s extremely hard for them to escape that without changing what they’re saying. And if you’re forcing them to change what they’re saying, it turns the game of cat and mouse into a strategic victory.’

The results were conclusive. The technology was able to detect 94% of the propaganda, with only a 0.005% false-positive rate. It could tell the difference between terrorist messaging and legitimate prayer videos or news coverage. And it could do it all in almost real-time, processing each video in the time it took to play. This meant that at the scale of YouTube’s then five million uploads a day, only 250 would be incorrectly flagged - enough for a single human moderator to check. It was entirely operationally viable. And they could prove it.

Running the tests

After nine months, the model was becoming more sophisticated. John’s team were optimistic that it could meet the exacting operational requirements they’d been set - but they needed to prove it. In particular, they had to demonstrate that the all-important false-positive rate was below the 0.005% threshold. To do that, they calculated, they needed a sample set of hundreds of thousands of randomly-chosen videos from across the global internet. So for several weeks they downloaded the first few thousand videos that were uploaded to the internet every hour of every day, avoiding any bias that might emerge from times of day or days of the week. Ed Sheeran, ping pong trick shots, cat videos, jihadist content… it all got scooped up and run through the model. Even the validation approach itself got tested and validated. The Chief Scientific Adviser to the UK Home Office vetted the process and pronounced himself satisfied.

The results were conclusive. The technology was able to detect 94% of the propaganda, with only a 0.005% false-positive rate. It could tell the difference between terrorist messaging and legitimate prayer videos or news coverage. And it could do it all in almost real-time, processing each video in the time it took to play. This meant that at the scale of YouTube’s then five million uploads a day, only 250 would be incorrectly flagged - enough for a single human moderator to check. It was entirely operationally viable. And they could prove it.

A new standard for technology platforms

Thanks to OSCT’s refusal to take no for an answer, Theresa May went onstage at the UN General Assembly armed with the knowledge that what she was demanding was possible. After she laid down her ultimatum to the tech companies, the world took notice.

Some of the attention was welcome. The Faculty offices hosted newspaper journalists and TV news crews in droves over the next few weeks, and the coverage they generated only heaped more pressure on the social networks to take the issue seriously. Within weeks, Mark Zuckerberg was publicly stating that companies like Facebook should be subject to more regulation, not less. 

Some of the attention was less desirable, though flattering, in a way. Daesh had been following the news coverage too, no doubt trying to figure out why their content wasn’t hitting its audience as well as before. Security insiders revealed that Daesh had nicknamed Faculty the ‘Dogs of Deletion’ in their internal conversations about the technology. The company took it as a compliment.

‘I went to the west coast nine times in two and a half years,’ says Matt. ‘Each time, we had to put evidence on the table to further the conversation. And when we pitched them to say, “Well, you know you’ve been saying you can’t do this, but we think you can, and here’s what we’ve done,” immediately the conversation went from a policy conversation, which was a bit binary, into a technical conversation, where they wanted to look underneath the bonnet and really understand what we’d done and how we’d done it.’

In the year following the release of Faculty’s classifier, YouTube reported it was using AI to remove more than 80% of violent extremist content before it was flagged by users. Twitter was able to block 96% of terrorist accounts before they could even send their first tweet, and Facebook had implemented AI to take down 99% of terrorist content within 24 hours of its first release, much of it within the crucial first two hours. The Home Office also made the service free to a host of smaller social media platforms who lacked the resources of the bigger players. As Theresa May put it, it was ‘a major step forward in reclaiming the internet from those who would use it to do harm.’

For Faculty, the small startup was suddenly on the map. ‘Serious people in government were telling their colleagues, “This is a company you should be speaking to,”’ recalls Angie. But the real significance of the project was felt in the wider world: for anyone who wanted to go to a concert, enjoy a drink outside, or just walk down the street without fear of being attacked with a knife or rammed with a car.

‘It seems a long time ago now,’ says Matt. ‘But terrorism was the number one national security threat in that moment. We know the importance of social media in our lives, and some of the harm, unfortunately, that it can help facilitate. So demonstrating exactly what capabilities could be brought to bear to reduce the dissemination of that content definitely had a part to play.’

‘A lot of people in different countries were looking at the problem, trying to get the tech companies to tackle it,’ says Tom. ‘And we subsequently heard independently from international partners that this was the big thing that really shifted the needle.’