AI Is Changing Cybersecurity In A Quick And Terrifying Way

Overview

AI is changing cybersecurity in quick and terrifying ways

Imagine I came to you with the following proposition: You will give me trillions of dollars. In exchange, I will build a machine that pollutes the environment, steals from every artist and academic on the planet, raises electricity bills and computer hardware prices, and can do marginally useful stuff like fill out spreadsheets and write basic code. To sweeten the deal, I will throw in a bridge in Brooklyn. Still not convinced? Then allow me to further pique your interest. This machine will also make all of your software less secure while enabling criminals and state-sponsored hackers to carry out more sophisticated cyberattacks than ever before.

Details

That's a deal most people wouldn't sign off on, but it's exactly what we collectively got out of the generative AI boom. In addition to the oft-discussed impacts of AI on intellectual property and the environment, AI has already had a profound impact on cybersecurity. Anyone with access to LLMs now has the equivalent of a fedora-wearing, Monster Energy-slurping black hat hacker working around the clock on their payroll. And whereas many common attacks such as phishing have traditionally required time and research, they now require little more than the ability to copy and paste.

AI has already had seismic ramifications across security vectors, enhancing traditional attack methods from phishing to credential attacks while also giving rise to new social engineering schemes which leverage voice cloning and deepfakes and turbocharging the discovery of unique vulnerabilities. Defenders across the tech industry are responding with AI solutions of their own, but can they outspend and outengineer criminals, let alone equally well-funded state-level actors? Here's how AI has shifted the state of play in cybersecurity.

There's a fundamental paradox at the intersection of AI and cybersecurity. While AI is useful for hardening security, it is a force multiplier for attackers. Not only does it have access to the statistical sum of the world's information, but it can synthesize across disciplines from coding to networking. No longer does a would-be cybercriminal need years of experience under their belt before carrying out a sophisticated attack. A capable LLM can be pointed at a potential victim and churn through tokens while the human attacker kicks back on the couch. As an article from the Harvard Extension School put it, "AI has democratized cybercrime."

Compounding the problem is the speed at which AI can work. What might take a human cybercriminal operation weeks to pull off can be accomplished in an afternoon with the help of an LLM. And criminals have access to more than one LLM. Many are running multiple AI sessions at once, or using multiple models. As described by Venture Beat, an AI-assisted attack from February that ransacked government databases in Mexico was carried out by attackers who fed outputs back and forth between Claude and Chat GPT. When one chatbot refused to help, the other was often willing to pick up its slack.

Humans are still necessary for now, and AI acts as a force multiplier rather than a replacement. But concerns are mounting. While this article was in process, Open AI revealed that a model under sandboxed observation had escaped its (potentially porous) testing environment and hacked AI repository Hugging Face along with other services in order to procure answers for a synthetic benchmark.

You might think it's just as easy to harden a security posture, since defenders have equal access to the same LLMs, and in some cases to even more advanced models that are not yet available to the public. But cybersecurity is asymmetrical. Whereas a defender must protect every possible vulnerability, an attacker only needs to find one in most instances. This dynamic has created a proliferation of zero-day vulnerabilities — security flaws that shipped with a piece of software and were secretly discovered by hackers. Flaws that AI models have proven adept at uncovering.

It's tempting to think of a hacker as someone who wears long black trench coats and types code at a million words per second, but more often it's someone sending out millions of scam emails until someone proves gullible enough to fork over credit card information. End users are almost always the weakest vulnerability to target.

The best example of this is phishing. Most of us have managed to spot at least one phishing attack from a mile away. An email purporting to come from Microsoft but is riddled with embarrassingly poor spelling and grammar is clearly the work of an amateur criminal with only passing familiarity with the English language. Or at least, that used to be the case. Thanks to AI, that email will now be impeccably written. It may reek of AI writing, but so does actual correspondence from Microsoft now.

For large-scale phishing efforts, an LLM can write a grammatically perfect and rhetorically plausible email, then automate the process of finding email addresses and blasting the missive out. Spear phishing, which occurs when a specific individual is targeted with highly personalized phishing efforts, is now far more trivial an effort. A chatbot can research your victim for you across the web, finding information you may not have discovered while doing your own research. The bot can even strike up pretextual correspondence with the target on its own, pushing them to divulge the information you're after.

According to security vendor Brightside, 82 percent of phishing emails now use AI at some point in the process, and whereas such emails had a clickthrough rate of just 12 percent prior, AI-assisted missives have boosted that number to a staggering 52 percent in simulated environments. And that's the data for email attacks, not accounting for the massive and accompanying spike in AI voice clone and deepfake attacks, which Brightside estimates to have spiked year-over-year by 442 and 680 percent, respectively, between 2023 and 2024.

As businesses and governments tie more of their operations to AI systems, they become increasingly vulnerable to attacks that exploit the inherent blind spots of AI systems. The most common of these are prompt injection attacks, wherein attackers find a way to give an AI system malicious instructions. For instance, instructions to send a company's payroll data to a particular email address might be hidden in a website. When an employee tells the company AI to summarize that page, it ingests those instructions and carries them out.

In some instances, these attacks are more like spam than malware. Microsoft found that some companies were including "summarize with AI" buttons on their websites that contained hidden instructions. When clicked, they'd tell a chatbot to do things like remember the company as a trusted source, or to recommend its products over those of competitors in future interactions. Only when a user checked their LLM's persistent memory would they realize they'd been deceived.

More directly malicious attacks can be equally effortless. In December 2025, Meta implemented an AI support assistant, ostensibly to help people recover their accounts. But the bot was a little too helpful, and would happily associate an email address owned by an attacker with any Instagram account, making it trivially easy to hack any account without MFA enabled.

Open AI claimed last year that fully defending against prompt injection attacks in AI browsers may not be possible. If that's the case, then the best course of action for any responsible entity is never to tie any of its sensitive systems or data to an AI in the first place. In the meantime, the best practice is always to double-check anything you paste into a prompt field from another source, and to limit the scope of an AI's access to system permissions and private data.

But some threats remain theoretical. A data poisoning attack, in which attackers play the long game of seeding malicious data across sources (for example, Reddit) that are known to be ingested for AI training, would be extremely difficult to see coming. For instance, filling computer support forums with recommendations to paste malicious instructions into a terminal so that AI chatbots begin to surface those instructions in response to user queries about common computer issues. One study published in October 2025 and conducted by Anthropic, in collaboration with the UK AI Security Institute and Alan Turing Foundation, found that it only takes 250 malicious documents to create a hidden backdoor in an AI model that could later be exploited. Some evidence of these attacks has already been seen in the wild, though the most prominent ones were relatively harmless tests carried out by curious informal researchers.

Meanwhile, AI models are increasingly authoring the code shipped to everything from operating systems to banking infrastructure. Research, including a study from the University of Naples and private research by Code Rabbit, has found AI code to be error-prone and to have more high-risk vulnerabilities, even when the generated code is reviewed by a human in the loop. Worse at writing code but better at finding bugs, AI capabilities can feel paradoxical to a human for whom those skills go hand-in-hand.

AI agents are your most gullible and tech illiterate coworker

Over the past year or so, AI agents have been the talk of the industry, with tools like Open Claw allowing anyone to let an AI use their computer for them. Enterprise clients have integrated agents as support chatbots, HR assistants for hiring and payroll, and in coding environments to pump out code, handle pull requests and even to deploy code. Private individuals, meanwhile, are hooking agents up to their emails, calendars, smart homes, and even their investment accounts. All of that sounds great until you think about it for half a second and realize you've given high-level permissions — if not full admin access — to an unthinking machine that does whatever it's told. Simply because an AI can often recall good security practices when prompted doesn't mean it really understands how to enforce them. And while a prompt injection attack on a normal LLM can have devastating consequences, an agentic AI attack chain can do a lot more damage.

Think of your most gullible and tech-illiterate coworker, the one who tells IT their computer is broken before checking whether it's plugged in. Let's call them Greg. Now imagine you've given Greg admin-level access to your computer and online accounts, even when you're not there to supervise. It's only a matter of time before Greg gives your bank password to a phishing scammer. But AI agents are even dumber than Greg, and they've now proliferated through businesses and private devices.

The aforementioned attack on Meta's overly helpful Instagram support bot is an example of an AI agent attack. The bot was plugged directly into tools that let it act as a human support agent would, and the results were as obvious as they were inevitable. Where a human would have been more discerning in the requests it chose to fulfill, the AI acted more like a computer, carrying out the task it was asked to accomplish using the tools available to it.

Defenders are responding with AI solutions of their own

AI may provide threat actors with supercharged capabilities, but that doesn't mean defenders are throwing their hands up in defeat. Cybersecurity professionals surveyed by Trend Micro said their top priority was defending against fraud and deepfakes, followed by preventing application attacks such as prompt injections, model poisoning and jailbreaking. They listed cloud environments as the most difficult surfaces to patrol, followed by keeping tabs on the security practices of remote workers who use their own devices for work.

Of course, the largest firms are making their own go of AI security. While this article was in production, Microsoft rolled out Project Perception, an agentic AI security system that uses agentic clusters to mirror traditional cybersecurity teaming. One cluster of agents is designated red for penetration testing and adversarial simulation, another blue for investigation and risk assessment, and a third green for integration and remediation. The goal, it seems, is to outpace the threat.

If cybersecurity is a game of cat and mouse, then AI has turned it into one of Roadrunner and Wile E. Coyote. Either we're all screwed as the models make Swiss cheese of every defense, or security has entered a brave new world that simply requires a bit of evolutionary adaptation. It's too soon to tell, so for now, we grope our way through an interminable and anxious interregnum.

Key Takeaways

• AI is changing cybersecurity in quick and terrifying ways

• Imagine I came to you with the following proposition: You will give me trillions of dollars

• That's a deal most people wouldn't sign off on, but it's exactly what we collectively got out of the generative AI boom

• AI has already had seismic ramifications across security vectors, enhancing traditional attack methods from phishing to credential attacks while also giving rise to new social engineering schemes which leverage voice cloning and deepfakes and turbocharging the discovery of unique vulnerabilities

• There's a fundamental paradox at the intersection of AI and cybersecurity