AI as a Weapon

Social engineering has always been a cornerstone of cybercrime. However, unlike in the past, today’s cybercriminals are using AI to power their scams. They are creating phishing emails and conversations in multiple languages, thereby eliminating some of the key indicators traditionally used to identify scam attempts.

Imitating Language Patterns

AI-powered social engineering does not merely mean improved grammar and syntax in phishing emails. Through sophisticated prompt engineering, threat actors are increasingly able to flawlessly mimic the language style of specific individuals or institutions. This means that the email is not only well-written but also almost indistinguishable from a genuine one. Bank notifications, news stories, and social media posts—as well as stolen communication data—can also be used as training material to uncover meaningful connections and generate so-called “spear phishing” emails.

This is why it is crucial—just as the NIS2 directive already demands—to include the supply chain in cybersecurity risk assessments: if hackers gain access to the email accounts of business partners and extract relevant conversations, it becomes easy for them to send the recipient a seemingly authentic but malicious phishing email. What cyber actors previously had to create manually and painstakingly is now automated by AI. As a result, spear-phishing emails are now problematic not only because of their quality but also due to their increased frequency.

AI Generates Voices and Images

This new AI-enhanced precision is not limited to text: AI can also generate voices and images. Audio and video deepfakes, which imitate CEOs or other trusted individuals, have already been used to authorise fraudulent transactions. It is important to understand that producing such video and audio recordings is a relatively simple process and is already widely offered as a service on the digital underground market.

Transforming a deepfake into a real-time video simulation—i.e., enabling direct interaction with others—is still a challenge. While some tools can already generate very convincing deepfakes in real time, someone who knows the impersonated person in real life can usually detect the forgery. Nevertheless, it is becoming clear that audio and video transmissions no longer guarantee secure identification. A growing number of underground services now exist that are capable of bypassing “Know Your Customer” (KYC) systems, such as those used to open bank accounts.

Private individuals are also in the crosshairs of cybercriminals. Criminals deliberately exploit the human tendency to share personal information online. Younger generations in particular are fond of posting personal videos and audio clips on social media, thereby unknowingly providing cybercriminals with the raw material needed for deepfake models. In so-called “virtual kidnapping” schemes, such artefacts—particularly voice recordings—are misused. This enables cybercriminals to convincingly fake a child’s abduction and blackmail the parents, for example by playing the alleged child’s voice in the background during a phone call.

Jailbreak – Overcoming Ethics and Morality

To prevent large language models (LLMs) from being used for such malicious purposes, manufacturers generally equip them with safety measures. However, this doesn’t necessarily stop hackers. With the help of so-called “jailbreaks,” it is possible to bypass the built-in restrictions of LLMs—achieved through clever prompt engineering. A whole “Jailbreak-as-a-Service” portfolio has now emerged on the dark web. This makes it possible even for technically inexperienced cybercriminals to rent a manipulated LLM.

In practice, this works as follows: the “renter” is given access by the provider to an LLM interface that appears completely ordinary but is already unlocked via an adaptive prompt running in the background. Examples of tools that bypass the embedded ethical guidelines of legitimate AI models include “EscapeGPT” and “LoopGPT.”

Security Flaws in Software Code

Currently, the cyber underground primarily uses compromised LLMs to optimise existing processes—for example, in creating malicious content. AI allows for the scaling and acceleration of time-consuming procedures, making attacks more efficient. However, it is still incapable of inventing entirely new methods; it can only adapt those that already exist. This becomes particularly evident in the area of exploit development.

AI has been used for some time in specially trained models to identify security flaws in software code. Exploits—i.e., the exploitation of such flaws—contain small software modules. Since AI is capable of generating software code, it should, in theory, be possible to combine both steps: vulnerability scanning and malware code generation. Such an AI-powered, automated zero-day machine would be a highly welcome tool for cybercriminals.

However, reality paints a different picture. While AI can assist in individual processes, it is not (yet) capable of developing exploits independently. It requires human guidance—such as documentation of a vulnerability or instructions to generate code for specific purposes. The invention of new business models or attack vectors appears even more remote. AI is only capable of stimulating human creativity—which, in the hands of malicious actors, can have dire consequences.

Agentic AI – Modular Cyber Attacks on a Budget?

Even though criminal activities are already being supported by AI in a variety of ways, there are still certain limitations. For example, if a threat actor wants an AI to analyse large datasets, they will need highly capable programmes and systems. The cost of extensive data analytics would significantly reduce profits, which may deter financially motivated cybercriminals.

However, a solution may be on the horizon. Agentic AI promises smaller AI assistants that can work autonomously. One “black hat” attack scenario could involve smuggling such an agent onto a victim’s system. This agent would locally analyse the data and only send the relevant content back to the attacker. A second agent could analyse the network and suggest which criminal business model would be most promising. A third AI agent would then be programmed to implement this model. The “hacker” thus has a toolbox of different agents at their disposal and sends out only the necessary programmes—while the selection and deployment of AI assistants could itself be handled by another agent. Early proof-of-concept successes suggest that such a world is not far off.

Protecting Against Criminal AI is a Joint Effort

Malicious AI introduces a new dimension to cybercrime. Thanks to AI-supported social engineering attacks, deepfakes, and tailor-made malware, scams have become more sophisticated and harder to detect than ever before. Businesses and individuals alike must learn to protect themselves against these threats. At the same time, governments and the cybersecurity industry face the challenge of developing effective defence mechanisms without stifling innovation. Vigilance, technical safeguards, and international cooperation are essential elements—because in a digital, interconnected world where deception is becoming ever more seamless, critical thinking remains the final line of defence.

Richard Werner

is Security Advisor at Trend Micro.