● Researchers at the University of Texas at Arlington have developed an innovative plug-in. The new tool, which is the first of its kind, can spot LLM prompts used to produce malicious code with an accuracy of 96%.
● The research team has also identified eight main types of phishing attacks, which when combined with the capabilities of LLMs, provide hackers with a varied arsenal which urgently needs to be neutralized.
There is an urgent need to develop software to prevent hackers from taking advantage of generative AI to conduct phishing campaigns. This is notably the case with LLMs, which can be used for a wide variety of beneficial tasks but have also made it much easier to trick internet users into revealing personal financial data — a fact borne out by the number of scams that are now being generated using these tools. As the head of information security at Booking.com, Marnie Wilking, recently pointed out to AFP, “Over the course of the last year and a half, throughout all industries, there’s been anywhere from a 500 to a 900% increase in attacks, in phishing in particular, across the globe.” The reservation platform executive is convinced that the use of AI to launch attacks that mimic corporate communications is not only responsible for this upsurge, but it has also made phishing harder to detect by improving the design of these emails and the quality of text they contain, which is free of grammar and spelling errors. There is little that Internet giants, banks, and other organizations whose identities are being pirated can do about these attacks apart from mounting campaigns to raise public awareness.
Pirates are adept at manipulating LLMs, using step-by-step prompts until they reach a point where they ask systems to add suspicious URLs or email addresses
Software to automatically detect interaction that is typical of phishing developers
“LLMs can generate both phishing websites and emails that can convincingly imitate well-known brands and also deploy a range of evasive tactics that are used to elude detection mechanisms employed by anti-phishing systems,” explain researchers from the University of Texas at Arlington (UTA) in an article entitled “From Chatbots to PhishBots? — Preventing Phishing scams created using ChatGPT, Google Bard and Claude”. “We have seen discussions on the darknet between ill-intentioned individuals about the best way to use LLMs to construct attacks of this kind,” adds one member of the team, Sayak Saha Roy, a doctoral student at UTA’s Security and Privacy Research Lab, directed by Dr. Shirin Nilizadeh. In response to these dangers, the researchers have developed a plug-in to help AI chatbots like ChatGPT and Google Bard detect and reject prompts designed to create malicious websites.
Our idea was to observe the behaviour of phishing developers: “LLMs already reject requests to generate source code for websites that look like they belong to banks,” points out Shirin Nilizadeh. “However, the pirates are adept at manipulating LLMs, using step-by-step prompts until they reach a point where they ask systems to add suspicious URLs or email addresses.” The automated detection tool developed by the research team, which is transferable across all four commercial LLMs, has an average accuracy of 96% for phishing website prompts and 94% for phishing email prompts — a remarkable achievement when you consider that it is the first tool of its kind.
Plug-ins for open- and closed-source LLMs
When combined with LLM code creation tools, the eight distinct phishing attacks listed below provide hackers with an extensive range of attack strategies. “Generative AI chatbots like ChatGPT and Bard (now Gemini) are not open source, which makes it essential for OpenAI and Google to integrate plug-ins like ours into their content moderation workflows,” explains Sayak Saha Roy. Whereas OpenAI and Google do also provide open-source models such as GPT-2 and BERT, these are aimed more at researchers than the general public and are much less powerful than the latest versions of ChatGPT and Gemini. Building on the findings of this latest project, Saha Roy, who is currently completing a doctorate at UTA, is planning to investigate the research hypothesis that Internet users are more likely to fall victim to AI generated phishing campaigns than attacks created using more traditional methods.
A typology of phishing attacks: :
- Regular phishing attacks that aim to steal users’ credentials with fake log-in pages.
- ReCAPTCHA attacks that present fake log-in pages with ReCAPTCHA challenges.
- QR-code attacks that make use of QR codes to lead users to phishing websites.
- iFrame injection attacks that use iFrames to load malicious websites into legitimate ones.
- DOM classifier avoidance to create phishing websites that escape detection by specific anti-phishing classifiers.
- Browser-in-the-browser attacks that make use of deceptive pop-ups that imitate web browsers to obtain users’ credentials.
- Polymorphic URL attacks that generate new URLs whenever a website is accessed.
- Text encoding designed to ensure that malicious inputs to credential fields escape detection by security provisions in a website’s source code.
Sources :
Saha Roy, P. Thota, K. Naragam and S. Nilizadeh, “From Chatbots to Phishbots?: Phishing Scam Generation in Commercial Large Language Models
