The email Turing Test

With the growing use of large language models (LLMs) like ChatGPT and Gemini, an increasing amount of the text we engage with is being generated by automated systems rather than a human typing away on their keyboard. In particular, one of the productivity benefits of LLMs is its ability to help us generate, tweak or rephrase emails, which is perhaps the format many of us spend the most time reading and writing. In fact, AI has already infiltrated our mailbox, even beyond the autocomplete function of Office 365 and Gmail, with a wide variety of plugins that can help summarize, compose, correct and organize emails.

Content generation is rapidly expanding and changing, and email messages are part of this evolution. Many of us have observed incoming messages that are “overly polished” or written in a style that deviates from the typical writing of the sender. This begs the question when we receive an email: was this email written by a human or a machine?

One may also ask whether it is genuinely important to answer this question. One can make the argument that if LLMs are indeed quickly becoming an indispensable productivity tool that saves people a ton of time, so be it. Moreover, if the sender of the email is not a native speaker of the language in use, LLMs can be a godsend in helping the writer correct and avoid embarrassing grammar mistakes. LLMs can also help craft an email in a certain intended style for someone not versed in local business etiquette (e.g., someone conducting business in a foreign country or a recent graduate), such as asking the LLM to compose the email with a business-oriented and polite tone.

Is AI-generated text likelier to be spam or phishing?

This is all well and good when the sender is a colleague or friend, or someone legitimately sending us a useful email. But of course, as with any technology, there can be nefarious use cases. LLMs can generate very compelling and personalized spam emails. Even worse, they can craft highly targeted spear-phishing or business email compromise (BEC) attacks, mimicking the tone, style, and signature of the impersonated sender. Therefore, being able to identify whether an email was generated by a human or machine may actually serve as a valuable indicator for an email spam filter or a phishing detector. 

In fact, our research group is actively working on building such a detector, building on recent research advances in this area. The goal is to use an AI model to detect whether the text is AI-generated. Like a snake eating its own tail.

Detecting whether email text was generated by AI is particularly challenging because email text can often be very short and terse. This leaves us with less text to analyze, though the length of the text is a factor we consider when determining whether it was AI-generated. We (and others) have noticed that AI-generated text is often verbose, long-winded, and overly formal. Another factor we consider is the use of certain phrases that are more likely to be generated by an AI than a human. For example, the phrase: “as of my last knowledge update.”

What does the future hold?

It seems that we are entering an age where humans might not directly generate much of the text we read. This will probably have a huge impact on many parts of our lives: our everyday social and business interactions, our classrooms and educational institutions, and the content we consume. It will also have an enormous effect on cybercrime and cybersecurity. The challenge faced by cybersecurity researchers is that criminals now have a lower barrier to creating sophisticated spear phishing and BEC attacks. But the corresponding opportunity is that LLMs and other foundational AI models can be used to detect and prevent these attacks with much higher accuracy. And so, the cat-and-mouse game continues.