AI safety and security concerns are coming to the fore

Usually any time there is a great technological leap forward it takes a while before all the cybersecurity implications are fully appreciated but with the rise of large language models (LLMs) the security of an artificial intelligence (AI) that can be attained and maintained is rapidly becoming a major differentiator between one service and another.

For example, Anthropic is now inviting researchers to crack an AI Safety System, dubbed Constitutional Classifiers, that provides a set of principles each instance of its AI model must follow. The company is offering $20,000 to any researcher that can circumvent these safeguards.

That offer comes on the heels of the controversy surrounding the release of DeepSeek, an open source AI model created in China for allegedly a fraction of the cost of other similar AI models. The only issue is the DeepSeek AI model is riddled with security flaws that makes it applicability in an enterprise IT setting suspect.

Arguably, the intense debate surrounding DeepSeek is a godsend for cybersecurity teams. The truth is the AI safety and cybersecurity protocols that organizations have adopted range from nonexistent to lax. In the absence of any formal policy, many end users are sharing sensitive data with providers of AI models with little to no appreciation for how that data might one day be used to train the next generation of a model that is likely going to be capable of spitting that data back out to anybody who cares to craft the right prompt. At the other end of the extreme, some organizations are banning usage of AI models with no ability to enforce those policies. As far as many end users are concerned, AI models are just another shadow IT service to be surreptitiously employed as they see fit.

Hopefully, more organizations in consultation with their cybersecurity teams will soon be implementing the controls needed to put some teeth into the policies they define. In the meantime, cybersecurity teams need to take the time required to understand exactly how each AI service is using the data it gains access to, starting with any commitment made to ensure that customer data isn’t going to be used to train an AI model. The issue is that commitment might only apply to data that the end user has checked a box to specifically exempt from any default setting that no end user is likely to find buried in the hundreds of pages of any software licensing agreement.

It already requires a fairly experienced lawyer to navigate the terms and conditions of most existing software licensing agreements and as more caveats are added in the age of AI, the chances end users are not going to understand the implications of sharing data with an external AI service only increase.

In the meantime, cybercriminals are clearly honing their AI jailbreaking skills, otherwise known as leveraging prompt engineering skill to generate output that circumvents the AI guardrails that are supposed to prevent output containing sensitive data from being created in the first place.

Ultimately, it’s not a matter of if there will be major cybersecurity incidents involving AI so much as it is when and to what degree. The only thing not nearly as certain, however, is how organizations will respond when those breaches, if they have not already been experienced, soon inevitably occur.