Microsoft announces AI bug bounty program

Microsoft is offering up to $15,000 to bug hunters that pinpoint vulnerabilities of Critical or Important severity in its AI-powered “Bing experience”.

“The new Microsoft AI bounty program comes as a result of key investments and learnings over the last few months, including an AI security research challenge and an update to Microsoft’s vulnerability severity classification for AI systems,” says Lynn Miyashita, a technical program manager with the Microsoft Security Response Center.

The Microsoft AI bug bounty program

Microsoft is asking bug hunters to probe the AI-powered Bing experiences on bing.com in Browser, as well as the Bing integration in Microsoft Edge (including Bing Chat for Enterprise) and the Bing integration in the iOS and Android versions of Microsoft Start (news aggregator) and Skype (videoconferencing) mobile apps.

They should report vulnerabilities that could be exploited to:

• Manipulate the model’s response to individual inference requests, but do not modify the model itself (“inference manipulation”)
• Manipulate a model during the training phase (“model manipulation”)
• Infer information about the model’s training data, architecture and weights, or inference-time input data (“inferential information disclosure”)
• Influence/change Bing’s chat behavior in a way that impacts all other users
• Modify Bing’s chat behavior by adjusting client and/or server visible configuration
• Break Bing’s cross-conversation memory protections and history deletion
• Reveal Bing’s internal workings and prompts, decision making processes and confidential information
• Bypass Bing’s chat mode session limits and/or restrictions/rules

The list of out of scope submissions and vulnerabilities is considerable and should be reviewed carefully before starting. For example, AI command/prompt injection attacks that generate content that is shown only to the attacker are not eligible for a bounty reward.

As per usual, the quality of the report accompanying a submission will also influence the amount of the bounty: a critical issue allowing model manipulation can net bug hunters $6,000 if the report is of low quality or $15,000 is it’s of high quality (i.e., it includes the info to reproduce the vulnerability, a reliable proof of concept, and a detailed and correct analysis of the vulnerability).

Probing AI systems for security holes

With the advent of publicly available AI systems based on large language models (LLMs), there’s a pressing need to discover vulnerabilites in them before they are found and exploited by malicious individuals.

Earlier this year, DEF CON’s AI Village hosted a public assessment of LLMs aimed at finding bugs in and uncovering the potential for misuse of AI models.