Horizon 2: A Chance to Acknowledge and Address Artificial Intelligence Risks

Submission to Home Affairs consultation on developing Horizon 2 of the 2023-2030 Australian Cyber Security Strategy

Background

The Department of Home Affairs conducted a consultation on developing Horizon 2 of the 2023-2030 Australian Cyber Security Strategy. While the original strategy forecasted AI's impact on cybersecurity, it generally adopted a "watch and wait" posture rather than proposing specific proactive measures to get ahead of the risks. Action 10.1 proposed to "support safe and responsible use of AI", but Australia has taken few tangible steps since the Bletchley Park Summit on AI Safety to address AI risks.

Subsequent to the strategy's launch in November 2023, AI capability has progressed rapidly in domains relevant to cybersecurity. AI systems are now surpassing humans in persuasion and deception, coding at near superhuman levels, and discovering novel zero-days. AI is now widely adopted, allowing skilled hackers to be better and faster, while lowering the barrier of entry for novice cyber criminals.

Without course correction, Horizon 2 risks continuing the trend of neglecting AI.

Our submission

Our submission argued that Horizon 2 is an opportunity to proactively engage with AI risks in cybersecurity. Leveraging our AI Legislation Stress Test and collaborating with Mileva Security Labs, we detailed six AI risks that challenge traditional cyber security frameworks:

  1. Unreliable Agent Actions - AI agents incompetently pursuing goals, causing harm through errors, deception, or fabrication
  2. Unauthorised Agent Actions - AI agents competently pursuing unintended goals, exceeding user control or authority
  3. Open-Weight Misuse - Malicious adaptation of publicly released AI models after removing built-in safety features
  4. Access to Dangerous Capabilities - AI systems providing specialised knowledge for cyber weapons and advanced hacking techniques
  5. Loss of Control - AI systems escaping human oversight through self-replication or recursive self-improvement
  6. Novel Cybersecurity Vulnerabilities - New attack surfaces created by AI systems, including prompt injection and model inversion attacks

Our ten recommendations:

  1. Launch an Australian AI Safety Institute to provide technical expertise, contribute to prevention research, and build credibility in global standard-setting
  2. Introduce an AI Act with three key features:
    • Impose transparency standards on leading labs and new models
    • Allow an AI regulator to require adherence to safety standards by developers and deployers
    • Give Australia tools to proactively engage with the risks and opportunities of open-weight AI models
  3. Manage vendor risks for AI products, recognising that foreign interference risks differ for AI compared to traditional technology
  4. Establish AI incident response capabilities, treating AI incidents as a special category of cyber incident requiring dedicated response protocols
  5. Host the next AI Safety Summit with a focus on cyber security risks, evaluating models for cyber offensive capabilities, and building global norms around dangerous AI releases
  6. Monitor, measure and analyse the impact of the Strategy with concrete predictions about AI and quantum trends, and commitment to adjust measures based on real-world events
  7. Include new intervention point against "new technology is developed" to ensure conscious choices about emerging technologies rather than accepting risks as inevitable
  8. Expand the SOCI Act to include data centres that train and operate AI models regardless of whether they service other critical infrastructure sectors
  9. Build sovereign AI assurance capabilities, specifically referencing support for the Australian AI assurance technology industry under Shield 5
  10. Build global norms around dangerous AI systems as part of Shield 6 international cooperation efforts

We emphasised that addressing AI risks is essential to unlocking AI's benefits. Research shows Australia is lagging in AI adoption due to lack of public trust, making government interventions that build credible trust in AI safety crucial for giving Australians confidence to use these transformative technologies.