The Importance of Managing AI Risk in Software Engineering

How to fast-track safely to hyper-productive AI-augmented engineering

Artificial intelligence is transforming software engineering at extraordinary speed. AI coding assistants and developer tools are now used in about 90% of engineering organisations and the tools are embedded across engineering workflows, promising dramatic productivity improvements.

But with this transformation comes a new challenge for technology leaders: how to unlock the productivity benefits of AI while managing the associated risks.

AI is top of mind for all Boards and as a result, so too is AI-risk management as the ‘first do no harm’ principle is central to Board’s fiduciary responsibilities.

Platforms such as Plandek’s Developer Productivity Insight (DPI) platform are essential tools to help organisations achieve the transition to AI-augmented engineering rapidly and safely.

The CTO’s dilemma in 2026

The expectations placed on CTOs have never been greater.

In boardrooms around the world, two seemingly contradictory mandates dominate the agenda:

• Lead the AI transformation

• Ensure there is no increase in risk.

On the one hand, organisations expect AI to deliver dramatic productivity improvements—some estimates suggest 5-10x increases in engineering productivity. On the other, CEOs and boards are increasingly focused on the risks introduced by uncontrolled AI adoption.

Many boards see AI as either:

• a potential saviour, enabling companies to move faster and innovate more effectively

• or a potential disaster, introducing new operational, legal and security risks – due to the emergence of “AI comprehension debt.”

The stakes are therefore high. Managing existential risk is the primary fiduciary responsibility of Board members. CTOs therefore face the challenge of balancing rapid technological adoption with robust governance.

The fastest technology adoption curve in history

The adoption of AI tools in software engineering has been extraordinarily rapid.

In less than two years:

• AI tools such as GitHub Copilot, Cursor, Claude and Windsurf have reached approximately 95% penetration across engineering teams

• AI is now used across the entire software development lifecycle (SDLC)

• Many organisations are experimenting simultaneously with multiple tools.

However, adoption patterns vary widely.

Some organisations have implemented structured rollout programmes with governance and oversight, while others have allowed ad-hoc experimentation without clear guardrails.

Common challenges include:

• lack of visibility into how AI tools are used

• limited measurement of productivity impact

• little understanding of new risk exposures

• absence of formal policies or controls.

Different paths to AI adoption

Organisations currently fall across a spectrum of AI adoption maturity.

At one end are companies with ad-hoc AI adoption:

• AI tools used informally across teams

• few or no governance guardrails

• limited visibility into usage or outcomes

• unclear understanding of risk

In these environments, organisations may see limited productivity improvements and high risk exposure.

At the other end are organisations that treat AI adoption as a planned change programme. These organisations implement:

• clear policies and governance frameworks

• strong visibility into AI usage

• metrics for measuring productivity and risk

• structured rollouts and guardrails

In these environments, organisations can achieve high productivity gains with lower risk, unlocking the potential of AI-augmented engineering.

AI risk management is now a governance requirement

AI risk management in software engineering is not simply a best practice. Increasingly, it is becoming a formal requirement under corporate governance, regulatory frameworks and emerging AI legislation.

Three major oversight umbrellas apply.

1. Corporate governance responsibilities

Under the UK Corporate Governance Code, Boards of listed companies must oversee and manage material technology risks.

Even in non-listed companies, Directors retain fiduciary responsibilities to shareholders to ensure that significant operational risks—including AI risks—are properly managed.

As AI becomes embedded in software development, it clearly falls within the category of material technology risk that Boards must oversee.

2. Regulatory accountability frameworks

Industry regulators expect organisations to manage AI-related risks under existing governance frameworks.

Examples include:

FCA and SEC accountability frameworks

The FCA Senior Managers and Certification Regime (SM&CR) for example require senior executives to demonstrate accountability for technology risk.

The regime requires SMF24 – Chief Operations Function (which covers IT), to take “all reasonable steps” to manage AI risks and they are required to demonstrate continuous monitoring and governance of AI risks.

Similar expectations to manage AI risk in software engineering apply in healthcare, aviation, critical infrastructure and other regulated sectors.

And ISO’s new 42001 AI Risk Management Framework is focused specifically on the challenge of implementing AI tools at pace whilst managing AI risk, safety and ethics.

3. Emerging AI legislation

The EU AI Act represents the most significant regulatory framework governing AI deployment. Article 26 — Obligations of Deployers comes into force on 2 August 2026.

This regulation applies to:

• organisations operating within the EU

• organisations selling products or services into the EU.

Key obligations include:

• assigning human oversight of AI systems

• monitoring system behaviour

• logging outputs and incidents

• maintaining traceability and accountability.

Software engineering teams deploying AI tools must therefore ensure their processes meet these requirements.

Understanding the risks of AI in software engineering

Despite rapid adoption, the risks introduced by AI tools in software development are often not completely understood. Several categories of risk are emerging – some less obvious than others.

Intellectual property ambiguity

Many generative AI tools do not guarantee exclusive rights to their outputs.

This creates potential issues including:

• unclear code ownership

• open-source license contamination

• patent novelty conflicts

Some tools may also train on user inputs, raising additional concerns around intellectual property and proprietary data.

Security vulnerabilities

AI-generated code may introduce security weaknesses.

Examples include:

• insecure coding patterns

• missing validation logic

• hallucinated APIs or dependencies.

These issues can lead to:

• increased defect rates

• hidden security vulnerabilities

• greater exposure to exploitation.

Regulatory compliance exposure

Regulators increasingly expect organisations to implement:

• AI usage policies

• vendor risk assessments

• data residency controls

• human oversight mechanisms

• explainability for critical decisions

Uncontrolled developer usage of AI tools may violate these expectations.

Data leakage and confidentiality risks

Developers may unintentionally expose sensitive information when interacting with public AI tools.

Examples include:

• proprietary source code

• confidential algorithms

• customer data and staff PII violating data protection laws such as GDPR

• infrastructure credentials

How Plandek enables safe AI-augmented engineering

To unlock AI’s productivity potential safely, organisations need visibility, measurement and governance.

Plandek is a leading Developer Productivity Insight (DPI) platform. It sits across DevOps toolsets to extract the data-footprint of engineering teams (including the use of AI tools). It provides an intelligence layer which enables organisations to:

• Track and drive safe AI tool roll-out – and understand AI tool impact, in order to drive productivity and maximise safety

• Build a culture of engineering excellence:

  • align and improve around KPIs

  • better focus resource on value creation

• Manage and report AI risk, productivity & business value to stakeholders

The RACER framework for safe AI adoption

Plandek provides a structured approach to managing AI adoption through its RACER methodology.

This framework enables organisations to:

• Track and control AI tool Rollout

• Understand AI tool usage and Adoption across teams and different areas of the code base

• Identify and remove Constraints to AI tool use thereby increasing the speed of transition and productivity impact across the SDLC

• Measure and augment the Engineering Impact of AI tools – in terms of reducing AI risk and increasing productivity impact

• Quantifying the business Results of transitioning to AI-augmented engineering – in terms of accelerated value delivery.

By combining structured governance with real-time engineering data provided by the Plandek platform, the RACER framework helps organisations transition safely toward AI-augmented engineering.

Measuring AI risk in software engineering

Effective AI governance and risk management requires measurable indicators of AI risk.

Plandek helps organisations monitor key AI risk and compliance metrics across several dimensions.

Example AI risk management metrics are shown below.

Code quality and defect risk

Before AI risk can be managed, organisations must first identify where AI-generated code exists.

Key metrics include:

• percentage of AI-generated code that is traceable

• percentage of pull requests indicating AI assistance

• repositories with AI usage logging enabled

• production code with unknown origin

Organisations can also compare quality outcomes between AI-generated and human-written code using metrics such as:

• defect density per 1,000 lines of code

• rework rates

• review cycles

• post-merge bug rates

Security vulnerability exposure

AI-generated code can introduce security issues including:

• injection flaws

• insecure deserialization

• hardcoded secrets

• weak authentication patterns

Key metrics include:

• vulnerability density in AI-generated code

• percentage of AI code triggering SAST or DAST alerts

• mean time to remediate AI-related vulnerabilities

• percentage of critical vulnerabilities linked to AI code

Architectural integrity

AI tools may bypass internal design standards or introduce architectural drift.

Key indicators include:

• AI-generated code violating architectural policies

• services introduced outside approved patterns

• technical debt increases linked to AI output

• backlog growth attributable to AI-generated code

Software supply chain and licensing risk

AI outputs may include:

• incompatible open-source licenses

• copyrighted material

• outdated or vulnerable dependencies

Monitoring metrics include:

• dependencies flagged by software composition analysis tools

• licensing conflicts

• CVEs introduced through AI-generated dependencies

Hallucination and logical integrity risk

AI models may fabricate:

• nonexistent APIs

• incorrect configuration values

• invalid security assumptions

Metrics to track include:

• references to non-existent APIs

• runtime failures caused by hallucinated logic

• correction rates for AI-generated documentation

Human oversight effectiveness

The biggest risk is not that AI makes mistakes—it is whether humans detect them.

Metrics include:

• percentage of AI pull requests reviewed by senior engineers

• review duration comparisons

• rejection rates

• auto-merge rates without human review

Production stability and incident monitoring

Ultimately, AI risk manifests in production.

Important indicators include:

• production incidents linked to AI-generated code

• severity distribution of those incidents

• time to detect and recover

• financial impact of AI-related failures

Safety first – the future of AI-augmented engineering

AI will fundamentally reshape how software is built.

Organisations that successfully manage the transition will gain major advantages in:

• productivity

• innovation speed

• engineering efficiency

• competitive positioning

But achieving these benefits requires governance, measurement and visibility.

By combining engineering data with AI risk frameworks, organisations can safely move toward hyper-productive AI-augmented engineering.

Platforms such as Plandek provide the insights and structure required to manage this transition—helping technology leaders deliver faster while maintaining the levels of safety, compliance and oversight demanded by Boards and regulators.

For more information contact:

Charlie Ponsonby

cponsonby@plandek.com

Will Lytle

wlytle@plandek.com