Artificial intelligence in product development is evolving from a tool to an autonomous agent that makes independent decisions and generates code. This transition promises accelerated processes and reduced initial costs, but simultaneously creates more complex challenges concerning responsibility, cost, and control. In practice, it is already observed how initial savings from AI у automation can turn into significant unforeseen expenses due to a lack of proper risk management.
The increase in AI-related incidents directly confirms this trend. The AI Incident Database recorded 362 incidents in 2025, compared to 233 in 2024, according to a Stanford HAI report. This indicates an urgent need for enhanced control and proactive risk management that goes beyond simple model performance monitoring.
Product responsibility: who pays for AI errors?
When an AI agent takes on developer functions, creating code segments, testing them, or making architectural decisions, a fundamental question arises: who is responsible for potential errors, biases, or vulnerabilities integrated into the final product? The traditional model, where responsibility is clearly assigned to the development team, becomes blurred.
Shifting responsibility from humans to AI agents is not just a legal issue but also an operational challenge. If an AI generates code containing a logical error or vulnerability, who is responsible for its detection, correction, and consequences? The developer who integrated the AI agent? The team that approved its output? Or the AI solution provider? The absence of clear responsibility attribution protocols can lead to significant delays in problem resolution, reputational damage, and financial penalties.
The cost of invisible risks: why cheaper upfront means more expensive later
The initial appeal of using autonomous AI agents lies in the promise of rapid prototyping and reduced development costs. However, these savings are often illusory if hidden risks are not considered. Unpredictable AI behavior, data bias leading to discriminatory decisions, or the integration of security vulnerabilities can have significant consequences.
For example, if an AI agent developing a financial product inadvertently creates an algorithm that unfairly assesses credit risks for certain demographic groups, it could lead to multi-million dollar fines and lawsuits. The costs of investigation, correction, compensation, and reputation recovery would far exceed any initial savings. The increase in spending on AI governance platforms, which Gartner predicts will reach $492 million this year and exceed $1 billion by 2030, underscores the growing need for tools to manage these risks.
Control over autonomy: how to maintain manageability in AI development
As AI agents become more autonomous, traditional control mechanisms become insufficient. New approaches are needed to ensure transparency, traceability, and auditability of AI у decisions. This includes not only monitoring the final product but also controlling the development process itself.
To maintain manageability, an architectural approach is recommended, involving: a) clear delineation of roles between humans and AI agents; b) implementation of a ‘human-in-the-loop’ for critical decisions; c) use of tools for visualizing and explaining AI logic; d) automated auditing of AI-generated code and decisions. This allows for early problem detection and understanding of their root causes.
A common mistake: evaluating AI success solely on technical metrics
Often, companies implementing AI in development focus exclusively on technical metrics: model accuracy, code generation speed, or the number of automated tests. While these metrics are important, they do not reflect the full picture of success and potential risks. True success in an AI project is determined by its impact on business outcomes, compliance with regulatory requirements, and the absence of unforeseen negative consequences.
A common mistake is ignoring non-technical aspects: ethical considerations, social impact, legal risks, and long-term operational costs. For instance, an AI agent might generate code with high technical accuracy but which violates data privacy principles or produces biased results for certain user groups. Success evaluation must be comprehensive, including ROI, reduced time-to-market, fewer errors, and compliance with internal policies and external regulations.
Architectural approach to AI risk management
Effective AI risk management requires a systematic architectural approach. It begins with defining clear business objectives and metrics that go beyond mere technical efficiency. In practice, this means integrating risk management principles at every stage of the AI system lifecycle: from design and development to deployment and monitoring.
NIST AI RMF 1.0 structures AI risk management around four functions: Govern, Map, Measure, and Manage. The Govern function involves establishing policies and procedures for AI risk management. Map is about identifying and categorizing risks. Measure is the quantitative and qualitative assessment of these risks. Manage involves developing and implementing strategies to mitigate them.
For banking and financial institutions handling sensitive data, implementing Data Governance and Master Data Management (MDM) practices is critical as a prerequisite for successful AI utilization. Without quality, consistent, and clean data, any AI system risks encountering the ‘garbage-in, garbage-out’ problem. Platforms like UnityBase (an open-source low-code platform developed by InBase) enable the creation of single sources of truth for corporate data, ensuring its quality and integrity. Scriptum (a low-code BPM platform on UnityBase from InBase) can provide version control and document auditing, which is crucial for the transparency and traceability of AI decisions.
Regulatory landscape: EU AI Act and its impact on product development
This year, the regulatory landscape for AI is changing significantly. The European Union is introducing comprehensive regulation – the EU AI Act. Most of its obligations, including requirements for high-risk AI systems and transparency obligations for limited-risk systems, will come into effect on August 2, 2026, according to official publication. This means that companies developing or implementing AI solutions must adapt their processes now to meet new standards for safety, transparency, and ethics.
For IT directors, this means not just formal compliance but integrating responsible AI principles into architecture and operational processes. This includes developing internal AI Governance policies, conducting regular assessments of AI’s impact on human rights and fundamental freedoms, and ensuring the explainability and auditability of AI decisions. System integrators, such as companies within the Intecracy Group alliance, help clients develop and implement architectures that meet these requirements, minimizing risks and ensuring the sustainable development of AI initiatives.
Readiness checklist for managing autonomous AI system risks
- Are business success metrics defined for AI projects beyond technical model accuracy (e.g., ROI, reduced time-to-market, fewer errors)?
- Is there an AI Governance policy that defines responsibility for the development, implementation, and monitoring of AI systems?
- Has a risk assessment been conducted for risks associated with AI agent autonomy (unpredictable behavior, bias, errors)?
- Are control and audit mechanisms in place for AI agent decisions to ensure transparency and traceability?
- Has data preparation (quality assessment, MDM, Data Governance) been performed before using it for training AI models?
- Is there an incident response plan for AI system operations, including recovery and root cause analysis procedures?
- Have the requirements of the EU AI Act been considered during the design and implementation of AI solutions?