Your AI fraud detection model is an attack surface: how adversaries exploit it

Banks across Africa are deploying machine learning (ML) models for fraud detection. Major banks in Rwanda, Kenya, Nigeria, and South Africa have invested in AI-powered transaction monitoring, fintechs build ML into their core product from day one, regulators encourage it, and vendors sell it.

None of this is wrong. ML-based fraud detection catches patterns that rule-based systems miss, and it scales and adapts as volumes grow. For a region whose payment volumes increasingly run through mobile money rails, it is a necessary step.

What rarely reaches the boardroom is that the AI model is itself an attack surface, and that the security teams responsible for these institutions are often not equipped to defend it.

This article covers how adversaries target ML-based fraud detection systems, what that means for East African banks specifically, and what your security programme should include to address these risks.

The assumption that needs correcting

Most banks treat their fraud detection model as a black box that sits behind the firewall. It ingests transaction data, it outputs risk scores, and the security team worries about everything else: network perimeter, application vulnerabilities, access controls.

The problem is that ML models are software. They have inputs, outputs, dependencies, and failure modes, and they can be manipulated and exploited like any other component of your technology stack. The difference is that most penetration testing scopes, vulnerability assessments, and security audits do not cover them.

If your fraud detection model has never been subjected to adversarial testing, you do not know whether it works under adversarial conditions. You only know it works on historical data.

Four ways attackers target AI fraud detection

1. Training data poisoning

This is the slowest and most damaging of the four: instead of evading the model, the attacker corrupts it.

How it works: An attacker with access to the data pipeline, whether through a compromised internal system, a third-party vendor, or an insider, introduces carefully crafted transactions into the training data over weeks or months. These transactions are designed to look legitimate but carry characteristics that mirror planned future fraud patterns.

When the model is retrained (as most production models are, on a regular schedule), it learns to associate those characteristics with legitimate activity. The attacker has created a blind spot.

Why it is dangerous: NIST's adversarial machine learning taxonomy (NIST AI 100-2) catalogues a body of research showing that targeted poisoning of a small fraction of a training set can degrade a model's behaviour on the specific cases an attacker cares about while leaving aggregate accuracy almost unchanged. The model looks healthy on dashboards. It is compromised underneath, and the targeted blind spot is exactly where the fraud will run.

East African context: Many banks in the region retrain models using transaction data that flows through multiple systems, including vendor-managed platforms. If those pipelines lack integrity controls, poisoning is a live operational risk. The March 2026 bank fraud in Rwanda demonstrated how vendor platform access can be exploited. The same access could be used to poison training data rather than steal funds directly.

2. Adversarial evasion attacks

This is the most immediate threat. The attacker crafts transactions designed to bypass the model's detection logic.

How it works: The attacker probes the model's behaviour by submitting transactions and observing which ones are flagged and which are not. Over time, they build an understanding of the model's decision boundaries. They then modify their fraudulent transactions just enough to fall on the "legitimate" side of those boundaries.

The probing can be systematic. Gradient-based techniques such as the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) generate adversarial inputs from the model's gradients, so they assume white-box access to the model. An attacker without that access can train a substitute model on the decisions they observe, craft adversarial inputs against it and rely on those inputs transferring to the real model, or use query-based black-box methods that need only the model's outputs.

Practical example: A fraud ring targeting mobile money transactions learns that the bank's model weighs transaction velocity heavily. Instead of sending 50 rapid transfers, they space transactions at intervals that fall just below the velocity threshold, while varying amounts by small random increments to avoid pattern matching. Each transaction looks clean on its own; only the aggregate shows the fraud.

East African context: Mobile money transaction volumes in East Africa create a signal-to-noise problem that makes evasion easier. When a bank processes millions of mobile money transactions daily, an attacker only needs to make their fraudulent transactions look sufficiently similar to the legitimate baseline. The sheer volume of normal activity provides cover.

3. Model inversion and extraction

This attack targets the model itself as a source of sensitive information.

How it works: By sending a large number of carefully designed queries to a model's API or scoring endpoint and analysing the responses, an attacker can reconstruct the model's decision logic (model extraction) or infer characteristics of the training data (model inversion). In banking, the training data contains customer transaction histories, so model inversion can leak sensitive financial information.

Why it matters for banks: If an attacker can extract your fraud detection model, they can test evasion techniques offline before deploying them against your production system. They get unlimited attempts without triggering any alerts. If they can invert the model, they gain intelligence about your customers' transaction patterns, which is valuable for targeted fraud and pretexting campaigns against staff and customers.

East African context: Several banks in the region expose fraud scoring through APIs that serve mobile banking and agent banking applications. If these endpoints return detailed risk scores or confidence levels rather than simple approve/deny decisions, they provide exactly the feedback an attacker needs for model extraction.

4. Data pipeline manipulation

The model is only as trustworthy as the data it receives at inference time. Corrupting the input pipeline is often easier than attacking the model itself.

How it works: The attacker compromises a system upstream of the fraud detection model, such as the transaction enrichment layer that adds merchant category codes (MCCs), geolocation data, or device fingerprints. By manipulating these features before they reach the model, the attacker causes the model to make decisions based on false information.

Practical example: A compromised integration layer changes the merchant category code on a suspicious card-not-present transaction from "wire transfer service" to "grocery store." The fraud model, which weighs MCC heavily in its risk scoring, assigns a low risk score. The transaction clears.

East African context: The integration complexity of East African banking, where a single transaction might traverse core banking, mobile money platforms, payment switches, and vendor-managed middleware, creates multiple points where input data can be manipulated. Each integration point is a potential attack surface. For guidance on securing these integration layers, see our article on API security in banking.

What the frameworks say

The NIST AI Risk Management Framework (AI RMF) and the OWASP Machine Learning Security Top 10 both address these risks explicitly. The EU AI Act is often cited here, but read it carefully: Annex III, point 5(b) lists AI used to evaluate creditworthiness or set credit scores as high-risk "with the exception of AI systems used for the purpose of detecting financial fraud". A fraud-detection model is therefore not high-risk under the Act by default, although a model that also drives credit decisions can be. The Act's expectations for high-risk systems (risk management, human oversight, and tested robustness) remain a sensible benchmark for any model that moves money. These references matter for African institutions because they shape what international partners, auditors, and correspondent banks increasingly expect.

In East Africa, regulators are moving in the same direction. The Central Bank of Kenya (CBK) Guidance Note on Cybersecurity requires supervised institutions to assess the risks of their technology deployments, which extends to AI and ML systems. The National Bank of Rwanda's (BNR) cybersecurity expectations for supervised institutions do not yet include AI-specific provisions, but the general requirement for structured risk assessment and penetration testing applies to AI systems as much as to any other technology.

Institutions that wait for explicit AI regulation before securing their ML systems are taking a risk: the attacks described in this article are already in use, whatever the regulatory timetable.

Practical security checklist for AI fraud detection

These are the specific actions your security and data science teams should be taking.

Data pipeline integrity

• Map every data source that feeds your fraud detection model, including third-party vendor data
• Implement checksums and validation rules on training data ingestion
• Monitor for statistical drift in training data distributions (sudden shifts may indicate poisoning)
• Restrict write access to training data stores to the minimum necessary personnel
• Log and alert on all modifications to training datasets

Model security

• Include ML models in your penetration testing scope
• Conduct adversarial resilience testing before deploying models to production
• Rate-limit and monitor API endpoints that expose model predictions
• Return minimum necessary information in model responses (binary decisions, not confidence scores)
• Test model behaviour under adversarial input conditions as well as on historical test sets

Operational controls

• Maintain human review for transactions above defined value thresholds, regardless of model score
• Implement model performance monitoring that tracks accuracy on specific transaction categories alongside aggregate metrics
• Establish a model retraining review process that includes security sign-off
• Document model lineage: what data was used, when it was trained, what changed between versions
• Run red team exercises that include ML evasion scenarios

Governance

• Assign clear ownership of AI/ML security within your risk management framework
• Include AI model risks in your incident response plan
• Report model performance and security metrics to the board as well as to the data science team
• Ensure AI systems are covered in regulatory compliance reporting to BNR or CBK

The human override problem

One of the most dangerous patterns in AI fraud deployments is over-reliance on the model. When a well-funded AI project is deployed, there is institutional pressure to trust it. Analysts who override the model's decisions are questioned. Alert volumes are managed by raising thresholds. Human review is reduced to cut costs.

This creates exactly the conditions an adversary wants. A model that is trusted implicitly is a model that can be exploited without triggering human scrutiny.

The correct architecture is defence in depth: ML models as one layer of detection, with human analysts reviewing edge cases and high-value decisions, rule-based systems catching known patterns, and anomaly detection watching for model degradation. No single layer should be trusted absolutely.

What this means for your security programme

If your institution is deploying or operating AI-based fraud detection, your security programme needs to evolve. Specifically:

1. Expand your assessment scope. Your next security assessment should include the ML pipeline: training data sources, model endpoints, feature engineering systems, and model management infrastructure.

2. Test adversarially. Standard functional testing tells you the model works. Adversarial testing tells you whether it still works when someone is actively trying to break it. The two answer different questions.

3. Secure the pipeline as well as the model. Many attacks target the data flowing into the model rather than the model itself. Input validation, pipeline integrity monitoring, and vendor data source security are where the practical risk reduction happens.

4. Maintain human judgement. Use the model to support your fraud analysts' decisions. Transactions above material thresholds deserve human eyes whatever the model says.

5. Monitor for degradation. A model that slowly becomes less effective at detecting specific fraud patterns is a model that may have been poisoned. Track category-specific performance metrics alongside aggregate accuracy.

How we can help

IMIZI Cyber is a Kigali-based firm providing manual penetration testing for banks, fintechs, telecoms, government, and healthcare institutions across Africa. Our engagements are grounded in recognised offensive-security methodology and evidence-led reporting that a board and a regulator can both act on. The people who find the vulnerabilities are the people who explain them to you.

Our penetration testing and security assessment engagements extend to the AI and ML attack surface: data pipeline integrity, model API endpoint security, and adversarial resilience testing. If your institution is deploying AI for fraud detection and has not assessed the security of that deployment, we can scope an engagement covering both conventional application security and the ML-specific risks outlined here. See our penetration testing and security assessments service pages, or contact us to discuss your requirements.