Your AI fraud detection model is an attack surface. Here is how adversaries exploit it.

Banks across East Africa are deploying machine learning models for fraud detection. Major banks in Rwanda, Kenya, and Uganda have invested in AI-powered transaction monitoring. Fintechs are building ML into their core product from day one. Regulators are encouraging it. Vendors are selling it.

None of this is wrong. ML-based fraud detection catches patterns that rule-based systems miss. It scales. It adapts. For a region processing billions in mobile money transactions annually, it is a necessary evolution.

But here is what is not being discussed in most boardrooms: the AI model itself is an attack surface. And the security teams responsible for protecting these institutions are, in most cases, not equipped to defend it.

This article covers how adversaries target ML-based fraud detection systems, what that means for East African banks specifically, and what your security programme should include to address these risks.

The assumption that needs correcting

Most banks treat their fraud detection model as a black box that sits behind the firewall. It ingests transaction data, it outputs risk scores, and the security team worries about everything else: network perimeter, application vulnerabilities, access controls.

The problem is that ML models are software. They have inputs, outputs, dependencies, and failure modes. They can be manipulated, deceived, and exploited, just like any other component of your technology stack. The difference is that most penetration testing scopes, vulnerability assessments, and security audits do not cover them.

If your fraud detection model has never been subjected to adversarial testing, you do not know whether it works under adversarial conditions. You only know it works on historical data.

Four ways attackers target AI fraud detection

1. Training data poisoning

This is the most patient and damaging attack. The attacker does not try to evade the model. They corrupt it.

How it works: An attacker with access to the data pipeline, whether through a compromised internal system, a third-party vendor, or an insider, introduces carefully crafted transactions into the training data over weeks or months. These transactions are designed to look legitimate but carry characteristics that mirror planned future fraud patterns.

When the model is retrained (as most production models are, on a regular schedule), it learns to associate those characteristics with legitimate activity. The attacker has created a blind spot.

Real-world parallel: In 2023, researchers at CSIRO demonstrated that poisoning just 3% of a financial transaction training dataset could reduce a fraud detection model’s accuracy by over 20 percentage points on targeted transaction types, while overall accuracy metrics remained stable. The model looked healthy on dashboards. It was compromised underneath.

East African context: Many banks in the region retrain models using transaction data that flows through multiple systems, including vendor-managed platforms. If those data pipelines lack integrity controls, poisoning is not hypothetical. It is an operational risk. The bank fraud incident in Rwanda earlier this year demonstrated how vendor platform access can be exploited. The same access could be used to poison training data rather than steal funds directly.

2. Adversarial evasion attacks

This is the most immediate threat. The attacker crafts transactions that are designed to bypass the model’s detection logic.

How it works: The attacker probes the model’s behaviour by submitting transactions and observing which ones are flagged and which are not. Over time, they build an understanding of the model’s decision boundaries. They then modify their fraudulent transactions just enough to fall on the “legitimate” side of those boundaries.

This is not guesswork. Techniques like the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) allow systematic generation of adversarial inputs, even without direct access to the model’s internals.

Practical example: A fraud ring targeting mobile money transactions learns that the bank’s model weighs transaction velocity heavily. Instead of sending 50 rapid transfers, they space transactions at intervals that fall just below the velocity threshold, while varying amounts by small random increments to avoid pattern matching. Each individual transaction looks clean. The aggregate is fraud.

East African context: Mobile money transaction volumes in East Africa create a signal-to-noise problem that makes evasion easier. When a bank processes millions of mobile money transactions daily, an attacker only needs to make their fraudulent transactions look sufficiently similar to the legitimate baseline. The sheer volume of normal activity provides cover.

3. Model inversion and extraction

This attack targets the model itself as a source of sensitive information.

How it works: By sending a large number of carefully designed queries to a model’s API or scoring endpoint and analysing the responses, an attacker can reconstruct the model’s decision logic (model extraction) or infer characteristics of the training data (model inversion). In banking, the training data contains customer transaction histories, so model inversion can leak sensitive financial information.

Why it matters for banks: If an attacker can extract your fraud detection model, they can test evasion techniques offline before deploying them against your production system. They get unlimited attempts without triggering any alerts. If they can invert the model, they gain intelligence about your customers’ transaction patterns, which is valuable for targeted social engineering attacks.

East African context: Several banks in the region expose fraud scoring through APIs that serve mobile banking and agent banking applications. If these endpoints return detailed risk scores or confidence levels rather than simple approve/deny decisions, they provide exactly the feedback an attacker needs for model extraction.

4. Data pipeline manipulation

The model is only as trustworthy as the data it receives at inference time. Corrupting the input pipeline is often easier than attacking the model itself.

How it works: The attacker compromises a system upstream of the fraud detection model, such as the transaction enrichment layer that adds merchant category codes, geolocation data, or device fingerprints. By manipulating these features before they reach the model, the attacker causes the model to make decisions based on false information.

Practical example: A compromised integration layer changes the merchant category code on a suspicious card-not-present transaction from “wire transfer service” to “grocery store.” The fraud model, which weighs MCC heavily in its risk scoring, assigns a low risk score. The transaction clears.

East African context: The integration complexity of East African banking, where a single transaction might traverse core banking, mobile money platforms, payment switches, and vendor-managed middleware, creates multiple points where input data can be manipulated. Each integration point is a potential attack surface. For guidance on securing these integration layers, see our article on API security in banking.

What the frameworks say

The NIST AI Risk Management Framework (AI RMF) and the OWASP Machine Learning Security Top 10 both address these risks explicitly. The EU AI Act classifies financial fraud detection as a high-risk AI application, requiring conformity assessments, human oversight, and robustness testing.

In East Africa, regulators are moving in the same direction. The CBK’s 2024 guidance on technology risk management requires banks to assess risks associated with all technology deployments, which includes AI and ML systems. BNR’s cybersecurity requirements for supervised institutions do not yet include AI-specific provisions, but the general requirement for comprehensive risk assessment and penetration testing applies to AI systems as much as to any other technology.

Institutions that wait for explicit AI regulation before securing their ML systems are taking a risk. The attacks described in this article are happening now, not in some future regulatory cycle.

Practical security checklist for AI fraud detection

This is not a theoretical exercise. These are specific actions your security and data science teams should be taking.

Data pipeline integrity

• Map every data source that feeds your fraud detection model, including third-party vendor data
• Implement checksums and validation rules on training data ingestion
• Monitor for statistical drift in training data distributions (sudden shifts may indicate poisoning)
• Restrict write access to training data stores to the minimum necessary personnel
• Log and alert on all modifications to training datasets

Model security

• Include ML models in your penetration testing scope
• Conduct adversarial robustness testing before deploying models to production
• Rate-limit and monitor API endpoints that expose model predictions
• Return minimum necessary information in model responses (binary decisions, not confidence scores)
• Test model behaviour under adversarial input conditions, not just on historical test sets

Operational controls

• Maintain human review for transactions above defined value thresholds, regardless of model score
• Implement model performance monitoring that tracks accuracy on specific transaction categories, not just aggregate metrics
• Establish a model retraining review process that includes security sign-off
• Document model lineage: what data was used, when it was trained, what changed between versions
• Run red team exercises that include ML evasion scenarios

Governance

• Assign clear ownership of AI/ML security within your risk management framework
• Include AI model risks in your incident response plan
• Report model performance and security metrics to the board, not just to the data science team
• Ensure AI systems are covered in regulatory compliance reporting to BNR or CBK

The human override problem

One of the most dangerous patterns we see in East African banks is over-reliance on the model. When a well-funded AI project is deployed, there is institutional pressure to trust it. Analysts who override the model’s decisions are questioned. Alert volumes are managed by raising thresholds. Human review is reduced to cut costs.

This creates exactly the conditions an adversary wants. A model that is trusted implicitly is a model that can be exploited without triggering human scrutiny.

The correct architecture is defence in depth: ML models as one layer of detection, with human analysts reviewing edge cases and high-value decisions, rule-based systems catching known patterns, and anomaly detection watching for model degradation. No single layer should be trusted absolutely.

What this means for your security programme

If your institution is deploying or operating AI-based fraud detection, your security programme needs to evolve. Specifically:

1. Expand your assessment scope. Your next security assessment should include the ML pipeline: training data sources, model endpoints, feature engineering systems, and model management infrastructure.

2. Test adversarially. Standard functional testing tells you the model works. Adversarial testing tells you whether it works when someone is actively trying to break it. These are fundamentally different questions.

3. Secure the pipeline, not just the model. Most attacks target the data flowing into the model, not the model itself. Input validation, pipeline integrity monitoring, and vendor data source security are where the practical risk reduction happens.

4. Maintain human judgement. AI should augment human decision-making in fraud detection, not replace it. Transactions above material thresholds deserve human eyes regardless of what the model says.

5. Monitor for degradation. A model that slowly becomes less effective at detecting specific fraud patterns is a model that may have been poisoned. Track category-specific performance metrics, not just aggregate accuracy.

How we can help

We are an OSCP-certified security firm based in Kigali, working with banks, fintechs, and regulated institutions across East Africa. Our penetration testing and security assessment engagements now include evaluation of AI and ML system security: data pipeline integrity, API endpoint security, and adversarial robustness assessment.

If your institution is deploying AI for fraud detection and you have not assessed the security of that deployment, we can scope an engagement that covers both the traditional application security and the ML-specific risks outlined in this article. See our penetration testing and security assessments service pages for details, or contact us to discuss your specific requirements.