Fraud Metrics: How to Measure the Effectiveness of an Anti-Fraud Architecture

The Boundaries of Measuring an Anti-Fraud Architecture

Before calculating precision, recall, or fraud loss, it is necessary to fix the frame of reference: what exactly is measured, at what moment the result is recorded, and along which axes the data is broken down. Without these agreements, the same numbers will mean different things to an analyst, a product team, and a CFO. Two anti-fraud loops with identical models will show incomparable metrics if one counts events at the session level and the other at the transaction level, or if the fraud confirmation windows differ by an order of magnitude.

Which Event Is the Unit of Measurement

The unit of measurement of an anti-fraud architecture is the atomic event to which the system’s decision and the final outcome (fraud or legitimate) are tied. The choice of unit determines what ends up in the numerator and denominator of each metric, so an error here distorts the entire reporting.

In payment anti-fraud, this unit is most often an individual transaction — an authorization request with a unique identifier, an amount, a timestamp, and a set of attributes of the payer and recipient. It is by transaction that approve/decline is recorded, and it is to the transaction that a chargeback or confirmation of legitimacy is tied.

Depending on the business context, the unit may be a different object:

• Application — in lending and insurance, where fraud is detected at the underwriting stage rather than at payment.
• Session or authentication attempt — in onboarding and KYC scenarios, where anti-fraud assesses the totality of the user’s actions: document upload, passing the liveness check, biometric matching.
• Case — an aggregate of several related events grouped by user, device, or chain of operations. A case is convenient for investigations but complicates comparison with models operating at the level of individual requests.

The unit of measurement must be the same at all stages: in the logs of the anti-fraud engine, in the metrics data mart, and in the financial reports. If the model assesses each transaction while the operations team measures effectiveness by cases, the model’s precision and the process’s precision will describe different realities. When designing reporting, the first step is to fix a mapping table: which object is considered an event, what its mandatory attributes are, and how it is identified across all systems.

How the Anti-Fraud Decision and the Final Outcome of an Event Are Recorded

Two independent facts are tied to each event: the anti-fraud system’s decision and the final outcome — the event’s real status (fraud or a legitimate operation). The «decision + outcome» pair forms a cell of the confusion matrix: a true positive, a false positive (false block), a false negative (missed fraud), or a true negative (correct pass).

The anti-fraud decision is the action applied to the event at the moment of processing: approve (let through), decline (reject), review (send to manual review), challenge (request additional verification — 3-D Secure, OTP). In a multi-layered architecture, the decision may be formed sequentially: first-level rules → scoring model → threshold logic → manual review queue. For correct accounting, both the intermediate decisions of each layer and the final aggregated decision that determined the event’s fate are recorded.

The decision is recorded together with the context: the score value, the rule or feature with the largest contribution that triggered, the threshold applied, the model version, and the rule set. Without these data it is impossible either to reproduce the decision-making logic or to conduct a retrospective analysis when thresholds change.

The final outcome is the confirmed status of the event, obtained from a source external to the anti-fraud system. For transactions, the main signals are chargebacks, customer reports of unauthorized operations, investigation results, and data from payment systems (Visa and Mastercard TC40/SAFE reports). For applications — the fact of default, established during fraud verification. For KYC events — the result of manual document review, cross-checking against external databases, or confirmation of the use of a fake identity.

The key difficulty: the final outcome becomes known later than the decision, sometimes significantly. A chargeback may arrive 30–120 days after the transaction. This creates the problem of incomplete labeling: at the moment the report is generated, some events do not yet have a confirmed status.

How the Fraud Attribution Window and the Confirmation Delay Are Set

Too short a window leaves some real frauds unconfirmed — they are erroneously counted as legitimate operations, and the metrics show understated losses and overstated precision. An excessively long window deprives reporting of timeliness: fresh data is constantly recalculated, and decisions are postponed.

The length of the window depends on the type of business and the confirmation channel:

• Payment transactions. The main signal is a chargeback. Under Visa and Mastercard rules, a cardholder can initiate a dispute in the standard case within 120 days of the operation date, but the median time for a chargeback on fraudulent operations to arrive falls within 30–45 days. An operational window of 60–90 days is a common compromise between completeness and timeliness.
• Credit applications. Fraud manifests after the first default occurs or during an unscheduled document check — the window may be 90–180 days.
• KYC/onboarding. Forged documents or compromised identities are often detected during repeat checks, counterparty complaints, or overlaps with external stop lists. A fixed window is less applicable here — more often, retrospective relabeling is used: when a new signal appears, the event’s status is updated regardless of how long ago it occurred.

For operational monitoring, the concept of a «mature cohort» is applied: metrics are calculated only for events whose attribution window has fully elapsed. In parallel, a «preliminary» view is built on fresh data with a note that the figures will be recalculated after the cohort matures. This approach combines timeliness with accuracy and makes the source of uncertainty in each report explicit.

Which Breakdowns Are Mandatory for Reporting and Diagnostics

Aggregated metrics mask local problems. An overall detection rate of 85% may be composed of 98% on card payments and 40% on account-to-account transfers. An average false positive rate of 2% hides the fact that for a new channel it reaches 12%, while for the main one it is 0.5%. Without decomposition into breakdowns, the anti-fraud team will be unable either to diagnose degradation or to prioritize improvements.

The minimum set of breakdowns depends on the specifics of the business, but the following axes are critical for most anti-fraud architectures:

Category	Description
Channel and product	Mobile application, web, API integration, POS terminal — each channel has its own fraud profile, its own baseline share of fraudulent events, and its own level of expected loss. Metrics without a breakdown by channel and product are unsuitable for operational decisions.
Type of anti-fraud decision	A breakdown by decision (auto-approve, auto-decline, sent to review, challenged) shows what share of the flow the system processes automatically and what share it passes to manual review. This is directly related to the load on the operations team and the cost of the process.
Customer segment	New customers and returning users exhibit different patterns: for new ones there is insufficient behavioral history, so the false positive rate is usually higher. A breakdown by segment makes it possible to reveal whether the system is sacrificing the conversion of new customers to reduce overall losses.
Geography and currency	Fraud patterns vary by region. A separate view helps to promptly detect fraud surges in a specific country and apply local countermeasures.
Amount range	Metrics weighted by the number of events hide the concentration of losses in the high-value segment. A breakdown by amount range makes the picture of losses and false blocks more accurate.
Model and rule-set version	When updating models or adjusting rules, it is necessary to compare the effectiveness of the previous and current versions on the same flow. Without this breakdown, it is impossible to understand whether the change led to improvement or degradation.

Each of the listed breakdowns must be available in the metrics data mart both as a filter and as a grouping axis — for both regular reporting and ad hoc diagnostics during anomalies: a sharp rise in the decline rate, a spike in losses, or customer complaints about blocks.

Data and Labeling for Calculating Metrics

Any anti-fraud metric is derived from two entities: the system’s decision and the event’s real outcome. The decision is recorded at the moment of processing, the real outcome later, sometimes weeks later. Between them lies the task of labeling: which source confirms that an event was fraud or a legitimate operation, how to handle the absence of confirmation, and in what structure to store everything so that metrics are calculated reproducibly. Without a reliable layer of data and labeling, the precision, recall, fraud loss, and ROI of anti-fraud turn into numbers with no foundation.

Sources of Fraud and Legitimacy Confirmation

The main sources of fraud confirmation:

• Chargebacks and disputes. The customer or the issuing bank disputes the operation. For payment fraud this is the most reliable signal, although it arrives with a delay of 30–120 days and includes a share of «friendly fraud» — disputing a purchase that was actually made. The chargeback reason code makes it possible to separate unauthorized operations from disputes over product quality or processing errors.
• Investigation results. Anti-fraud analysts review suspicious events manually: they analyze logs, link data from different systems, and, if necessary, request confirmation from the customer. Investigation labels are the most accurate but expensive — the analysts’ resource is limited, and it is impossible to cover every event with an investigation.
• Customer feedback. Reports of unauthorized access, complaints about charges, support inquiries. A valuable signal, but with incomplete coverage: some customers do not notice small charges, and some do not reach out.
• Signals from partners and external databases. Data from payment networks (TC40/SAFE in the Visa and Mastercard ecosystems), law-enforcement agencies, and industry consortia for exchanging fraud information. These sources record confirmed fraud, but with their own delay and without full coverage.

To confirm legitimacy, the reverse logic is used: if an event has received no negative signal over the entire attribution window period, it is considered legitimate. The assumption works only with a sufficiently long window — otherwise some fraud will not yet have had time to manifest, and the labels will be biased.

Rules for Accounting for Disputed Cases and Missing Labels

Far from every event receives an unambiguous label. Some cases remain in the «gray zone», and the reliability of all derived metrics depends on the rules for accounting for such cases.

Disputed cases arise when signals contradict each other. The customer claims fraud, but the investigation finds no confirmation. A chargeback is received with a «fraud» reason code, yet the logs show standard cardholder activity. «Friendly fraud» is a separate category: formally the operation is disputed, but in fact the goods were received. If all such cases are unconditionally labeled as fraud, the model’s precision will be artificially understated. If they are excluded, the loss metrics will stop reconciling with the accounting data.

A working approach is to introduce a separate labeling status for disputed cases and to calculate metrics in two modes: «strict» (confirmed fraud only) and «extended» (confirmed + disputed). The difference between them shows the sensitivity of the metrics to the interpretation of borderline cases. A significant gap between the modes signals insufficient labeling quality and the need to expand investigation coverage.

Missing labels are an even more common problem. An event was blocked by anti-fraud, the customer did not reach out, there is no chargeback, and no investigation was conducted. Whether this was real fraud or a false block is unknown. In machine learning this is selection bias: the model is evaluated only on events with feedback, while blocked ones fall out of the evaluation.

Several techniques are used to reduce the bias:

• Selective pass-through: a small share of suspicious events is deliberately let through or processed in observation mode without blocking, in order to obtain labels for the «borderline» zone.
• Retrospective labeling: analysts periodically take a random sample of blocked events and conduct an investigation on them.
• Separate accounting: when calculating metrics, blocked events without labels are counted as a separate category and not mixed with confirmed fraud.

Each approach carries its own trade-off: selective pass-through allows controlled losses for the sake of metric accuracy, retrospective labeling requires analyst resources. The choice depends on the cost of an error in the specific business and the volume of the blocked flow.

How to Store Versions of Rules and Models with a Link to Thresholds

An anti-fraud system is not a static filter. Rules are updated, thresholds shift, ML models are retrained. If a change is not recorded, in a month it will be impossible to establish why precision fell: whether fraudster behavior changed or the cause is a new threshold deployed without a record.

The minimal versioning scheme includes three elements:

Rule-set version. Each configuration change — adding, removing, or modifying a condition or threshold — is recorded as a new version with a timestamp, author, and reason. A convenient format is storing the rules as code or configuration files in a version control system (Git), where each commit automatically contains a diff, date, and description.

ML model version. The model is identified by a unique tag: the training date, the hash of the training set (or its time range), and the set of hyperparameters. At each deployment to production, the following are recorded: the version, the activation date, the decision threshold, and offline metrics (precision, recall, AUC on the validation set).

Linking the decision to the version. Each decision of the anti-fraud system is saved together with the identifier of the rule and model version that produced it. Without this field, retrospective analysis is impossible: it is impossible to understand which threshold was in effect at the moment a specific event was processed.

Such linkage makes it possible to compare metrics before and after a threshold change, isolate the effect of a new rule, and roll back changes when indicators degrade. It is also necessary for audit and compliance — regulators and internal auditors expect the company to be able to explain on what basis a specific decision was made at a specific moment.

A Metrics Data Mart Linking Events and Decisions to Cases and Money

Data on events, decisions, labels, and rule versions is stored in different systems: transaction processing, the anti-fraud engine, CRM, the investigation system, accounting. To calculate metrics, they need to be consolidated into a single analytical table — a data mart — where one row corresponds to one event and contains everything needed for the calculations.

The data mart performs three functions. The first is reproducibility: any metric is calculated by a SQL query to a single table, without ad hoc joins and manual exports. The second is a single source of truth: the discrepancy between the anti-fraud decision and the final label is visible in one row rather than in five disparate systems. The third is analytics in the mandatory breakdowns: channel, product, fraud type, customer segment, rule version.

The data mart is updated taking into account the label delay. A chargeback that arrived 45 days after a transaction must update the label of the corresponding event, and the metrics for the period must be recalculated. The data mart is not a «snapshot» — it is supplemented as new confirmations arrive.

The Minimum Set of Data Mart Fields for Anti-Fraud

The composition of the fields depends on the specifics of the business, but below is the core without which calculating the main metrics is impossible:

• event_id — a unique identifier of the event (transaction, session, application).
• event_timestamp — the time the event occurred.
• event_amount — the amount in monetary terms (for calculating fraud loss and prevented loss).
• event_currency — the currency (for correct aggregation).
• channel / product / segment — breakdowns for reporting.
• antifraud_decision — the system’s decision: approve, decline, review.
• antifraud_score — the numerical score for building threshold curves (ROC, precision-recall).
• decision_timestamp — the time the decision was made (for calculating latency).
• rules_version_id — the identifier of the rule-set version at the moment of the decision.
• model_version_id — the identifier of the ML model version at the moment of the decision.
• threshold_applied — the threshold by which the score was converted into a decision.
• final_label — the final label: fraud, legitimate, disputed, unlabeled.
• label_source — the source of the label: chargeback, investigation, customer_report, partner_signal, timeout (the label assigned upon expiry of the window without signals).
• label_timestamp — the time the label was assigned or last updated.
• case_id — the identifier of the investigation (if conducted), for linking with the case-management system.
• loss_amount — the actual losses on the event (may differ from event_amount in the case of partial refunds or compensations).
• prevented_amount — the prevented losses (filled in for blocked events confirmed as fraud).

This set makes it possible to calculate all the key metrics: from the confusion matrix and precision/recall to fraud loss rate, prevented loss, the cost of a false block, and anti-fraud ROI. The breakdown fields provide analytics in the mandatory dimensions, and the version fields provide the ability to isolate the effect of a specific change.

Fraud Detection Quality Metrics

An anti-fraud system makes decisions on each event: let through, block, or send to review. Some decisions will turn out to be correct, some erroneous. Quality metrics measure this ratio precisely, in order to manage it deliberately.

The key difficulty is class imbalance. In payment systems, the share of fraudulent operations rarely exceeds 0.1–0.5% of the total flow: for every real fraud transaction there are thousands of legitimate ones. Under such conditions, the familiar accuracy (the overall share of correct answers) loses its meaning: a model that declares everything legitimate will show an accuracy above 99% but will not detect a single fraud. That is why a set of specialized metrics is used for anti-fraud, each of which answers its own question.

The Confusion Matrix for Anti-Fraud Decisions

The foundation of quality assessment is the confusion matrix. It classifies each decision along two axes: what the system decided and what the real outcome was. For the binary «fraud / not fraud» task, four categories result:

• True Positive (TP) — the system blocked the event, and it really turned out to be fraudulent. A correct trigger.
• False Positive (FP) — the system blocked the event, but it was legitimate. A false block, causing direct damage to customer experience and revenue.
• True Negative (TN) — the system let the event through, and it really was legitimate. A correct decision, making up the overwhelming majority of the flow.
• False Negative (FN) — the system let the event through, but it turned out to be fraudulent. Missed fraud, leading to direct financial losses.

For anti-fraud tasks, both axes of error are critical. FPs create a load on manual review and increase customer churn. FNs lead to losses, compensations, and reputational risks.

The matrix is built on labeled data — on events for which the final outcome is confirmed. If confirmation arrives with a delay or covers not the entire flow, the matrix will be incomplete. When interpreting quality metrics, this must be constantly taken into account: a metric is no more accurate than the labeling on which it was calculated.

The Share of True Positives on Fraud

Precision (the share of true positives, or detection accuracy) answers the question: what part of the blocked events was really fraudulent?

Precision = TP / (TP + FP)

If the system blocked 1,000 events in a month, of which 800 were confirmed as fraud and 200 turned out to be legitimate, precision equals 80%. Every fifth block is erroneous.

High precision is critical for several reasons. False blocks reduce conversion: a customer whose transaction is declined without reason is highly likely to leave for an alternative service. Every false positive that ends up in the manual review queue generates operational costs — analyst time, load on support, delay in service. In the payment business, the cost of a single false block can many times exceed the average transaction value if the loss of the customer’s lifetime value is taken into account.

The Completeness of Fraud Detection

Recall (the completeness of detection, or sensitivity) answers the reverse question: what share of all the real fraudulent events did the system detect?

Recall = TP / (TP + FN)

If 500 confirmed fraud events occurred in a period and the system detected 400, recall is 80%. The remaining 100 events are missed fraud, leading to direct losses.

In anti-fraud, recall takes a priority position, because missing fraud, as a rule, costs more than a false block. A missed fraudulent transaction is a confirmed loss: the funds are already gone, and recovering them is far from always possible. A systematically low recall signals that some fraud patterns are not detected at all and that losses will mount.

The pursuit of maximum recall without controlling precision leads to the opposite problem — mass false blocks. Most anti-fraud teams set a target level of recall (for example, «detect at least 85% of fraud by amount») and then optimize precision under that constraint.

The combination of precision and recall is formalized through the F1-score — the harmonic mean: F1 = 2 × Precision × Recall / (Precision + Recall). F1 is convenient for a quick comparison of models, but for business decisions it is insufficient: it assigns equal weight to both types of error, whereas the cost of FP and FN differs several times over. When choosing the operating point, priority is given to the economic analysis of errors.

The Shares of False Blocks and Missed Fraud

False Positive Rate (FPR) — the share of legitimate events erroneously blocked by the system:

FPR = FP / (FP + TN)

FPR answers the question: what percentage of legitimate customers suffered from false positives? The difference from precision is fundamental: precision counts errors among the blocked, while FPR counts them among all legitimate ones. With strong class imbalance, even a small FPR translates into a huge absolute number. If there are a million legitimate transactions a month, an FPR of 1% means 10,000 erroneously declined operations.

False Negative Rate (FNR) — the share of fraudulent events missed by the system:

FNR = FN / (FN + TP) = 1 − Recall

FNR is the mirror metric to recall. It directly shows the scale of the «leak»: an FNR of 20% means that every fifth fraud slips past anti-fraud.

Joint analysis of FPR and FNR gives a complete map of errors. Lowering the trigger threshold reduces FNR (more fraud is caught) but increases FPR (more legitimate ones are blocked). This is a fundamental trade-off that cannot be eliminated — the balance can only be shifted in a direction economically advantageous for the specific business.

In reporting, it is advisable to track FPR and FNR in three projections: percentage (model quality), absolute number (operational load), monetary equivalent (business consequences).

Threshold Curves and Choosing the Operating Point

An anti-fraud model returns a score — a numerical estimate of the probability of fraud. The binary decision to «block or let through» is formed by comparing the score with a threshold. Changing the threshold redistributes errors: when lowered, recall grows but precision and FPR worsen; when raised, the opposite.

Two tools make it possible to visualize the trade-off:

The ROC curve is plotted in the coordinates «True Positive Rate (recall) — False Positive Rate». Each point on the curve corresponds to one threshold value. The area under the curve (ROC AUC) summarizes the model’s ability to distinguish fraud from legitimate at all possible thresholds: 1.0 is perfect separation, 0.5 is random guessing.

The Precision-Recall curve (PR curve) is plotted in the coordinates «Recall — Precision». For tasks with strong class imbalance, characteristic of anti-fraud, the PR curve is more informative than ROC. The reason: the denominator of FPR contains the number of legitimate events, which is orders of magnitude larger than the number of fraudulent ones. Because of this, FPR remains low even with a significant number of false blocks, and the ROC curve looks more optimistic than the real quality. The PR curve is free of this effect, since precision and recall operate only on the number of fraud events and model triggers.

ROC AUC is convenient for comparing models at the development stage. PR AUC is closer to operational reality and better shows the model’s behavior in production.

The choice of the operating point on the curve is a business decision determined by the ratio of the cost of FP and FN. A typical approach: fix a target recall (for example, ≥ 90% by fraud amount) and choose the threshold with the maximum precision under that constraint. An alternative path is to set an acceptable FPR (for example, ≤ 0.5% of legitimate transactions) and look for the threshold with the maximum recall. In both cases, the operating point is fixed, documented, and revised when the attack profile or business conditions change.

The Detection Rate and Its Correct Denominator

Detection rate is one of the most common and at the same time most ambiguous metrics in anti-fraud. The formula is simple — the share of detected fraud out of the total number of fraud events. In effect this is recall, but errors in the calculation occur regularly and are almost always related to an incorrect denominator.

The correct denominator is the total number of confirmed fraudulent events for the period, including those missed by the system. It is precisely «including the missed ones» that is the key condition. If only the detected cases are counted in the denominator, the metric turns into a tautology and always equals 100%.

Typical errors in forming the denominator:

• Frauds detected after the fact through chargebacks, customer complaints, or external investigations are not counted — the detection rate is overstated.
• The denominator includes only events that passed through the system. If part of the flow bypasses anti-fraud (whitelisted transactions, an alternative channel), fraud in that segment ends up in neither the numerator nor the denominator. The metric is valid for the covered segment, but not for the business as a whole.
• The aggregation period does not coincide with the fraud confirmation window. If the detection rate is calculated for January while chargebacks on January transactions arrive through March, the denominator at the moment of calculation will be incomplete. The solution is to calculate the metric with a delay sufficient to accumulate confirmations, or to explicitly indicate the degree of data maturity.

Metrics of Losses and Prevented Damage

Precision, recall, and the confusion matrix answer the question «how accurately does the model work». The business needs a different answer: how much money was lost, how much was saved, and whether the result justifies the investment. Financial anti-fraud metrics translate technical indicators into the language of profit and loss. Without them, it is impossible to justify a budget, compare the effectiveness of two rule versions, or prioritize improvements.

Fraud Losses as an Amount and a Share of Turnover

Fraud Loss is the total amount of funds lost as a result of fraudulent actions over the reporting period. The basic reference point for the entire economics of anti-fraud.

The absolute value of Fraud Loss shows the volume of direct damage, but in isolation it is uninformative. A company with a turnover of 500 million rubles and losses of 5 million is in a different situation than a company with a turnover of 50 billion and the same 5 million. The standard practice is to calculate the Fraud Loss Rate:

Fraud Loss Rate = (Fraud Loss / Total transaction volume) × 10,000

The result is expressed in basis points (bps): 1 bps = 0.01%. This unit is adopted in the payment industry and the financial sector, since fraud shares usually amount to tenths and hundredths of a percent.

For a correct calculation of Fraud Loss, it is necessary to define the composition of the numerator:

• Direct losses — the amount of transactions recognized as fraudulent and not recovered (chargebacks, debits, customer compensations).
• Operational losses — the costs of investigation, restoring access, reissuing cards or credentials, legal support.
• Fines and penalties — payment-system fines for exceeding chargeback thresholds, regulatory penalties.

The denominator also requires a precise definition: the volume of approved transactions, the volume of all initiated transactions, or revenue. The choice affects the final value, so it must be fixed in the methodology and not changed between comparison periods.

When monitoring Fraud Loss Rate, the dynamics matter. Growth in turnover with an unchanged absolute level of losses means a decline in the indicator — and vice versa. Seasonal spikes (sales, holiday periods) can temporarily distort the picture, so to identify stable trends it is advisable to use a rolling average over 3–6 months.

Prevented Losses and Net Losses After Compensations

Prevented Fraud Loss is the amount of funds that the anti-fraud system did not allow to be withdrawn or debited. A key metric for justifying the value of anti-fraud, but it is precisely this one that is associated with the most methodological errors.

For a complete picture, Net Fraud Loss is needed — the net losses taking into account all compensations:

Net Fraud Loss = Gross Fraud Loss − Chargeback recoveries − Insurance compensations − Recoveries from fraudsters − Other reimbursements

Gross Fraud Loss reflects the full amount of confirmed fraud before recoveries. Net Fraud Loss is the real damage remaining on the organization’s balance sheet. The difference between them is an indicator of the effectiveness not only of detection but also of the recovery processes.

The ratio of prevented and net losses forms the Prevention Rate:

Prevention Rate = Prevented Fraud Loss / (Prevented Fraud Loss + Net Fraud Loss) × 100 %

A Prevention Rate in the range of 90–97% is characteristic of mature anti-fraud systems in the financial sector, although specific values depend on the type of product, the risk profile, and the attack landscape.

Losses and Prevented Damage in the Mandatory Breakdowns

Aggregated Fraud Loss and Prevented Fraud Loss are useful for top-level reporting but unsuitable for diagnostics. For operational decisions, the metrics are calculated in breakdowns:

By fraud type. Losses from credential theft, social engineering, forged documents, synthetic identities, and promo abuse have different economics and require different controls. Mixing them into a single figure hides the growth of individual categories.

By channel and product. Mobile application, web interface, API integration, POS terminal — each channel has its own risk profile. Losses on credit cards, debit cards, transfers, and e-wallets are analyzed separately.

By stage of the customer journey. Fraud at onboarding (registration with forged documents, synthetic identities), at authentication (session hijacking, biometric spoofing), during an operation (unauthorized transactions) — each stage is associated with its own set of controls and level of losses.

By geography. Regional breakdowns are critical for companies in several markets: legislation, attack schemes, and available verification tools differ.

By decision cohort. Losses among automatically approved operations, operations after manual review, and operations after escalation show the effectiveness of each processing stage.

By time period. In addition to the standard weekly and monthly views, it is useful to track losses tied to specific events: the launch of a new product, a change in thresholds, a model update.

A practical criterion for the completeness of the breakdowns: if, during an anomalous rise in Fraud Loss Rate, it is impossible to determine within 15–30 minutes which segment caused the rise, the set of breakdowns is insufficient.

Attribution of Prevented Damage Across the Layers of the Anti-Fraud Architecture

A modern anti-fraud architecture consists of several sequential layers: identity and document verification, device and session checks, a scoring model, real-time rules, behavioral analysis, manual review. Without attributing the contribution of each layer, it is impossible to understand which components justify the costs and which duplicate one another.

Attribution answers the question: which layer stopped a specific fraudulent operation? For each blocked event confirmed as fraud, the first layer that issued the blocking decision is determined. The sum of such events for each layer forms the Layer-Attributed Prevented Loss:

• The identity verification layer — operations rejected at the stage of document verification, biometrics, or liveness detection.
• The device/session intelligence layer — blocks based on signs of a compromised device, anomalous geolocation, emulators.
• The scoring model — operations that scored above the cutoff threshold.
• Real-time rules — triggers based on velocity limits, blacklists, transaction patterns.
• Manual review — fraud detected by analysts when working through the queue.

Attribution to the first layer does not show the depth of protection: if one layer is disabled, some operations will be intercepted by the next ones. To assess the incremental contribution of each layer, controlled experiments are used, described in the section on proving effectiveness.

Attribution is also necessary when calculating the cost of each layer: knowing the cost of operating a component and the damage it prevented, one can assess its individual ROI and justify a decision to scale or replace it.

How to Calculate Anti-Fraud ROI from Prevented Losses and Costs

Anti-fraud ROI links the results of the system’s work with the costs of creating and operating it:

ROI = (Benefits − Costs) / Costs × 100%

The difficulty lies not in the formula but in correctly filling in the numerator and denominator.

Benefits:

1. Prevented Fraud Loss — confirmed prevented losses (with deduplication of attempts and confirmation of labels).
2. Reduction of operational costs — savings from automating checks previously performed manually: the difference in the cost of processing before and after implementation, multiplied by the volume of processed events.
3. Reduction of fines and penalties — the difference between actual and projected fines without anti-fraud control (given historical data or industry benchmarks).
4. Preserved conversion — revenue preserved by reducing the share of false blocks. This component is often underestimated, although it can exceed Prevented Fraud Loss.

Costs (Total Cost of Ownership):

1. Licenses and subscriptions — the cost of the platform, API calls, external data sources.
2. Infrastructure — servers, cloud resources, network equipment for on-premise deployment.
3. Personnel — the salaries of analysts, rule developers, investigation specialists, ML engineers (in-house and external contractors).
4. Integration and implementation — one-time costs for connection, scenario configuration, migration.
5. Support and development — model updates, rule adaptation, staff training, technical support.

A typical mistake is accounting only for licensing costs while ignoring personnel. According to industry research in the financial services sector, the ratio of technology cost to personnel cost in anti-fraud units is roughly 1:2–1:3. Incomplete cost accounting distorts ROI upward.

Another trap is equating all blocked operations with prevented damage. If the system declined 10,000 transactions worth 100 million rubles, this does not mean that 100 million was prevented: some of the declined operations were legitimate, and some of the fraudulent ones would have been intercepted at subsequent stages (a chargeback refund, a block by the counterparty). A correct calculation counts only confirmed fraud among the blocked operations.

Recommendations for increasing the reliability of ROI:

• Separate benefits into confirmed (backed by data) and estimated (based on modeling), presenting them separately.
• Use a conservative scenario for reporting: an ROI of 300% with high reliability carries more weight than 800% with dubious assumptions.
• Recalculate ROI quarterly — both the attack landscape and the operating costs change.
• Fix a baseline — the level of losses before implementation or when a component is disabled — so that the difference is measurable rather than hypothetical.

The Cost of Errors and the Cost of Anti-Fraud Processes

Every decision of an anti-fraud system has a cost, and every error has a price tag that is rarely limited to the amount of a single transaction. A false block, a manual review, an incident investigation, the operation of the decision-making loop — all of this generates expenses that directly affect ROI. To manage the balance, formulas and methods for calculating the cost at each stage are needed.

The Cost of a False Block in Money and Conversion

A false block (false positive) is a rejection of a legitimate user erroneously classified as a fraudster. At first glance the loss is pinpoint: one operation that did not take place. In practice the consequences are much broader, and in a number of industries the losses from false positives exceed the direct losses from fraud.

Direct financial losses. The minimum unit of damage is the amount of the blocked transaction. When even 1–2% of legitimate operations are declined at high volume, the total amount for a month runs into the millions. According to research by the Aite-Novarica Group, the volume of erroneously declined transactions in global e-commerce is estimated at hundreds of billions of dollars annually — a figure that many times exceeds the direct losses from confirmed fraud.

Conversion losses. A blocked operation is not only lost revenue here and now. A user who received a rejection at onboarding is highly likely to leave for another service. Studies show that up to 40% of users abandon registration in financial services due to excessive friction at the verification stage. Each such departure is a lost Customer Lifetime Value (CLV), wasted CAC, and a negative impact on the sales funnel. The formula for the conversion cost of an FP:

Cost_FP = N_FP × (average transaction value + CLV × P_churn + CAC)

where N_FP is the number of false blocks for the period, and P_churn is the probability of churn after a rejection. The P_churn coefficient depends on the industry, channel, and scenario: for a repeat card block during payment — 20–30%, for a rejection during initial onboarding — significantly higher.

Reputational losses. According to PwC data, a third of consumers stop interacting with a brand after a single bad experience. In digital channels, where switching to an alternative takes seconds, the reputational effect of false positives is felt quickly, although it is hard to monetize directly.

The cost of an FP varies substantially across breakdowns: channel (online, mobile, offline), operation type (payment, onboarding, withdrawal), customer segment (new, VIP, corporate). By industry estimates, a large bank spends $500–1,500 investigating a single alert, and if 90–95% of alerts are false, the total costs of processing FPs reach tens of millions a year. When configuring anti-fraud thresholds, the cost of a false block must be counted on par with the cost of missed fraud, and the operating point optimized taking both values into account.

The Cost of Manual Review and Investigation

A significant share of events ends up in an intermediate zone — manual review. This is a manageable process, but an expensive one, and its cost depends on the volume of alerts, the qualifications of the analysts, and the tools.

Cost structure. The main components: the salaries of analysts and fraud operators, the cost of tools (case management, the review interface, access to external databases and data enrichment), and the Average Handling Time (AHT) — the average time to process a single alert. According to industry data, AHT in an anti-fraud unit is on the order of 30 minutes. The cost of a single manual review in the financial sector ranges from $15 to $50 and higher depending on the complexity of the case and the region.

Formula:

Cost_review = N_alerts × AHT × hourly_rate + tools_cost + overhead

where N_alerts is the number of alerts sent to manual review, AHT is the average processing time in hours, hourly_rate is the fully loaded cost of an analyst’s hour (with taxes, benefits, management), tools_cost is the amortized cost of tools, and overhead is overhead costs (training, turnover, quality control).

The scaling problem. Manual review does not scale linearly. When the flow grows 2–3 times, maintaining the SLA requires proportionally increasing the staff, but the market for fraud analysts is limited, and hiring and onboarding take 2–4 months. A queue forms, response time grows, and along with it the window of exposure in which a fraudster can complete an operation before detection.

The impact of the false positive rate. If the share of false alerts is 90%, then 9 out of 10 reviews are wasted work. Reducing the FP rate by even 10 percentage points at 50,000 alerts a month cuts the load by 5,000 cases, which is equivalent to saving 2,500 person-hours. This financial effect must be factored in when assessing the ROI of any improvement to the model or rules.

Key operational metrics of manual review: AHT by case type, analyst utilization (the share of working time on processing vs. waiting), backlog (queue length), SLA compliance (the share of cases closed on time), the conversion rate from review to confirmed fraud (routing accuracy). The last indicator is especially important: if fewer than 5–10% of manual reviews end in fraud confirmation, a revision of the routing rules or thresholds is needed.

The Cost of an Incident and Recovery Actions

The fraud cost multiplier. According to the LexisNexis True Cost of Fraud study (2025), for every dollar of direct losses merchants spend an average of $4.60 in total costs — a 32% increase since 2022. For financial organizations the multiplier is $4.23–4.76 depending on the segment.

Components of the incident cost:

• Investigation: identifying the attack vector, establishing the scale of the damage, gathering the evidence base. Takes from a few hours (a typical refund) to weeks (a scheme with a network of front accounts). The cost can reach hundreds of thousands of rubles, and significantly more when external forensic specialists are engaged.
• Compensations: refunding the victim, chargeback costs, payment-system fines when the acceptable fraud level is exceeded.
• Recovery: blocking compromised accounts, reissuing cards or tokens, updating rules and models, patching vulnerabilities. For large-scale incidents — a security audit, revision of processes, staff retraining.
• Regulatory costs: filing suspicious activity reports (SAR), interacting with the regulator, potential fines for violating AML/CFT requirements.
• Reputational damage: a decline in customer trust, churn, increased acquisition costs. According to industry surveys, about 79% of financial organizations record a negative impact of fraud incidents on customer trust.

The formula for the total cost of an incident:

Cost_incident = direct_loss + investigation + compensation + recovery + regulatory + reputation_impact

For each category of fraud incident (by attack type, channel, product), it is useful to calculate a standard cost card with ranges for each component. This makes it possible to model the economic effect of a change in detection rate: knowing that the average incident costs X rubles, it is easy to estimate how much a 1 percentage point increase in detection completeness will save.

The Cost of a Single Anti-Fraud Decision by Processing Stage

How much does a single anti-fraud system decision cost from the moment an event arrives to the issuance of a verdict?

Processing stages and their cost components:

1. Intake and enrichment — receiving the event, normalization, calls to external sources (device fingerprint, IP reputation, database checks). The cost is formed from computing resources and the price of requests to external APIs (from tenths of a cent to several cents per request).
2. Scoring and rules — running through ML models and a rule set, computing the risk score. The cost is determined by infrastructure and the amortization of model development costs. With a sound architecture, the cost of inference is fractions of a millisecond of CPU time.
3. Decision routing — automatic approve/decline or escalation to manual review. The cost of routing is minimal, but the price of an error is maximal: a case incorrectly sent to review means excess costs, one incorrectly let through means a potential incident.
4. Manual review (if applicable) — the cost is calculated above.
5. Feedback and updating — recording the outcome, retraining models, adjusting rules. The costs of this stage are often underaccounted, although it is precisely this stage that determines the system’s adaptability.

Formula:

Cost_decision = (infra_cost + enrichment_cost + model_amort) / N_auto + P_manual × Cost_review

where N_auto is the number of automatically processed events, P_manual is the share of events in manual review, and Cost_review is the cost of a single manual review.

The key optimization lever. The largest contribution to the cost comes from manual review. If P_manual = 10% and Cost_review = $25, the contribution of the manual stage is $2.50 per event on average, which is 10–50 times the cost of automatic processing. Any reduction in the share of manual escalations while maintaining detection quality yields a nonlinear economic effect.

For the calculation, it is necessary to keep records of the processing time at each stage of the pipeline (latency breakdown), record the cost of external calls (enrichment, checks), and allocate infrastructure and personnel costs in proportion to the load. The result is the unit economics of anti-fraud, making it possible to compare versions of the system, assess the effect of a new layer of checks, and justify investments in automation.

When the cost of a false block, a manual review, an incident, and each decision is expressed in rubles or as a percentage of turnover, the dialogue with management stops being a conversation about abstract precision and recall — and becomes a conversation about economics.

Monitoring and Proving Effectiveness in Production Operation

A system that showed excellent results in a pilot is capable of degrading unnoticed: latency grows, a manual review queue accumulates, the model stops catching new attack schemes. To detect a problem in time and prove that anti-fraud really brings value, a separate layer of operational metrics is needed — observing the «health» of the decision-making loop in real time.

Metrics of Latency and Availability of Decision-Making Loops

An anti-fraud decision works on the critical path of a transaction: until the system returns a verdict, the payment, registration, or loan issuance is suspended. Response speed and availability directly affect conversion.

Latency is measured by percentile distribution, not by the average. The median (p50) shows the typical user experience, but it is precisely the «tail» — p95, p99, p99.9 — that exposes architectural bottlenecks. A system with an average latency of 80 ms and p99 = 4 s means that every hundredth request slows down 50-fold. At 100,000 events per hour, such a «tail» affects 1,000 transactions and generates tangible conversion losses.

Target thresholds (SLOs) are fixed separately for each stage:

• end-to-end latency — from the arrival of the request to the return of the final decision;
• inference latency — the work of the ML model or rules engine excluding network delays;
• enrichment latency — calls to external data sources (scoring bureaus, sanctions lists, device fingerprint providers).

If enrichment takes 70% of the total time, optimizing the model will not yield a significant effect. A breakdown of latency by stage is a prioritization tool.

Availability of the loop is defined as the share of successfully processed requests over a period. For critical anti-fraud services, the benchmark is 99.95% and above (~22 minutes of total downtime per month). It is important to track not only full unavailability (HTTP 5xx, timeouts) but also «silent» failures: situations where the system formally responds but returns a decision based on simplified logic (fallback), skipping some checks. The share of fallback decisions is a separate metric with its own alerting threshold.

The Load on Manual Review and the Quality of Queue Operation

Some transactions end up in manual review — and it is precisely here that an operational «bottleneck» arises that is capable of devaluing even a well-trained model.

Key queue metrics:

• The volume of incoming cases per period. Growth with a stable transaction flow signals a decline in the model’s «confidence», a shift in thresholds, or the emergence of a new type of attack.
• Backlog (the size of the unprocessed queue). Steady growth means that the team’s throughput is below the incoming flow.
• Dwell time (the time from entering the queue to a decision). Reflects the real delay for the customer. The SLA is set by operation type: for real-time payments — minutes, for onboarding — hours.
• The share of cases processed within the SLA. A drop below the target level (for example, 90%) requires an immediate response.
• Hit rate — the share of cases confirmed as fraud. A low hit rate (below 30–40%) means that analysts spend most of their time on false positives — a direct signal to calibrate the model or the routing rules.
• Consistency of analysts’ decisions. When there are several operators, it is useful to periodically measure inter-rater agreement (Cohen’s kappa coefficient) on overlapping cases. Low agreement indicates vague criteria or insufficient training and affects the quality of the labeling that ends up in the model’s training set.

Controlling Data Drift and Signal Quality as Attacks Adapt

An ML model is trained on historical data and captures statistical regularities relevant at the moment of training. Over time the regularities change, and the model degrades. In anti-fraud the process is aggravated by the fact that the adversary deliberately adapts: having discovered blocking patterns, fraudsters change tactics to look closer to a legitimate user.

Two types of drift:

• Data drift — a change in the distributions of input features while the relationship between them and the target variable remains unchanged. Example: the average transaction value rose due to inflation, but the size of the transaction by itself still does not determine fraud.
• Concept drift — a change in the relationship itself between features and the target variable. A typical situation in anti-fraud: fraudsters switch to new scenarios, and features previously characteristic of fraud begin to occur among legitimate users.

Drift control is built in three layers:

1. Monitoring the distributions of input features. Statistical tests (Kolmogorov–Smirnov for continuous, chi-square for categorical) and divergence metrics (PSI — Population Stability Index) compare the current distribution with the baseline fixed at training. A PSI > 0.2 is usually interpreted as a significant shift. This layer detects data drift but does not guarantee the detection of concept drift.
2. Monitoring model outputs. Tracking the distribution of scores and the shares of decisions (approve / decline / review). If the model has sharply started sending transactions to review more or less often with a stable input flow — an indirect indicator of a shift in the regularities. The signal is available before the true labels are obtained.
3. Monitoring business metrics with a delay. As true labels arrive, precision, recall, and other metrics are calculated on rolling windows. A drop in recall with stable precision is a typical sign of concept drift: the model «does not see» new schemes but continues to correctly identify old ones.

For each layer, alert thresholds and a response procedure are set. Minor data drift with stable business metrics may not require retraining — it is enough to record the fact and observe. Significant concept drift, confirmed by a drop in recall, requires accelerated updating of the model or rules.

Controlling the quality of external data sources deserves separate attention. If a device fingerprint provider changed the response format, stopped returning some fields, or increased the share of empty values — this is a drift of the input signal, capable of leading to the same consequences as model drift. Monitoring the fill rate (the share of filled fields) and the distributions of responses from external services is a mandatory element of the operational loop.

How to Prove Incremental Effect Through Control Groups

Metrics of prevented damage are based on an assumption: «if anti-fraud had not blocked this transaction, the losses would have been X». It is impossible to verify the assumption directly — a blocked transaction does not go through, and its true outcome is unknown. For rigorous justification, a control group is needed.

Incremental effect (incremental lift) is the difference in the target indicator between the group to which the anti-fraud loop is applied and the control group where the loop is weakened or disabled. Only through this difference can the system’s contribution be separated from natural fluctuations in the fraud level.

Experiment methodology:

1. The transaction flow is randomly split into a test and a control group. The test group goes through the full set of rules and models, the control group through a simplified loop (basic rules without ML scoring) or a loop with lowered thresholds.
2. Both groups are observed for a period sufficient to accumulate a statistically significant sample and obtain true labels.
3. Fraud rate, fraud loss, the share of false blocks, and conversion are compared. The difference is the incremental contribution of the loop being tested.

The main limitation is the cost of the experiment: weakening anti-fraud for the control group means deliberately allowing elevated fraud on part of the traffic. Therefore:

• The control group is minimal for statistical significance, usually 5–10% of traffic. The size is determined by power analysis: at α = 0.05, a power of 0.80, and an expected effect size, the minimum sample size is calculated.
• The experiment is conducted on a limited segment (low-amount transactions or a channel with a historically low fraud level).
• The period is fixed in advance and not extended after interim results — this rules out the multiple-comparisons error.

The result is expressed concretely: «ML scoring reduces the fraud rate by Δ percentage points while increasing the share of false blocks by Y percentage points». Such proof carries more weight than a calculation of prevented damage based on model assumptions, and it is precisely this that stakeholders expect — from the CFO to the regulator.

How to Compare Versions of Rules and Models on Live Traffic

Rules are updated, models are retrained, new data sources are connected. Each change requires an objective assessment: did it get better, worse, or is the effect insignificant?

Two main approaches:

Shadow mode (shadow testing). The new version receives the same data flow as the operating loop (champion), but its decisions are not applied to transactions. Both sets of predictions are logged, and after accumulating data, the metrics are compared. The advantage is zero risk to the business. The limitation is that metrics dependent on a real action (conversion after a block) cannot be assessed, since the challenger model’s decision is not executed.

A/B testing (champion / challenger). Traffic is randomly split: some is processed by the operating loop, some by the new one. Both issue real decisions, which makes it possible to measure the full set of metrics. The condition is clean randomization: splitting by the hash of the transaction or session identifier rules out systematic differences between the groups.

The choice of approach is determined by the level of confidence and the acceptable risk:

• A radically new model first undergoes shadow testing: latency, score stability, the distribution of decisions, and consistency with the operating model on cases with a known label are checked.
• After a successful shadow stage — an A/B test with a limited share of traffic (5–15%). Fraud rate, false positive rate, and other business metrics are observed.
• With statistically significant improvements, the share of the challenger model is gradually increased to 100%.

Methodological nuances:

• Statistical significance. The difference is confirmed by a test (a z-test for proportions, bootstrap for complex metrics). Without this there is a risk of mistaking a fluctuation for an improvement.
• Multiple metrics. Anti-fraud is assessed by a combination: precision, recall, FPR, latency, manual review volume, fraud loss. An improvement in recall with a significant rise in FPR is not an improvement but a redistribution of errors. The decision to switch is made according to a predefined set of criteria with thresholds for acceptable changes.
• Observation period. Sufficient to include cycles of business activity (weekdays and weekends, the beginning and end of the month) and to obtain true labels for at least some transactions. A typical minimum is two to four weeks.
• Versioning. Each version is recorded with a link to the thresholds, feature configuration, and the time range of its operation. Without versioning it is impossible to relate a change in metrics to a specific update — neither to reproduce the result nor to roll back in the event of a regression.