Sector
Insurers have always modelled. The question is not whether you can model, it is whether you can defend a model more complex than the one it replaces, in front of the actuarial function, internal control, a supervisor and a policyholder.
Your data does not describe risk, it describes your process. An unreported claim does not exist in the database. A file reclassified by hand carries the fingerprint of the handler who reclassified it. A model trained carelessly on that history learns your past decisions, not the underlying risk, and faithfully reproduces the biases of the very system it is meant to replace.
The problem is sharpest in fraud. Confirmed fraud is rare, and it is only known where somebody looked. A model trained on detected fraud learns current targeting and confirms it. Without a randomly drawn control sample, independent of the targeting, there is no honest way to estimate a base rate, and therefore no way to claim you are doing better than before.
The duty to justify is structural, and it is poorly understood by people arriving from other sectors. At equal performance, an explainable model and an opaque one are not worth the same, because the second costs more to defend before a supervisor, a policyholder or a court. So the right question is not "which model performs best" but "what performance gap justifies paying that defence cost". That gap is measured, not assumed.
Finally, portfolios drift. Random cross-validation on history produces numbers that are flattering and wrong. The only evaluation that predicts real behaviour is one run on a period after the training window, on the portfolio as it is rather than as it was.
Four frameworks overlap, and they govern different objects: the AI system, the decision, the model, and the provider.
AI systems intended for risk assessment and pricing in relation to natural persons in life and health insurance are classified as high risk under Annex III. That brings a risk management system, training data governance, technical documentation, logging, transparency, human oversight and a demonstrated level of accuracy and robustness. Worth noting: other lines of business do not fall under this entry, which exempts you from nothing else, starting with the GDPR.
A person has the right not to be subject to a decision based solely on automated processing which produces legal effects concerning them or similarly significantly affects them. Declining cover, cancelling a policy or applying a significant loading fall within scope. In design terms this means planning for human intervention, for the person to express their point of view and contest the outcome, and therefore for a usable trace of what drove the system’s proposal.
The system of governance, the actuarial function and the risk management function cover the models that feed pricing, reserving and risk assessment. A machine learning model entering that chain enters the validation perimeter: documentation, independent review, ongoing monitoring, proportionate to its place in the chain. A better model built without the actuarial function is not an advanced model, it is a model that will not be deployed.
DORA governs digital operational resilience in the financial sector: managing ICT third-party risk, a register of contractual arrangements, testing, and reporting major incidents. The architectural consequence is immediate: an inference service called at a third party is an ICT third-party provider, with the contractual and monitoring requirements that follow. Running the model in your own environment is not only a sovereignty choice, it is often the shorter path.
The ACPR, the French prudential supervisor, has published work on the governance of artificial intelligence algorithms in the financial sector, particularly on explainability, model validation and internal control. It echoes the documentation demands found elsewhere. It is worth reading before choosing a model family rather than after: it describes fairly precisely what will be asked for during a review, and therefore what needs producing during the project rather than reconstructing afterwards.
Success criterion
Share of files handled with no human rework at the chosen threshold, and extraction error rate measured separately on pricing-relevant fields. An error on a date of birth and an error in an address line are not worth the same: aggregating them would produce a reassuring, useless number.
Success criterion
Precision at the top of the queue at constant review capacity, estimated on the random sample rather than on files the previous system already targeted. With selection rates tracked by segment, to check the gain does not come from concentrating on one population.
Success criterion
Recall on the high-stakes categories and precision on the rest, at an alert rate the team can absorb, measured on a sample reviewed by handlers. Complemented by first-response time before and after, on the same perimeter and seasonality, at comparable volume.
Success criterion
Discriminatory power and calibration on a period after the training window, plus stability of contributions over time. And an explicit proxy test: if removing a prohibited variable changes nothing because a proxy stands in for it, the model is not compliant, whatever it scores.
These failures look the same from one insurer to the next. They all come down to an evaluation that does not measure what people think it measures.
That is the real question, and it comes before performance. At equal performance, an explainable model and an opaque one do not have the same value, because the second costs more to defend in front of a supervisor, a policyholder or a judge. We build explainable by default, and we keep an opaque model only if its performance gap is measured and judged large enough to pay that defence cost. The ACPR’s work on the governance of AI algorithms (the French financial supervisor) describes fairly precisely what will be asked for: better to read it before choosing a family of models than after.
Yes, and it has to be treated as such. An inference service called from a third party is a third-party ICT service provider under DORA, with the contractual, register and monitoring requirements that come with it. It is one of the reasons we favour inference on your own infrastructure: running the model in-house is not only a sovereignty choice, it is often the shortest path to compliance.
With a control sample drawn at random, independent of the current targeting and maintained over time. Fraud is only known where it was looked for: a model trained on detected fraud learns the existing targeting and confirms it. Without a random sample, it is impossible to honestly estimate a base rate, so impossible to claim you do better than before. We measure precision at the top of the queue on that sample, not on the cases the old system already targeted.
Yes, and we involve them from scoping, not at the end. A learning model that feeds pricing, reserving or risk assessment falls within the validation perimeter under Solvency II: documentation, independent review, monitoring over time. A better-performing model built without the actuarial function is not a model ahead of its time, it is a model that will not be deployed. We produce the validation material, we do not sign off the validation in your place.
An hour is enough to tell whether your subject stands up. If it does not, we will tell you then, not after three months of project.
