Explaining decisions made with AI — a statutory requirement (2/2)

More details…

This is the second part of my posts reviewing the ICO/Turing Institute guidance on AI explainability of algorithmic decisions. This post adds further detail to the overview given in part 1 and doesn’t really make that much sense on it’s own…

Explaining decisions made with AI — a statutory requirement (1/2)

Legal basis

While we are only more recently seeing explicit legal considerations regarding algorithms, existing regulations and legal provisions, even if they do not mention AI specifically (often deliberately kept solution neutral) do impact to algorithmic / AI based decision making: GDPR / GDPR UK is possibly being the most general and relevant as it covers the right to an explanation and certain right sand obligations in regards to data (especially ‘big’ data) processing, the DPA in regards to avoiding discrimination (as part of decision making), administrative law with its right for judicial review and explanation, more recently IP and copyright law in regards to training-data and ownership of model outputs (especially in case of generative models) but also a raft of other laws and regulations including, but not limited to

• e-Privacy legislation
• Law Enforcement Directive
• Consumer Rights legislation
• Financial Services legislation
• Competition law
• Human Rights Act
• Legislation about health and social care
• Regulation around advertising and marketing
• Legislation about school admissions procedures

The EU, as other countries and bodies, are also drafting AI specific advisory, best practices and regulatory frameworks, e.g.

Home

For the purpose of these posts I want to highlight some aspects from the ICO/Turing Institute guidance on AI explainability pertaining to explainability:

• GDPR UK is the main legislation in the United Kingdom that explicitly states a requirement to provide an explanation to an individual.\
• The right for individuals to be informed about the use and (certain) details of algorithmic decision making, as well as the right of access to PII being used, and that data being rectified.
• The right to object to use of PII (under certain circumstances). Profiling for direct marking can be absolutely objected to.
• Article 22 of the GDPR gives individuals the right not to be subject to a solely automated decision producing legal or similarly significant effects. There are some exceptions to this and in those cases it requires organisations to adopt suitable measures to safeguard individuals, including the right to obtain human intervention; allow individuals to express their view and contest the decision.
• A DPIA is always required for any systematic and extensive profiling or other automated evaluation of personal data which are used for decisions that produce legal or similarly significant effects on people.
• Explainability becomes a requirement where the system is autonomous and makes decisions without human involvement or creates significant consequences. In such cases the implementer must explain the logic, provide the right to access human intervention and explain potential impact.
• Requirements of fairness, transparency and honesty, accountability, minimisation of data captured and accuracy can be derived from GDPR.
• The DPA also allows individuals to obtain human intervention, express their point of view, and obtain an explanation of the decision and challenge it.
• Autonomous weapons systems are not covered by GDPR, but the DPA provides certain rights balanced with safeguarding national security in regards to access to system use and logic. See https://www.ceps.eu/wp-content/uploads/2021/04/AI-Presentation-CEPS-Webinar-L.-Sioli-23.4.21.pdf
• Administrative law and the Equality Act 2010 prohibits the discrimination based on personal characteristics age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion and belief, sex, sexual orientation. If AI is used in decision making you need to be able to demonstrate that the decisions do not causes the decision recipient to be treated worse than someone else because of one of these protected characteristics; or results in a worse impact on someone with a protected characteristic than someone without one.

As a result the comment state that an explainability ‘policy’ (in addition to various other required documentation) should be drafted, outlining the rationale behind, goals and process of achieving explainability (and that this should be done prior! to starting system implementation to guide design, build and operation).

Why given explanation

It is important to recognise that there are number of reasons to provide explanation

• to support those designing, building and maintaining the system
• to enable quality assurance, compliance and oversight
• to inform ‘implementers’ i.e. users of the decision in making ‘good’ decisions and explain the decisions to those being impacted by the decision. This impacts how we design functionality, say of a diagnostics systems.
• to allow those being impacted by the decisions to be informed, object and challenge. This impacts how we design products, product outputs such as repots and certificates, but also the services within which the decision making happens.

As such, explainability is important ad- and post-hoc.

Example for explains in practice

Kings College has provided a number of sample applications to demonstrate good explainability and specifically a provenance based appraoch to data governance (see below):

ProvExplain

What goes into an explanation

I personally find the structure the ICO/Turing document provides confusing and convoluted, however I believe the general guidance to be useful.

Aspects to cover (based on context):

• Rationale: explaining how the decision was derived, what constraints do apply
• Responsibility: who does what and when, this is all about ownership
• Data: what data is used for training and validating the system, how is it used, how was validity of the data ensured and what data is used during operation to derive a decision
• Fairness: how is it ensured that the decisions are unbiased and fair
• Safety and performance: how is the proper performance of the system and process ensured (accuracy, reliability, security)
• Impact: what are impacts of the system output and related decisions and how are detrimental impact on individuals and society avoided.
  Note: while not statutory, organisations should also consider what detrimental impact (reputational, legal, operational, other) may result for the organisation and how to mitigate these. But of course, that’s ‘at the organisational’s pleasure’.

The document suggests that when providing the above explanations, they should be viewed from two perspective:

• Process-based: what does all the above mean for the process via which the system is designed / implemented and operated and how the output are generated
• Output-based: what does all the above mean for the output / decisions derived?

Give explanations in context

The document identifies five contextual factors that have an effect on the purpose an individual wishes to use an explanation for, and consequently how we should deliver explanations, and in turn will inform which of the explanation types above you focus on:

Domain
The domain of use or use-case (law enforcement, medical, ecommerce, marketing) will provide key context and what explanations to provide and at what level of detail (regarding the examples above: fairness vs safety/performance vs data privacy).
Where humans are involved — or not, you need to explain how this changes the dynamic and how the system output informs the human.

Impact
Impact such as life/death/health vs liberty/legal vs access to employment, education or political messages vs trivial impacts such as being served ads or assigned a seat on a plane will strongly influence what information at what level of detail and with what supplementary guidance is to be presented.

Data
Data considerations must consider the whole lifecycle from training, testing to operation. Consider social, behavioural and biophysical data as those datapoints that can be changed by the ‘owner’ will lead to an expectation of more explanations. For instance where biophysical data cannot be changed and is used for medical decision the decision itself and explaining what it means may be more important than discussing the data. The opposite may be true when discussing a legal judgement or a loan application.

Urgency
The timeframe between decision and need to act is highly relevant in terms of explanation. Where decision need to be made under pressure or timely (e.g. triaging emergency calls), understanding consequences is likely to be more important than justification or detail.

Audience
Who is the explanation given to? The same application is likely to have a variety of stakeholders with differing use-cases and informational needs, but also capabilities (expertise, experience, …)

Principles

The ICO/Turing document suggests the following principles

• be transparent
• be accountable
• consider the context you are operating in
• reflect on the impact of your AI system on the individuals affected, as well as wider society.

The first two relate to / ensure GDPR compliance. The principle of being transparent is an extension of the transparency aspect of principle in the GDPR (lawfulness, fairness and transparency). Explanations should explain why, when and who, and should be meaningful, appropriate, contextual and delivered at the right time.

Accountability is all about demonstrating compliance and implementing best practices and compliance by design and default. This must span the full lifecycle.

Choices should be deliberate and justified, evidenced. A point for contact should be available.

I think we can add a further principle, that of ‘clarity’ following article 12 of the GDPR requires you to provide information to the data subject in “concise, transparent, intelligible and easily accessible form, using clear and plain language…”.

Responsibly building algorithmic decision making systems

When responsibly building systems that apply algorithmic decision making we should consider the impact on individuals and society (note that impact to socity always mean impact to individuals but not vice versa):

Individual wellbeing

“Think about how to build and implement your AI system in a way that: fosters the physical, emotional and mental integrity of affected individuals; ensures their abilities to make free and informed decisions about their own lives; safeguards their autonomy and their power to express themselves; supports their abilities to flourish, to fully develop themselves, and to pursue their interests according to their own freely determined life plans; preserves their ability to maintain a private life independent from the transformative effects of technology; and secures their capacities to make well-considered, positive and independent contributions to their social groups and to the shared life of the community, more generally.

Wellbeing of wider society

Think about how to build and implement your AI system in a way that: safeguards meaningful human connection and social cohesion; prioritises diversity, participation and inclusion; encourages all voices to be heard and all opinions to be weighed seriously and sincerely; treats all individuals equally and protects social equity; uses AI technologies as an essential support for the protection of fair and equal treatment under the law; utilises innovation to empower and to advance the interests and well-being of as many individuals as possible; and anticipates the wider impacts of the AI technologies you are developing by thinking about their ramifications for others around the globe, for the biosphere as a whole and for future generations. “ p.44

Model types

The Annexe 2 of the ICO / Turing document provides an overview of various models, their uses and their level of explainability.

Explaining back box models

Annexe 3 of the ICO / Turing document provides an overview of supplementary models to aid explainability of complex / black-box models.

Process

The ICO / Turing document provides a slightly convoluated process to cover explainability as part of the systems delivery lifecycle. I personally feel that creating this as process was a mistake as it needs merging into the wider development approach and a focus on artefacts would have been more useful (also, a this point the ICO / Turing document becomes highly redundant and repetitive). Anyways, the general points of the the ‘process’ are generally valuable and highly the potentially enormous scale of the task to achieve explainability we mustn’t under-estimate.

While it is stressed that this must be started prior to system development, I think it is important to approach this, as we would with any reasonable system development in lean an iterative fashion, i.e. continuously and evolutionary as otherwise we will end up with Big Design Up Front and all the issues that come with it…

This should be worked in, as any other compliance approach.

The main point being, that we work toward ‘explainability by design’.

The suggested ‘tasks are:

1. Select priority explanations by considering the domain, use case and impact on the individual

During this (early) stage we will assess the overall direction and goals for explainability, i.e. how we will approach explainability; what will explain to whom and why. As discussed above this is all about context and impact.

We should expect the eed for multiple explanations for various stakeholders, and in fact, even multiple layers of explanations for the same stakeholders covering different situations or use-cases.

Remember to cover the various explanations types and apply the relevant principles discussed above.

2. Collect and pre-process your data in an explanation-aware manner

This step isn’t really any different to any other data-governance activity. Except, arguably, that we need to be more diligent due to the impact of our outputs and, if we use data to train neutral networks, the impossibility of high impact (retraining) if we had to remove individual’s data at a later stage.

It is recommended to use an auditable data model that allows to trace provenance of data and thus aid transparency, quality assurance and governance (e.g. PROV data model [PROV-DM] by W3C.
Examples: https://explain.openprovenance.org/loan/)

This in turn will aid explainability but also other GDPR related requirements.

Of course this step also needs to consider general data science related aspects such as the appropriateness, relevance, bias, availability etc of the data we use, as well as version control of our models, etc.

3. Build your system to ensure you are able to extract relevant information for a range of explanation types

The point of this stage is to make design decision so that we

• Choose the most appropriate models for the task
• Design / build explainability into our system

The goal is to select an appropriately explainable model. This may simply mean choosing intrinsically explainable model see above, or add supplementary models where our models are complex or ‘black box’ (see above).

“The ICO/Turing document suggests to consider whether:

• there are costs and benefits to replacing your current system with a newer and potentially less explainable AI model;
• the data you use requires a more or less explainable system;
• your use case and domain context encourage choosing an inherently interpretable system;
• your processing needs lead you to select a ‘black box’ model; and
• the supplementary interpretability tools that help you to explain a ‘black box’ model (if chosen) are appropriate in your context. “ p.64

4. Translate the rationale of your system’s results into useable and easily understandable reasons

In this stage we decide how the system’s statistic results are translated into actionable outputs for human use, i.e. how we integrate the system outputs into the wider decision making process.

A key consideration here is to mindfully balance the line between ‘causation which does not imply correlation.’

5. Prepare implementers to deploy your AI system

‘Implementers’ in this case means users of the system output (not delivery teams). So when human decision-makers are involved they must be appropriately trained and prepared to use your model’s results responsibly and fairly.

“Training should include conveying basic knowledge about the nature of machine learning, and about the limitations of AI and automated decision-support technologies. It should also encourage users (the implementers) to view the benefits and risks of deploying these systems in terms of their role in helping humans to come to judgements, rather than replacing that judgement. If the system is wholly automated and provides a result directly to the decision recipient, it should be set up to provide understandable explanations to them. “ p.51

We may also want to address decision-automation as well as automation distrust and other biases, and educate implementers as to the strengths and weaknesses of the system, and the use of statistical results for evidence based reasoning.

6. Consider how to build and present your explanation

Considering that a the models outputs will, in some shape or form be statistical, we need to have a clear way to convey the repeated intricacies and constraints to users and stakeholders.

We need to explain outputs in context, but also how the various inputs related to outputs on how changes inputs may affect outputs. We will also need to discuss global vs local aspects of the system.

At this stage it is worth considering overall explanations of the system for various documentation purposes, presentation of outputs to operators and decision ‘recipients’.

Dependent on process the explanation will be provided by a human, a system or both, which will affect what explanation capabilities we need to provide.

Roles & Responsibilities and Documentation

The document also includes a chapter on documentation and one on roles & responsibilities which may be worth glancing through but don’t add much to what else is covered in the document.