Explaining decisions made with AI — a statutory requirement (1/2)
A guide for non-technical stakeholders
As part of my work on an ML/AI related innovation project, with strong ethical implications, I’m reading a number of related documents. Here is the first of a series of articles summarising my learnings.
This article is a Tl;Dr summary of “Explaining decision made with AI” by CIO and Alan Turing Institute.
But why this summary: first, the original document is 100+ pages, the second that while there is interesting information, the document suffers what so many of such documents suffer from, namely that it is a badly structured, convoluted, redundant mess that isn’t quite sure whether it wants to be an overview, formal guidance, template or example. As thus it is overly wordy and in parts quite painful to read while at the first read possibly leaving more confusion than clarity. (Whoever writes such documents, please get human centric editors, will you?)
What is AI decision making?
‘Artificial Intelligence’ in this context is defined as algorithm-based technologies that previously would have required (or now go beyond) human cognitive capability. So one might chose to better replace AI in this context with algorithm, as the system we are talking about would include classic regression (which most us wouldn’t call ‘AI’), but also reaches into machine and deep learning algorithms where we definitely start to touch weak (if not quite yet strong/general) AI.
Also, what this document is all about is the use of algorithms to aid decision making in various context, be this by providing information based on which a human then makes a decision or fully automated decision making. As should be clear, requirements — statutory or voluntary — will differ very much on the context (what the output is used for) and whether or to what degree there is human involvement.
Such algorithmic outputs fall into three categories
- a prediction (eg this is likely to be cancer)
- a recommendation (eg this is a product you will like)
- a classification (eg this is a cat picture).
What is explainability?
Explainability is the ability to ‘rationalise’ and ‘communicate how certain input variables (we call them ‘features’) lead to outputs, why these outputs are valuable for the decision making process at hand, and what the constraints, impact and risks around these are (e.g. how reliable, accurate, sensitive, safe etc).
“In making decisions and performing tasks that have previously required the thinking and reasoning of responsible humans, AI systems are increasingly serving as trustees of human decision-making. However, individuals cannot hold these systems directly accountable for the consequences of their outcomes and behaviours.
The value of reflecting on the impacts of your AI system helps you explain to individuals affected by its decisions that the use of AI will not harm or impair their wellbeing. “ p. 44
Why explainability matters
As algorithmic decision making becomes more powerful we
- will see (in fact are already seeing) more and more responsibility handed over to algorithms (COMPASS in the US (a bad example) and HART in the UK (a better approach in terms of transparency and impacxt, but still controversial) are recidivism models used by law enforcement to predict risk of re-offending and consequently adjust sentencing or support programms (respectively).
- want highest degrees of accountability and governance due to the risks on organisations, individuals and society
- have to follow statuary and should follow voluntary best practices because compliance and ethics are valuable (beyond not getting sued)
Explainability matters because as humans (or systems) make decision it is important that we understand why.
For quality control (are the decision reliable, consistent, fair, accurate?), to provide the decision maker with context so they can make an informed decision (not all decision are clear cut, and require understanding of risks, impact, reliability, sensitivity, context), and for the purpose of providing a rationale to those who are at the receiving end in case they want to challenge or ‘just know’.
The vital important of explainability becomes clear when we consider algorithms used in HR to aid HR decision, in the criminal justice system to determine risk of re-offending and related actions the criminal justice system takes, facial recognition as part of border control, or medical diagnostics in form of analysing MRI scans for cancer diagnostics. We can easily see why those making decisions based on algorithmic outputs but also those impacted by these decision have a vested interest, an obligation in many cases, to understand how a certain system output (recommendation or decision) was derived at, how it is used and why, and what the constraints of the system are.
Obviously, context matters: consider vision systems at a security checkpoint vs. an OCR system scanning a customer feedback form. Very different ‘beasts’ with very different ‘needs’ for explainability and oversight.
Explainability and related training is also important because due to cognitive biases humans are not great at intuitively interpreting statistical data, judging risks, or correctly applying insights from general statistical analysis to individuals cases.
I would hazard a guess that explainability becomes less of an issue as technology becomes more advanced and reliable, but for now we are not there, and especially the more modern ‘cooler’ (NL) models we hear about in the press every day are very prone to biases and errors. But it is not just those, far more basic models used in the legal justice system or for loan applications have been criticised for using base (proxy) parameters such as postcode to inform their decisions…
Explainability matters because of impact, and because it enables stakeholders to make better judgements and challenge and manage outcomes.
You’ll want to have more explainability where the decision impacts human lives (say a court order) and less, where it doesn’t (say where you get served product recommendation or allocated a plane seat).
I personally think that explainability, ignoring statutory requirements, is an ethical obligation.
Explainability is a legal requirement, so is the right to obtain intervention
Taking about the former, there are a number of regulatory frameworks that drive towards explainability, the most relevant at this point in time — and I’m sure we’ll see more specifics emerging soon — in the EU (and respectively UK) are GDPR GDPR UK:
Articles 13–14 of the GDPR require that you proactively provide ‘…meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject…’ while article 15 gives individuals a right to obtain this information at any time on request.
Specifically the ICO/Turing write “You should also take into account the transparency requirements of the GDPR, which (at least in cases of solely automated AI decisions) includes: providing meaningful information about the logic, significance and envisaged consequences of the AI decision; the right to object; and the right to obtain human intervention.” p.21
It is worth reflecting on the essence of the above for a bit, and also recognise that this is not so different in spirit to what is required from organisation in regards to data in general, BUT, that it becomes much more complex to implement where, what we do with the data becomes more complex and opaque.
You need to design and build explainability in
The right for explanation, that there should also be no loss of accountability (who can I go to to challenge and take corrective action), the right for intervention, rights for transparency and objection, let alone domain specific regulations on safety and performance (see below) all have far-reaching consequences of how we design, build and operate systems that employ such algorithms.
For this reason we need to design and build explainability (ethics and other best practice aspects) into our systems from day one and ensure continuous compliance.
You need to choose the right model
“Higher-stakes or safety-critical applications will require you to be more thorough in how you consider whether prospective models can appropriately ensure outcomes that are non-discriminatory, safe, and supportive of individual and societal wellbeing.
Low-stakes AI models that are not safety-critical, do not directly impact the lives of people, and do not process potentially sensitive social and demographic data are likely to mean there is less need for you to dedicate extensive resources to developing an optimally performing but highly interpretable system. “ p. 67
“In cases where you are processing social or demographic data you may come across issues of bias and discrimination. Here, you should prioritise selecting an optimally interpretable model, and avoid ‘black box’ systems.
More complex systems may be appropriate in cases where you are processing biological or physical data, only for the purposes of gaining scientific insight (eg predicting protein structures in genomics research), or operational functionality (eg computer vision for vehicle navigation).
However, where the application is high impact or safety-critical, you should weigh the safety and performance (accuracy, security, reliability and robustness) of the AI system heavily in selecting the model. Note, though, that bias and discrimination issues may arise in processing biological and physical data, for example in the representativeness of the datasets these models are trained and tested on.“ p. 65
Why can explainability be difficult?
Looking at a very basic regression where we have a limited number of weighted input variables (features) that, when changed, lead to an equal (linear) shift in the output, e.g income (input) vs ability to pay bak a loan (output) for the purpose of risk analysis by a bank. Ignoring that this model is by far too simplistic, it is clear and easy to explain, in terms of how the model works, what data is being used, sensitivities and errors.
As we add more variables (features), add dependencies between variables, and allow for non linear behaviours or non standard distribution datasets, explainability becomes harder.
More so, when we combine models. Decision trees, where the system goes through a range of if/then decisions to, for instances classify objects are highly explainable. E.g is it plant / animal, if plant, is it vegetable or fruit, if fruit, is it red or other, if red … and so on, ultimately arriving at the classification of cherry. As the tree grows, we add weights to the tree or where we use many trees and average their results (‘random forest’), explainability becomes much harder. The same applies to other such ensemble methods which combine multiple/ different models to generate a single output.
But generally, such ‘feature’ based models, i.e. models where we have specific characteristics that we identify with some assumption on relevance and relationship to the the output are explainable-ish.
It gets very (impossibly?) hard to explain the most modern models such as neutral networks (used amongst other things for facial recognition, or are the basis for humanities most favourite AI play things ‘de jour’: generative AI such as chatGPT or Midjourney). Due to their nature, e.g. 1B+ weighted parameters in complex matrices that ‘somehow’ encapsulate information of how features relate to outputs, these models are explainable in terms of ‘how they generically work’, but much less so in terms of ‘how a specific output’, or how a specific shift in input may affect the output. While these models are perfectly deterministic, they are certainly non linear and behave a bit like chaotic models, in the sense that they are not easily predictable (otherwise we’d use more simplistic models).
Remember, some of these model are trained with controlled labelled (label == the correct classification or rating of the input) data e.g. 10000 pictures of a cats labeled with ‘cat’ and 10000 other pictures labelled as ‘not cat’, or 10000 essays each with a score (label). Training data and labels are given to the model and the internal parameters and functions are tweaked (automatically and or manually) until the model not only classifies / scores the training data perfectly, but has ‘encapsulated’ the essence or key characteristics of a cat picture or a 5star essay so well, that it can correctly judge other similar data. In these cases, it can easily be that there is no obvious rationale to why a certain parameter is as it is or why a certain input is classified as it is.
To make this worse, so called deep learning models are ‘let loose’ on data without such pre-judgement (i.e. labels), with the ‘task’ to find patterns. Obviously this makes explainability even harder.
We’ll see later how we can get a certain level of explainability back into systems that employ such models.
Please note that this post is only about explainability and not other concerns, shortcomings (or strengths and opportunities) of these models.
What makes for a ‘good’ explanation?
Well, it ‘depends’.
The ICO / Turing describe the need for appropriate explanation, dependent on domain (where do we make the decision — medical / judicial / employment / advertising, …), context (an oncologist deciding on cancer treatment, a judge deciding on length of sentencing, a university / government body or HR department deciding on authenticity and skill level of someone speaking English and therefore getting a job or VISA, a retailer seeing a mail shot), situation (law enforcement or medical services triaging an ongoing incident vs corporate leadership making long-term strategic decision), users (data analysts, law enforcement, judges, oncologists, patients).
They also state the need to cover aspects such as the rationale (why), process (what, how and who) in explanations, specifically covering aspects such as what data was used to train the model, what data is used to make specific decisions during use, who makes the decisions and to what purpose, and what the constraints are. All of these, of course, across the full lifecycle from design via build to operation.
The point being, that any stakeholders working on, using or being subject to algorithmic decision making should be given relevant information in the most appropriate way, considering context and situation, so that they can satisfy their needs, be they building a system, ensuring quality/reliability/performance, using the system and making decisions based on system outputs or understanding the output and decision and being able to challenge them.
Teams are advised to define a policy for explainability and how we govern the entire process before! we start building the system.
In terms of providing explainability the document mentions a variety of aspects and provides detailed guidance, but the ones that stood out to me were two aspects:
- It is highly important system outputs are presented in understandable, qualified and actionable way, including relevant supplementary information so that meaningful decisions can be made.
Dependent on audience this can range from a well worded explanation to a massive amount of supplementary statistical data, sensitive analysis or visualisation of input vs output interaction, or, say in the case of cancer diagnostics visual reference data for the radiologist to query and compare. - As mentioned above, certain models have perfect intrinsic explainability, other models less so. Where the latter are concerned we can supplement our model with additional explainable models that are not as accurate but ‘close’ and provide explainability. This can be achieved, for instance by overlaying or reverse engineering a more simplistic regression on top of a much more complex model.
- Provide additional insight. Supplementary models can help demonstrate variable relationships, sensitivities, error ranges, for instance by indicating through simulation of how individual variables contributed to a specific decision or how much certain variables would need to shift to shift the output.
- Have a dedicated model to provide the explanation: This is the one that can lead to a bit of a head-fuck: Assuming you are using a Convoluted Neural Network (CNN) for image recognition, why not overlay a Large Language Model (LLM) that ‘observes’ the CNN and creates a post rationalisation for why that specific image classification was achieved? While this is a promising approach, one can easily see how the interplay between two such models, both trained on the same data but one also trained on the output of the other can lead can lead to great, but also even less reliable outputs: How would we avoid — or even know about — one model’s hallucinations (a model making ‘stuff up’ i.e. providing facts based in it’s internal workings that do not represent reality but are correct as per the models internal ‘state’) not feeding the other model’s hallucinations, so that the model provides a perfectly reasonable sounding, but not true explanation, for a correct, or even incorrect, decision of the first model?
Where we stand
The document I’m summarising and commenting on is very much in line with other similar documents from other organisations, and strongly aligned with statutory regulations such as GDPR / GDPR UK and other regulations, and takes very much a ‘2022-stance’, acknowledging the current state of ‘AI’: We see this specifically in a strong preference for simple, explainable models and human oversight vs. complex, deep-learning models and fully automated solutions.
Their cautious approach is based on risk and ethics assessment, but also on the fact that some of the more simplistic models (e.g. RiskSLIM used in criminal justice) do produce comparably good results to some of the more fashionable but less explainable models. (There is clearly a tendency right now to jump on what’s cool and fashionable, maybe because of fomo).
A cautious and considerate approach, at this point, seems prudent, but I am certain, that we will see a point in the future where we will no longer need or be able to provide human oversight, where, in fact autonomous decision making is much better, and less biased than human decision making; as long as we get governance right and define desirable outcomes appropriately, or at least oversight and a way to interfere and course correct. Which links back to my earlier point about using models to explain models. We can easily see a dystopian self-referential loop here…
Slightly unrelated to this document, but still an important point, ICO / Turing raise the question of whether the team have considered the reason and impact, cost / benefit of using AI (to introduce or replace an existing process or tool)?
Interested in more detail around legal and regulatory concerns or the suggested process? Have a look at part 2 of this post series:
