This series contains general information and is neither formal nor legal advice. Readers and organisations must make their own assessment as to the suitability of referred standards and this material for their specific business needs.
Introduction
The new ISO 42001:2023 framework is an Artificial Intelligence (AI) Management System for your organisation, similar to, for instance, ISO 9001 (Quality Control) and ISO 27001 (Information Security).
In this three part series we will cover background information regarding AI (part 1), matters to consider when looking at developing (part 2), using or offering AI in your organisation, and how ISO 42001 can help you with this in a responsible manner (part 3).
We do this from the perspective of Governance, Risk and Compliance (GRC) as that is our specific area of expertise in this matter, and it is also what ISO 42001:2023 is all about.
10 Risks among the Opportunities
Many countries have done in-depth and practical reviews on the use of AI, both inventorying what is already in use, and looking forward to new applications.
During its review of AI in the prelude to the recently passed EU legislation on the same topic, the European Parliament concluded that AI is likely to touch every aspect of our society, and also add some new uses that haven’t yet been thought of. The article also raised a number of interesting insights and challenges. We picked a few and added some more below:
1. Underuse and Overuse of AI
Underuse of AI could actually be a major risk in terms of lost opportunities, lack of initiative, and low investments. Have you considered that yet?
Overuse can also be problematic. An application that proves to not be useful can result in wasted investment (and lost time!). For instance, trying to use an AI for explaining a complex issue in society is not going to be useful, since it is not suited to that purpose.
2. Accountability
Who is responsible if a bad AI decision results in unrecoverable consequences such as bad injury or death? One example of this is a self-driving car. Should the car owner still be responsible, even though they are no longer the driver? The car manufacturer, the programmers, or perhaps the supplier of the dataset that was used to train the AI?
In any case, in no circumstance can the AI be blamed, that would be silly.
In the streaming series Upload (Amazon Prime), car owners can actually select whether to protect the occupant of the car, or evade the external subject, should the self-driving car’s AI need to make that choice. An interesting take on the classic philosophical exercise called ‘the trolley problem’!
Accountability for manufacturers provides some incentive for providing a good product, but too much regulation could reduce innovation. No matter where you currently stand on this topic, we can all agree that it is an interesting discussion with no easy single answer that covers all situations. This example clearly shows that thought must go into any application all the way from the idea and design phase, throughout architecture and development, and more review before it is launched.
3. Consequences
A bad AI decision can also have other consequences that are detrimental to an individual, group, or society as a whole. Amazon trained an AI on its HR records, in order to automate the preselection stage of its hiring process and possibly remove human bias. It turned out doing the opposite: the AI was biased against women and minorities. Looking back, the reason is obvious – past hiring practices had that bias, so by teaching the AI using the existing corpus of employees, the company reinforced that issue.
4. The Myth That Computer Output Is Always Correct
This is a really big issue. Simply put, computers are as good as their programming, and AI models are as good as the data they were trained on. Mistakes are made, and so unfortunately the output is not always correct. Large scale incidents of this are the Robodebt debacle in Australia, a childcare issue in The Netherlands, and the Post Office scandal in the UK where hundreds of subpostmasters were wrongly prosecuted for theft or fraud while in fact there were bugs in the computer program that showed inconsistencies. People were imprisoned, families ripped apart, many left destitute, and some people died. I’m sorry to introduce such a serious note, but these issues are real and most countries will have at least one such example by now.
Not all these cases are AI related, but the main lesson here is that even a system designed to make decisions requires some human oversight and periodic critical checks. This is not new, but often disregarded either from the start (by those wanting to believe the myth), or in the name of savings during the lifetime of the system. Either way, an organisation cannot blame the AI system, it is not an entity that can bear that accountability or legal responsibility.
The 1983 classic ‘hacker movie’ Wargames explores various aspects of this theme in detail: a computer designed to run military strategy simulations ends up in charge of the nuclear arsenal when, in the name of efficiency, the ‘flawed humans’ previously in the process are taken out of the loop.
5. Verification
Whereas with earlier AI – typically the model would be extracted after the main learning phase to see what rules it derived from its input – this is exceedingly difficult with the modern LLMs: they are just too big. Currently, one of the best approaches is to ask the model critical questions that cover the scenarios and decisions that the system is to be involved in, thus confirming that indeed it is making the correct decisions.
This needs to be done on an ongoing basis as new data is processed. It is the equivalent of what software engineers call a ‘test suite’.
6. Transparency and Privacy
How do you design a system so that it will be trusted by users? Are there safeguards in place that prevent data from being collected that is not needed (under Australian privacy legislation, marketing is not a valid need), prevent different sets of data from being inappropriate combined or correlated, prevent personally identifiable information (PII) from somehow ending up in the dataset of an AI model, prevent data from being shared with other parties, and so on. Increasingly, just saying ‘trust us’ will not suffice.
7. Jailbreaking
As mentioned, it has been shown to not be terribly hard to trip up an AI-driven application. Recently the State Library of Queensland recently introduced Charlie, an AI persona to guide visitors through an area of expertise, World War I. But soon, some creative individuals convinced it to act as Doctor Who and some other anachronistic characters instead.
These results were not intended, obviously, but nigh impossible to prevent, particularly when an existing AI model is used rather than created from scratch (which presently would still be prohibitively expensive).
While this particular incident may be considered mostly funny, in a different context such a vulnerability could pose a security risk or even worse. The AIs are built with safeguards, but crafty people can bypass those by phrasing their prompts in certain ways.
8. Reality
Reality can be very subjective, since there are now systems that can create text, voice, paintings, photos, and videos of people that appear about as real as an original – except (for instance) that the person in question never said what is quoted… this is problematic. Most countries are currently working on legislation to ban such uses, however that may not prevent some less obvious cases. And once ‘out there’ (broadcast or put online) the damage is done. If viewers are less discerning or don’t have access to other sources of information, they can easily be taken in by these ‘deep fakes’.
9. Copyright
Copyright is the exclusive right of an original creator of a work to say who may (among other actions,) copy their work, and under what conditions. Without such permission, third parties do not have copyright.
While one might think that everything visible on the Internet is ‘free’, it is actually still copyright of the creator and made visible under their conditions. For instance, they might display ads alongside the content. Without explicit permission, you are not allowed to take content and put it on your own site.
Already battles are being fought by content creators such as online newspapers with, for instance, OpenAI. The allegation is that the creators’ content has been used by OpenAI to train their AI models, without an agreement (including potential payment) in place. OpenAI argues that the use of copyrighted materials in transformative ways does not violate copyright laws.
If we have a generative AI create new content, it is doing so based on its training: an amalgamation of ingested content. Who holds the copyright to the new “mixed” (derivative) work? In general copyright law it might be either you or the owner of the generative AI, depending on their terms of use. However, that is contingent on the original content being appropriately licensed, and such is not the case here – in fact, the origins of the constituent snippets are essentially unknown. Realistically, we have to assume that there is no license. It is currently a dark grey area. Ideally, and for lack of opportunity to otherwise credit the original content creators, you would indicate that a text, image or other content is AI generated.
10. Hallucinations vs Idempotency
Last but not least, LLMs exhibit what is commonly known as hallucinations: they periodically deliver results that superficially appear sensible, but are really nonsense. One reason this occurs is because LLMs are probabilistic systems, they make a prediction about for instance a next word based on the previous – just like the simple word prediction example from part 1 of this series. Additionally, they apply a random factor, so their response is not always the same, even for an identical prompt. Therefore, it is possible to get good (but not identical) answers for a time, then a bad one, and then good ones again, while using the same question all the way.
Software engineering has a concept called idempotency, meaning that for a given input, a program produces the same output. Well-engineered programs have this trait, making them robust for applying business rules. As shown above, LLMs do not have this trait by default and fundamentally, LLMs do not possess understanding or have true knowledge.
That said, even minor changes in the phrasing of a question (prompt engineering) can reduce occurrence of hallucinations, and having a process with a ‘human in the loop’ where human reviewers monitor and correct outputs, yields higher quality results.
Mitigations: the Bright Intern
Now that we’ve reviewed a lot of risks and instances where things could go wrong, what measures can we put in place to prevent issues?
Regard an AI-enabled system as a bright intern. You can let it do work with a certain amount of independence, but you never give it control over final decisions; it requires active supervision and review.
Where an AI-enabled system really differs from a bright human intern is that an AI does not become smarter or more trustworthy over time. This appears very counter-intuitive, but as we’ve seen in the various examples, things can (and do) go bad later. An AI is not a human, and while they can learn things, they do not gain experience in the same way that humans do.
In the 3rd and final part of this series, we will put together what we learnt about AI so far, and how ISO 42001:2023 can be used to responsibly develop and implement AI-enabled systems within your organisation.
Arjen Lentz
Senior Consultant (Governance, Risk, Compliance), Sekuro
More Articles
Sekuro’s Cyber Leadership Exchange 2024: Innovation, Governance and Resilience (Part 2)
Sekuro’s Cyber Leadership Exchange 2024: Innovation, Governance and Resilience (Part 1)
