Uncannily specific ads target you every single day. With the EU’s ‘Right to an explanation’, you get a peek at the algorithm that decides it. Oscar Schwartz explains why that’s more complicated than it sounds.

If you’re an EU resident, you will now be entitled to ask Netflix how the algorithm decided to recommend you The Crown instead of Stranger Things. Or, more significantly, you will be able to question the logic behind why a money lending algorithm denied you credit for a home loan.

This is because of a new regulation known as “the right to an explanation”. Part of the General Data Protection Regulation that has come into effect in May 2018, this regulation states users are entitled to ask for an explanation about how algorithms make decisions. This way, they can challenge the decision made or make an informed choice to opt out.

Supporters of this regulation argue that it will foster transparency and accountability in the way companies use AI. Detractors argue the regulation misunderstands how cutting-edge automated decision making works and is likely to hold back technological progress. Specifically, some have argued the right to an explanation is incompatible with machine learning, as the complexity of this technology makes it very difficult to explain precisely how the algorithms do what they do.

As such, there is an emerging tension between the right to an explanation and useful applications of machine learning techniques. This tension suggests a deeper ethical question: Is the right to understand how complex technology works more important than the potential benefits of inherently inexplicable algorithms? Would it be justifiable to curtail research and development in, say, cancer detecting software if we couldn’t provide a coherent explanation for how the algorithm operates?

The limits of human comprehension

This negotiation between the limits of human understanding and technological progress has been present since the first decades of AI research. In 1958, Hannah Arendt was thinking about intelligent machines and came to the conclusion that the limits of what can be understood in language might, in fact, provide a useful moral limit for what our technology should do.

In the prologue to The Human Condition she argues that modern science and technology has become so complex that its “truths” can no longer be spoken of coherently. “We do not yet know whether this situation is final,” she writes, “but it could be that we, who are earth-bound creatures and have begun to act as though we were dwellers of the universe, will forever be unable to understand, that is, to think and speak about the things which nevertheless we are able to do”.

Arendt feared that if we gave up our capacity to comprehend technology, we would become “thoughtless creatures at the mercy of every gadget which is technically possible, no matter how murderous it is”.

While pioneering AI researcher Joseph Weizenbaum agreed with Arendt that technology requires moral limitation, he felt that she didn’t take her argument far enough. In his 1976 book, Computer Power and Human Reason, he argues that even if we are given explanations of how technology works, seemingly intelligent yet simple software can still create “powerful delusional thinking in otherwise normal people”. He learnt this first hand after creating an algorithm called ELIZA, which was programmed to work like a therapist.

While ELIZA was a simple program, Weizenbaum found that people willingly created emotional bonds with the machine. In fact, even when he explained the limited ways in which the algorithm worked, people still maintained that it had understood them on an emotional level. This led Weizenbaum to suggest that simply explaining how technology works is not enough of a limitation on AI. In the end, he argued that when a decision requires human judgement, machines ought not be deferred to.

While Weizenbuam spent the rest of his career highlighting the dangers of AI, many of his peers and colleagues believed that his humanist moralism would lead to repressive limitations on scientific freedom and progress. For instance, John McCarthy, another pioneer of AI research, reviewed Weizenbaum’s book, and countered it by suggesting overregulating technological developments goes against the spirit of pure science. Regulation of innovation and scientific freedom, McCarthy adds, is usually only achieved “in an atmosphere that combines public hysteria and bureaucratic power”.

Where we are now

Decades have passed since these first debates about human understanding and computer power took place. We are only now starting to see them breach the realms of philosophy and play out in the real world. AI is being rolled out in more and more high stakes domains as you read. Of course, our modern world is filled with complex systems that we do not fully understand. Do you know exactly how the plumbing, electricity, or waste disposal that you rely on works? We have become used to depending on systems and technology that we do not yet understand.

But if you wanted to, you could come to understand many of these systems and technologies by speaking to experts. You could invite an electrician over to your home tomorrow and ask them to explain how the lights turn on.

Yet, the complex workings of machine learning means that in the near future, this might no longer be the case. It might be possible to have a TV show recommended to you or your essay marked by a computer and for there to be no-one, not even the creator of the algorithm, to explain precisely why or how things happened the way they happened.

The European Union have taken a moral stance against this vision of the future. In so doing, they have aligned themselves, morally speaking, with Hannah Arendt, enshrining a law that makes the limited scope of our “earth-bound” comprehension a limit for technological progress.

An earlier version of this article was published on Ellen’s blog.

Centrelink’s debt recovery woes perfectly illustrate the human side of data modelling.

The Department for Human Services issued 169,000 debt notices after automating its processes for matching welfare recipients’ reported income with their tax. Around one in five people are estimated not to owe any money. Stories abounded of people receiving erroneous debt notices up to thousands of dollars that caused real anguish.

Coincidentally, as this unfolded, one of the books on my reading pile was Weapons of Math Destruction by Cathy O’Neil. She is a mathematician turned quantitative analyst turned data scientist who writes about the bad data models increasingly being used to make decisions that affect our lives.

Reading Weapons of Math Destruction as the Centrelink stories emerged left me thinking about how we identify ‘bad’ data models, what ‘bad’ means and how we can mitigate the effects of bad data on people. How could taking an ethics based approach to data help reduce harm? What ethical frameworks exist for government departments in Australia undertaking data projects like this?

Bad data and ‘weapons of math destruction’

A data model can be ‘bad’ in different ways. It might be overly simplistic. It might be based on limited, inaccurate or old information. Its design might incorporate human bias, reinforcing existing stereotypes and skewing outcomes. Even where a data model doesn’t start from bad premises, issues can arise about how it is designed, its capacity for error and bias and how badly people could be impacted by error or bias.

Weapons of math destruction tend to hurt vulnerable people most.

A bad data model spirals into a weapon of math destruction when it’s used en masse, is difficult to question and damages people’s lives.

Weapons of math destruction tend to hurt vulnerable people most. They might build on existing biases – for example, assuming you’re more likely to reoffend because you’re black or you’re more likely to have car accidents if your credit rating is bad. Errors in the model might have starker consequences for people without a social safety net. Some people may find it harder than others to question or challenge the assumptions a model makes about them.

Unfortunately, although O’Neil tells us how bad data modelling can lead to weapons of math destruction, it doesn’t tell us much about how we can manage these weapons once they’ve been created.

Better data decisions

We need more ways to help data scientists and policymakers navigate the complexities of projects involving personal data and their impact on people’s lives. Regulation has a role to play here. Data protection laws are being reviewed and updated around the world.

For example, in Australia the draft Productivity Commission report on data sharing and use recommends the introduction of new ‘consumer rights’ over their personal data. Bodies such the Office of the Information Commissioner help organisations understand if they’re treating personal data in a principled manner that promotes best practice.

Guidelines are also being produced to help organisations be more transparent and accountable in how they use data to make decisions. For instance, The Open Data Institute in the UK has developed openness principles designed to build trust in how data is stored and used. Algorithmic transparency is being contemplated as part of the EU Free Flow of Data Initiative and has become a focus of academic study in the US.

Ethics can help bridge the gap between compliance and our evolving expectations of what is fair and reasonable data usage.

However, we cannot rely on regulation alone. Legal, transparent data models can still be ‘bad’ according to O’Neil’s standards. Widely known errors in a model could still cause real harm to people if left unaddressed. An organisation’s normal processes might not be accessible or suitable for certain people – the elderly, ill and those with limited literacy – leaving them at risk. It could be a data model within a sensitive policy area, where a higher duty of care exists to ensure data models do not reflect bias. For instance, proposals to replace passports with facial recognition and fingerprint scanning would need to manage the potential for racial profiling and other issues.

Ethics can help bridge the gap between compliance and our evolving expectations of what is fair and reasonable data usage. O’Neil describes data models as “opinions put down in maths”. Taking an ethics based approach to data driven decision making helps us confront those opinions head on.

Building an ethical framework

Ethics frameworks can help us put a data model in context and assess its relative strengths and weaknesses. Ethics can bring to the forefront how people might be affected by the design choices made in the course of building a data model.

An ethics based approach to data driven decisions would start by asking questions such as:

• Are we compliant with the relevant laws and regulation?
• Do people understand how a decision is being made?
• Do they have some control over how their data is used?
• Can they appeal a decision?

However, it would also encourage data scientists to go beyond these compliance oriented questions to consider issues such as:

• Which people will be affected by the data model?
• Are the appeal mechanisms useful and accessible to the people who will need them most?
• Have we taken all possible steps to ensure errors, inaccuracies and biases in our model have been removed?
• What impact could potential errors or inaccuracies have? What is an acceptable margin of error?
• Have we clearly defined how this model will be used and outlined its limitations? What kinds of topics would it be inappropriate to apply this modelling to?

There’s no debate right now to help us understand the parameters of reasonable and acceptable data model design. What’s considered ‘ethical’ changes as we do, as technologies evolve and new opportunities and consequences emerge.

Bringing data ethics into data science reminds us we’re human. Our data models reflect design choices we make and affect people’s lives. Although ethics can be messy and hard to pin down, we need a debate around data ethics.

The sexbots and robo-soldiers we’re creating today take Bladerunner and Westworld out of the science fiction genre. Kym Middleton looks at what those texts reveal on how we should treat humanlike robots.

It’s certain: lifelike humanoid robots are on the way.

With guarantees of Terminator-esque soldiers by 2050, we can no longer relegate lifelike robots to science fiction. Add this to everyday artificial intelligence like Apple’s Siri, Amazon’s Alexa and Google Home and it’s easy to see an android future.

The porn industry could beat the arms trade to it. Realistic looking sex robots are being developed with the same AI technology that remembers what pizza you like to order – although it’s years away from being indistinguishable from people, as this CNET interview with sexbot Harmony shows.

Like the replicants of Bladerunner we first met in 1982 and the robot “hosts” of HBO’s remake of the 1973 film Westworld, these androids we’re making require us to answer a big ethical question. How are we to treat walking, talking robots that are capable of reasoning and look just like people?

Can they suffer?

If we apply the thinking of Australian philosopher Peter Singer to the question of how we treat androids, the answer lies in their capacity to suffer. In making his case for the ethical consideration of animals, Singer quotes Jeremy Bentham:

“The question is not, Can they reason? nor Can they talk? but, Can they suffer?”

An artificially intelligent, humanlike robot that walks, talks and reasons is just that – artificial. They will be designed to mimic suffering. Take away the genuine experience of physical and emotional pain and pleasure and we have an inanimate thing that only looks like a person (although the word ‘inanimate’ doesn’t seem an entirely appropriate adjective for lifelike robots).

We’re already starting to see the first androids like this. They are, at this point, basically smartphones in the form of human beings. I don’t know about you, but I don’t anthropomorphise my phone. Putting aside wastefulness, it’s easy to make the case you should be able to smash it up if you want.

But can you (spoiler) sit comfortably and watch the human-shaped robot Dolores Abernathy be beaten, dragged away and raped by the Man in Black in Westworld without having an empathetic reaction? She screams and kicks and cries like any person in trauma would. Even if robot Dolores can’t experience distress and suffering, she certainly appears to. The robot is wired to display pain and viewers are wired to have a strong emotional reaction to such a scene. And most of us will – to an actress, playing a robot, in a fictional TV series.

Let’s move back to reality. Let’s face it, some people will want to do bad things to commercially available robots – especially sexbots. That’s the whole premise of the Westworld theme park, a now not so sci-fi setting where people can act out sexual, violent, and psychological fantasies on android subjects without consequences. Are you okay with that becoming reality? What if the robots looked like children?

The virtue ethicist’s approach to human behaviour is to act with an ideal character, to do right because that’s what good people do. In time, doing the virtuous thing will be habit, a natural default position because you internalise it. The virtue ethicist is not going to be okay with the Man in Black’s treatment of Dolores. Good people don’t have dark fantasies to act out on fake humans.

The utilitarian approach to ethical decisions depends on what results in the most good for the largest amount of people. Making androids available for abuse could be a case for community safety. If dark desires can be satiated with robots, actual assaults on people could reduce. (In presenting this argument, I’m not proposing this is scientifically proven or that it’s my view.) This logic has led to debates on whether virtual child porn should be tolerated.

The deontologist on the other hand is a rule follower so unless androids have legal protections or childlike sexbots are banned in their jurisdiction, they are unlikely to hold a person who mistreats one in ill regard. If it’s your property, do whatever you’re allowed to do with it.

Consciousness

Of course, (another spoiler) the robots of Westworld and Bladerunner are conscious. They think and feel and many believe themselves to be human. They experience real anguish. Singer’s case for the ethical treatment of animals relies on this sentience and can be applied here.

But can we create conscious beings – deliberately or unwittingly? If we really do design a new intelligent android species, complete with emotions and desires that motivate them to act for themselves, then give them the capacity to suffer and make conscientious choices, we have a strong case for affording robot rights.

This is not exactly something we’re comfortable with. Animals don’t enjoy anything remotely close to human rights. It is difficult seeing us treat man made machines with the same level of respect we demand for ourselves.

Why even AI?

As is often with matters of the future, humanlike robots bring up all sorts of fascinating ethical questions. Today they’re no longer fun hypotheticals. It is important stuff we need to work out.

Let’s assume for now we can’t develop the free thinking and feeling replicants of Bladerunner and hosts of Westworld. We still have to consider how our creation and treatment of androids reflects on us. What purpose – other than sexbots and soldiers – will we make them for? What features will we design into a robot that is so lifelike it masterfully mimics a human? Can we avoid designing our own biases into these new humanoids? How will they impact our behaviour? How will they change our workplaces and societies? How do we prevent them from being exploited for terrible things?

Maybe Elon Musk is right to be cautious about AI. But if we were “summoning the demon”, it’s the one inside us that’ll be the cause of our unease.

Follow The Ethics Centre on Twitter, Facebook, Instagram and LinkedIn.