Safety Regulation for Autonomous Systems

Software	None
Covers	Human factors, computer law, product development processes
Skills Required	Interest in AI and autonomous systems, interest in development processes
Challenge	Conceptual ???? Technical ??? Programming

Brief Description

The development, introduction and use of products that are safety-critical or might be used in complex or uncontrolled environments is governed in most jurisdictions by extensive guidelines and regulations. Some of these regulations have the force of law. But the rise of autonomous systems, and the increasing use of machine learning techniques to develop elements of control systems that are induced from training data rather than authored by a developer who understands how they work, may lead to the widespread use of systems for which the existing regulatory framework isn't adequate.

The challenge of this project is to investigate the methods for regulating and quality-assuring the development of autonomous systems or some other kind of complex computerized systems, to see what the practical and ethical challenges are for governing the development and use of such systems, so that they aren't used for evil and don't cause accidental harm, and the risks they entail are understood and considered, and how well the regulatory regime meets them.

In an article in IEEE Intelligent Systems, David Danks and Alex John London of Carnegie Mellon University consider this issue and suggest modelling the regulatory regime for the introduction of autonomous systems on the regulatory procedures for drugs.

A key source of ideas for this would the work of Nancy Leveson of MIT on understanding and preventing design errors in software and engineering development processes.

Variants

One way to take this project would be to identify one new class of complex or safety-critical computerized product, and investigate whether the legal frameworks and regulations and procedures that govern it are adequate and fit for purpose, or if this new class of product involves distinctive issues that need special treatment, or if the issues that need addressing apply to a larger class of systems.

This could focus on one aspect of the legal or ethical issues, or the practicalities of managing and regulating design and deployment to eliminate or mitigate certain types of risk.

This could consider the issues raised by the use of systems in different countries, and whether and how transnational regulation could work, or what would be needed in countries that currently have weak or non-existent regulation.

Back to