Part of Advances in Neural Information Processing Systems 6 (NIPS 1993)
Satinder Singh, Andrew Barto, Roderic Grupen, Christopher Connolly
While exploring to find better solutions, an agent performing on(cid:173) line reinforcement learning (RL) can perform worse than is accept(cid:173) able. In some cases, exploration might have unsafe, or even catas(cid:173) trophic, results, often modeled in terms of reaching 'failure' states of the agent's environment. This paper presents a method that uses domain knowledge to reduce the number of failures during explo(cid:173) ration. This method formulates the set of actions from which the RL agent composes a control policy to ensure that exploration is conducted in a policy space that excludes most of the unacceptable policies. The resulting action set has a more abstract relationship to the task being solved than is common in many applications of RL. Although the cost of this added safety is that learning may result in a suboptimal solution, we argue that this is an appropri(cid:173) ate tradeoff in many problems. We illustrate this method in the domain of motion planning.
"'This work was done while the first author was finishing his Ph.D in computer science
at the University of Massachusetts, Amherst.