Continuous-time neural reinforcement learning of working memory tasks

20^th March 2015, 11:00 Ashton Lecture Theater
Dr Sander Bohte
CWI, Life Sciences
Amsterdam

Abstract

As living organisms, one of our primary characteristics is the ability to rapidly process and react to unknown and unexpected events. To this end, we are able to recognize an event or a sequence of events and learn to respond properly. Despite advances in machine learning, current cognitive robotic systems are not able to rapidly and efficiently respond in the real world: the challenge is to learn to recognize both what is important, and also when to act. Reinforcement Learning (RL) is typically used to solve complex tasks: to learn the how. To respond quickly - to learn when - the environment has to be sampled often enough. For "enough", a programmer has to decide on the step-size as a time-representation, choosing between a fine-grained representation of time (many state-transitions; difficult to learn with RL) or to a coarse temporal resolution (easier to learn with RL but lacking precise timing). Here, we derive a continuous-time version of on-policy SARSA-learning in a working-memory neural network model, AuGMEnT. Using a neural working memory network resolves the what problem, our when solution is built on the notion that in the real world, instantaneous actions of duration dt are actually impossible. We demonstrate how we can decouple action duration from the internal time-steps in the neural RL model using an action selection system. The resultant CT-AuGMEnT successfully learns to react to the events of a continuous-time task, without any pre-imposed specifications about the duration of the events or the delays between them.

Short Bio:
-------------
Dr Sander Bohte heads the Neuroinformatics Lab at the Netherlands Centre for Mathematics and Computer Science, CWI. After graduating in expermental physics, he obtained his PhD at CWI in 2003 on the then-novel topic of spiking neural networks. In 2003-2004, he worked as a post-doc at the University of Colorado in Boulder on theoretical neuroscience and machine learning. After returning to CWI he worked as a scientific staff member on learning in multi-agent systems. Since 2009 he heads the Neuroinformatics Lab at CWI, working on biologically plausible information processing and learning in neural systems. His current research focuses on biologically plausible neural models of reinforcement learning and efficient (deep) neural networks based on spiking neurons. He teaches part-time at the University of Amsterdam on topics related to cognitive computing, and is an associate-edior for IEEE Transaction on Neural Networks and Learning Systems.