Rajesh Rao - Active Predictive Coding: A Sensory-Motor Theory of the Neocortex and a Unifying Framework for AI


Recent neurophysiological experiments indicate that almost all cortical areas, even those traditionally labelled as primary sensory cortices, are modulated by upcoming actions. Parallel evidence from neuroanatomical studies points to major outputs from deeper layer neurons across cortical areas to subcortical motor centers. To account for these findings, we propose that the neocortex implements active predictive coding (APC), a form of predictive coding that combines actions and hierarchical sensory-motor dynamics. We provide examples from simulations illustrating how the same APC architecture can solve problems that seem very different from each other - (1) how do we recognize an object and its parts using eye movements? (2) why does perception remain stable despite eye movements? (3) how do we learn compositional representations, e.g., part-whole hierarchies, and nested reference frames? (4) how do we plan actions in a complex domain by composing sequences of sub-goals and simpler actions, and (5) how do we learn episodic memories of our sensory-motor experiences and form abstract concepts such as a family tree? Besides making experimentally testable predictions in neuroscience, the APC model offers a unifying framework for addressing important problems in AI such as active sensing, compositional learning of world models, and hierarchical reinforcement learning.

SEC LL2.224