Joint CeNS Colloquium with SFB1032

Title: Three-factor learning rules: from reward to surprise

Location: Faculty of Physics, Kleiner Physikhörsaal N020 and online via Zoom

Host: Prof. Joachim Rädler, Dr. Friedhelm Serwane

Summary:
Neurons are connected by synapses that can change their strength. The change in strength, called synaptic plasticity, is related to learning and follows specific rules. The classical learning rule is of a Hebbian form with two factors: what matters is the activity of the sending neuron and the state of the receiving neuron, i.e., the change depends on locally available information. Theory has predicted and recent experiments from several labs have shown that two factors are not enough; we also need to consider widely broadcast neuromodulators that play the role of a third factor. The third factor can convey a signal that is related to reward [1] which has given rise to a whole research field of reward-based learning. We show how the resulting three-factor rules can be formulated on the level of spikes [2,3]. But humans and animals also learn in the absence of reward. In the second part of the talk, I will explain how surprise can replace reward and play the role of a third factor in the learning rule [4,5].

[1] W. Schultz, P. Dayan and R.R. Montague (1997)
A neural substrate for prediction and reward. Science 275, pp. 1593–1599.

[2] N. Fremaux and H. Sprekeler and W. Gerstner (2013)
Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons
PLOS Comput. Biol. 9: e1003024. doi:10.1371/journal.pcbi.1003024

[3] W. Gerstner, M. Lehmann, V. Liakoni, and J. Brea (2018)
Eligibility traces and plasticity on behavioral time scales: experimental support of NeoHebbian three-factor learning rules.
Front. Neural Circuits, 12:53 doi: 10.3389/fncir.2018.00053

[4] M.P. Lehmann, H.A. Xu, V. Liakoni, M.H. Herzog, W. Gerstner, and K. Preuschoff (2019)
One-shot learning and behavioral eligibility-traces in sequential decision making
eLife 8:e47463 doi: 10.7554/eLife.47463

[5] H.A. Xu, A. Modirshanechi, M.P. Lehmann, W. Gerstner, M.H. Herzog (2021)
Novelty is not Surprise: Human exploratory and adaptive behavior in sequential decision-making
PLoS Comput Biol 17: e1009070. doi: doi.org/10.1371/journal.pcbi.1009070

See also CeNS-colloquiums-website