PSYC 310 Lecture Notes - Lecture 11: Knockout Mouse, Striatum, Dopamine Receptor
7 May 2018
School
Department
Course
Professor
PSYC 318
Behavioural Neuroscience II
January 22nd, 2018
Lecture 5/24: Dopamine-dependent behavior
• Reinforcement learning & dopamine
o We think dopamine encodes a reinforcement learning signal that alerts the brain
as to when current reward or current expectations of reward are better or worse
than they were anticipated to be in the previous moment. Dopamine release
encodes a reward prediction error signal.
o Prediction error signal = actual – expected value of current situation
• This dopamine signal broadcast widely through the brain. Fluctuations indicate that
credit/blame have to be apportioned accordingly
• Dopamine neurons encode a prediction error
o Initial experiments were done in the 1970s on monkey recordings
▪ They would tie a monkey down in chair, place metal wires in midbrain,
record action potentials that came from neurons in the midbrain
▪ Monkey would do simple tasks
▪ For reward, a straw was put into monkey’s mouth and would deliver a
juice
▪ Every time there was a neuronal action potential detected in wire they put
a black dot on this graph
• Each line of black dots is a 2 second long recording in the
midbrain
• Raster plot graph
• Histogram at top where they add up all the black dots
• When they first hit record, in the first second there’s baseline
dopamine neuron activity, but when juice reward is administered
there’s a boost in dopamine activity, more action potentials
• Dopamine neurons tend to spike more when monkeys
unexpectedly receive juice rewards
o This was done in the 70s, now more elaborate techniques are used
o Animal is not getting any indication that it’s about to get koolaid
o They varied the animal’s knowledge of whether or not it will receive the juice
reward
▪ In top example, animal has no expectation of reward
▪ On the bottom example, guaranteed reward, animal is extremely well
trained and gets a clue noting that in 2 seconds a reward will be
administered, day after day for months
• When animal actually gets reward there is no change in
dopamine neuron activity
▪ 50% chance: dopamine neurons do fire, but amount of dopamine release
is about half as much as what would happen with unexpected reward
administration
▪ In situation where a cue comes and monkey knows that he has a 50%
chance of getting reward will happen, the sum of the neural activity at two
critical time (cue, reward) points equals amount of dopamine release that
happened when reward was unexpected or to the cue when the reward
was expected
find more resources at oneclass.com
find more resources at oneclass.com
▪ The cue indicates 75% reward, 75% of neural acitivty to cue and other
25% to the actual reward…
o Dopamine neurons abruptly stop firing when an expected reward is not received.
This is a negative prediction error that indicates to brain the current situation is
worse than was anticipated
• Expectations of reward affect learning
o Classic behaviors: phenomenon of ‘blocking’
▪ Two training phases: both just cues and rewards. Monkey is in chair,
presented with cue, give reward, see what it learns.
▪ You train monkey that some random symbol on screen preceded a
reward 2 seconds later (unconditioned stimulus)
• Train for weeks.
▪ In training phase 2: same cue appears, but in compound with a second
cue, same reward administered.
▪ What has animal learned about the second compound cue?
o Test: present each stimulus on its own: if you just present the second cue, the
blocking effect is administered when there’s no response to second cue
▪ Animal learns nothing about the second cue, it is fully redundant and
provides no new info about the animal’s world.
• Demonstration of blocking (from paper #4)
o Classic blocking phenomenon task
▪ Train group of rats: tone A produces juice reward, it comes on for about
30 seconds and juice reward is available when tone A is being played
• During training, (single cue training), animals begin to spend more
and more time in the food port while the tone is on
• Graph represents the % of time animals are spending in the food
port while the tone is on
• Over 5 days, animal really learns contingency, about 90% of time
the tone is on their head is in the food port licking the food
▪ Second phase: same tone is played, but this time it is also paired with a
light appearing above the food port, indicating as well that food is
available but only ever seen in compound with the tone
▪ When you introduce light there’s not really any change in behavior
▪ After 4 days of this, what happens if you just shine light but there’s no
tone?
• On test phase, there is no expectation of reward when you shine
the light, animals don’t stick their head in food port at all and
instead seem to be waiting for the tone
▪ The light was a redundant cue that blocked learning
o Second case: tone B predicts food, switch to tone A and light X in phase 2. Tone
A and the light are both new for this control group.
▪ When they hear tone A and see light it takes a couple days for them to
fully appreciate that these cues also predict cues like tone B did
▪ When you just play the light these animals do show some evidence that
they think light means food is available, they stick their heads more in
food port than the blocking group did.
o These test probes done during extinction conditions, where there is no food at all.
• Similar type of task in monkey when recording dopamine neurons
o How do dopamine neurons follow throughout this task?
find more resources at oneclass.com
find more resources at oneclass.com
o They train animal on phase one: monkey is looking at screen, learn that one cue
predicts reward and one predicts nothing.
o Training phase 2: all cues become compound cues
▪ Now different pairs of cues predict reward
▪ When a blocked cue appears it doesn’t change dopaminergic activity at
all.
• This cue is a perfect predictor of reward, but when animal sees it
on its own it doesn’t change dopaminergic activity or the monkey’s
behavior
• Gold symbol on its own indicates food is likely to come
• How does dopamine discriminate between cues when it reinforces neural activity?
o Why does blocking occur? When reward is delivered but it is predicted, there’s
no change in dopamine.
o Dopamine would be needed at that time to promote learning about antecedent
cues, when the reward is unexpected
o Dopamine release when things were unexpectedly occurring, promoting learning
about cues coming before event
• Second-order conditioning:
o Putting more cues into the mix with some kind of timing involved
o Creating expectations for reward in response to cues that were never explicitly
paired with reward
o Dopamine signals keep moving back to the earliest predictor of food
o This can happen even when you stop delivering food (extinction)
o Depending on the exact design of the experiment you can play around with when
the animal will expect food, at what probability, which cues add value and which
subtract value…
▪ Changes in dopamine neuron activity follow expectations closely
• Dopamine photo-stimulation prevents blocking
o In the blocking group, tone A predicts food so when tone A and light came on to
predict food, learning about the light was blocked
▪ Because dopamine neurons didn’t change their activity when reward
came
▪ What if at this moment, when tone light appears and reward comes, while
drinking the reward we stimulate dopamine activity?
▪ At this moment we assume the rat will think that this reward was better
than expected, and because the light was now paired with tone the light
must indicate a better reward
o In dopamine-cre rats using a cre dependent virus to drive ChR2 in midbrain
dopamine neurons
▪ Train them in the same way with tone A predicting food
▪ As soon as they introduce compound light, every time animal drinks
reward they stimulate dopamine neuron
• If it was a paired stimulation: they learned that the light has
meaning, because the first time they see the light they run over to
check the food port
o When you stimulate dopamine neurons at any time animals immediately think
something great just unexpectedly happened, their first instinct is to repeat
whatever they just did
▪ Shortcoming of this paradigm
• Pleasure versus prediction error
find more resources at oneclass.com
find more resources at oneclass.com
Document Summary
Dopamine release encodes a reward prediction error signal: prediction error signal = actual expected value of current situation, this dopamine signal broadcast widely through the brain. Fluctuations indicate that credit/blame have to be apportioned accordingly: dopamine neurons encode a prediction error. 25% to the actual reward : dopamine neurons abruptly stop firing when an expected reward is not received. This is a negative prediction error that indicates to brain the current situation is worse than was anticipated: expectations of reward affect learning, classic behaviors: phenomenon of blocking", two training phases: both just cues and rewards. Monkey is in chair, presented with cue, give reward, see what it learns: you train monkey that some random symbol on screen preceded a reward 2 seconds later (unconditioned stimulus, train for weeks. If you lose all dopamine receptor activity, severe parkinson"s situation and animals cannot initiate purposeful movement, and become locked-in".
