Biological Basis of the Code: Modeling Basal Ganglia and Frontal Cortex Interactions

The provided computational neuroscience code represents a model that simulates interactions between the basal ganglia and the frontal cortex in the context of reward-based learning. The model is designed to capture key mechanisms involved in decision-making and learning from environmental stimuli and rewards. Below, I outline the biological basis of the components modeled by the code:

Basal Ganglia

Actor-Critic Model

The basal ganglia are modeled using an Actor-Critic framework. This framework is inspired by the role of the basal ganglia in reinforcement learning and decision-making. In this model:

• Actor: Proposes actions based on current policy. This is akin to the action selection role of the basal ganglia, where actions are selected based on the reward prediction.
• Critic: Evaluates the action and updates the policy based on the reward received (dopaminergic signal). The Critic's functioning is reminiscent of the dopaminergic system, particularly the role of dopamine in signaling prediction errors and modulating learning.

Dopaminergic Learning

The dopamine signal (DA, Dopamine Activity) is critical, as it provides the reward prediction error used to update both the Actor and the Critic. This is biologically inspired by the role of midbrain dopamine neurons, which are known to encode rewards and prediction errors.

Frontal Cortex

LSTM (Long Short-Term Memory)

The model includes an LSTM network that acts as a proxy for the frontal cortex:

• Working Memory: The LSTM is used to maintain and update information over time, akin to how the frontal cortex is involved in working memory processes.
• Prediction: The LSTM's task is to predict its next inputs based on current and past information, mirroring the role of the frontal cortex in making predictions about future events.

Eligibility Traces

The LSTM uses eligibility traces (ETLSTM1), a mechanism that stores temporal information about the inputs received, allowing the network to credit rewards and outcomes back to previous states. This reflects synaptic plasticity principles in the brain, where recent activities have a higher probability of influencing learning.

Composite Model: Integration of Basal Ganglia and Cortex

The integration of the Actor-Critic model with the LSTM network reflects the interaction between the basal ganglia and the frontal cortex. These structures work in parallel:

• Interaction: The output of the LSTM (frontal cortex) from the previous time step is fed into the basal ganglia to inform decision-making processes.
• Feedback Loop: The basal ganglia influence action selection, while the frontal cortex adapts its predictions and internal states based on the outcomes and the dopamine signal, creating a closed-loop system.

Conclusion

Overall, this code simulates key neural circuits involved in decision-making and learning, emphasizing how the basal ganglia, with its action selection capabilities, interact with the cognitive and memory functions of the frontal cortex. This model captures essential aspects of neural processes, such as reinforcement learning dynamics, working memory, and predictive coding, using biologically inspired neural network architectures and learning rules.