Scientists discover how local brain cells hijack serotonin signaling

A new study published in Nature Communications reveals that a brain chemical called acetylcholine can directly trigger the release of serotonin. These results suggest that the brain uses a highly coordinated system to link different chemical messengers. The findings might help explain the biological roots of compulsive behaviors seen in psychiatric conditions like obsessive-compulsive disorder.

The brain relies on an array of chemical messengers to transmit signals across microscopic gaps between nerve cells. These chemicals allow the brain to coordinate everything from basic reflexes to abstract thoughts. The striatum is a deep brain structure that acts as a central hub for processing habits, movement, and goal-directed learning.

Within this region, a small population of cells known as cholinergic interneurons act like local conductors. Interneurons are specialized nerve cells that form connections between other neurons, helping to manage the flow of information within a specific local area. These particular interneurons release acetylcholine, a chemical messenger that helps the brain respond to important behavioral events. Acetylcholine works by binding to specific proteins on the surface of other cells, much like a key fitting into a lock.

Past research established that acetylcholine prompts the release of dopamine, which is the brain’s primary reward chemical. Because serotonin is another major chemical messenger heavily involved in mood and learning, researchers wanted to find out if acetylcholine could exert similar control over it. Serotonin imbalances are widely linked to psychiatric and addictive conditions.

Lior Matityahu, a researcher at the Hebrew University of Jerusalem, led the investigation alongside neurobiologists Joshua Goldberg from the same institution and Joshua Plotkin from Stony Brook University. They designed a series of experiments to observe exactly how these chemical systems interact in living brain tissue.

The research team started by introducing a genetic tool into the dorsal striatum of mice. The dorsal striatum is the upper section of the striatum, an area heavily involved in motor control and habit formation. The genetic tool was an engineered virus that caused the brain cells to produce a custom green fluorescent protein. This protein lights up only when it binds to serotonin, providing a visual cue for chemical activity.

By placing very thin slices of these brains under an advanced microscope, the team could watch serotonin fluctuations in real time. When the researchers delivered brief electrical pulses to the brain tissue, the green fluorescence spiked, indicating a robust release of serotonin. The glow then slowly faded over tens of seconds as the chemical naturally dissipated.

To verify that the glow was truly serotonin and not a related chemical like dopamine, the researchers bathed the tissue in dopamine. The fluorescence did not change, confirming the sensor’s accuracy. They also applied a selective serotonin reuptake inhibitor, a common type of antidepressant drug, which predictably altered the speed at which the fluorescent signal decayed.

Next, they applied a drug called mecamylamine to the tissue. Mecamylamine blocks nicotinic acetylcholine receptors, which are specialized docking stations on the surface of cells that respond to acetylcholine. With these receptors blocked, the amount of serotonin released dropped substantially. This drop indicated that acetylcholine was actively helping to drive the serotonin response.

The team also measured how far the serotonin traveled from the stimulation site. They measured the brightness of the fluorescence at microscopic increments moving away from the electrical source.

They found that the presence of active acetylcholine receptors allowed the serotonin signal to spread across a much wider area of the brain tissue. Blocking the receptors reduced this spatial footprint by nearly half. This means acetylcholine not only increases the volume of serotonin released but also dictates how far its message reaches.

The researchers repeated these steps in the lower part of the region, known as the ventral striatum. This area is known to have a much denser network of serotonin-producing fibers than the upper section.

To their surprise, the acetylcholine-blocking drug had no effect on serotonin levels in this lower region. This result revealed that the connection between acetylcholine and serotonin is highly localized to specific brain areas. The density of the serotonin fibers did not dictate the strength of the chemical interaction.

To prove that the cholinergic interneurons were the exact source of the acetylcholine driving this process, the team used optogenetics. Optogenetics is a technique that allows scientists to engineer specific brain cells so they can be activated by flashes of light. They genetically modified the cholinergic cells in mice so that a brief pulse of blue light would force them to fire.

When the researchers shined light on the tissue, the cholinergic cells fired simultaneously. Immediately, the green sensors lit up, showing a massive release of serotonin. To rule out other indirect causes, they added a mixture of drugs that block receptors for other common brain chemicals, such as glutamate.

The light-induced serotonin release remained entirely unchanged in the presence of these other blocking drugs. However, when they blocked the acetylcholine receptors again, the light-induced serotonin release vanished completely. This confirmed that the cholinergic interneurons alone were entirely responsible for triggering the local serotonin release.

To understand how this mechanism behaves in disease states, the team examined mice genetically engineered to lack a specific gene called Sapap3. Obsessive-compulsive disorder in humans is characterized by intrusive thoughts and repetitive actions that patients feel driven to perform. Mice missing the Sapap3 gene display similar behavioral loops, such as grooming themselves until they cause physical injury.

Previous studies had shown that the striatum in these mice contains too much acetylcholine. The researchers applied the same electrical stimulation to brain tissue from the genetically modified mice. They observed a massive surge in serotonin release compared to typical mice.

When they applied the drug that blocks acetylcholine receptors, the difference disappeared entirely. The underlying baseline serotonin system was normal, but the excessive acetylcholine was pushing the serotonin release into overdrive.

“Our findings show that the brain’s internal wiring allows one chemical system to take the wheel of another in a highly regional and specific way,” Goldberg and Plotkin explained in a press statement. They noted that this hijacked signaling system likely contributes to the repetitive actions seen in certain psychiatric conditions.

“In conditions like OCD, where cholinergic signaling may be dysfunctional, this normally helpful coordination may go into overdrive, which could help explain why certain behaviors become so difficult to stop,” they added.

While the study presents clear evidence of this chemical coordination in isolated brain slices, there are still open questions about how it operates in living, behaving animals. It is difficult to pinpoint the exact behavioral states that prompt cholinergic interneurons to fire synchronously in a natural environment.

The researchers suspect that stressful events or highly salient environmental cues might be required to trigger this linked chemical release outside the laboratory. For example, previous studies have shown that severe stress is required to prompt noticeable serotonin release when certain chemicals are introduced to the brain. Future research will need to establish the precise real-world conditions that cause acetylcholine to drive serotonin release.

Scientists will also need to determine if this hijacked chemical release happens in humans, or if it varies across different species. In addition, further studies might explore whether abnormal cholinergic activity contributes to the progression of Parkinson’s disease. Parkinson’s disease involves the loss of dopamine fibers in the striatum, which sometimes causes serotonin fibers to inappropriately adopt dopamine-related functions.

Understanding how different chemical systems regulate one another is a major step toward developing better treatments for psychiatric and neurological conditions. Current treatments for obsessive-compulsive disorder often rely on drugs that alter serotonin levels throughout the entire brain. Discovering how acetylcholine controls serotonin in a highly localized way might lead to more targeted therapies.

The study, “Synchronous activation of striatal cholinergic interneurons induces local serotonin release,” was authored by Lior Matityahu, Zachary B. Hobel, Noa Berkowitz, Jeffrey M. Malgady, Naomi Gilin, Joshua L. Plotkin & Joshua A. Goldberg.