Phytochemical Modulation of Neurotransmitter Systems and Neuroplasticity in Depression and Anxiety: A Mechanistic Review

Introduction

Anxiety and depression are among the most prevalent mental health disorders worldwide, profoundly affecting individual well-being and imposing a substantial social and economic burden on society (1, 2). These conditions often arise from a complex interplay of biological, psychological, and environmental factors, including genetic predisposition, chronic stress, unhealthy lifestyle habits, and growing social pressures associated with modern life (3, 4).

Pharmacotherapy remains the mainstay treatment for depression and anxiety, commonly involving synthetic agents such as monoamine oxidase inhibitors (MAOIs) and catechol-O-methyltransferase (COMT) inhibitors. Although these drugs demonstrate clinical efficacy, their long-term use is frequently limited by adverse effects, including sexual dysfunction, sleep disturbances, hepatotoxicity, and, in severe cases, serotonin syndrome. Despite the development of newer-generation antidepressants with improved pharmacological profiles, challenges related to toxicity, cost, and variable therapeutic response continue to restrict their broader applicability (5).

These limitations have prompted growing interest in safer, more sustainable alternatives, particularly herbal-based therapies. Medicinal plants are recognized as a valuable source of bioactive compounds, including flavonoids, alkaloids, and terpenoids, many of which modulate key neurotransmitter systems such as serotonin, dopamine, and gamma-aminobutyric acid (GABA), which are central to the regulation of mood and emotional balance (6). Through a phytopharmacological lens, several species, such as Centella asiatica (gotu kola), Lavandula angustifolia (lavender), and Ginkgo biloba, have demonstrated promising antidepressant and anxiolytic properties in preclinical studies. However, despite this growing body of evidence, a systematic understanding of their mechanisms of action, efficacy, and safety remains limited.

Therefore, this study aims to comprehensively explore the role of bioactive compounds derived from medicinal plants in modulating neurotransmitter pathways implicated in depression and anxiety, and to elucidate their potential as alternative or complementary therapeutic strategies for improving mental health.

Methodology

This review was conducted to identify and analyze the potential of herbal plants in the treatment of depression and anxiety, with a particular focus on their mechanisms of action through inhibitory neurotransmission pathways in the central nervous system (CNS). A traditional narrative literature review approach was employed, using electronic databases including PubMed and Google Scholar. The search strategy incorporated combinations of the following keywords: “depression,” “anxiety,” “treatment,” “herbal plants,” “inhibitory pathways,” and “neurotransmission.” The PRISMA flow chart of the methodology can be seen in Figure 1.

Both review and original research articles were considered, with priority given to publications from the past five years to ensure the inclusion of recent scientific developments. Eligible studies met the following inclusion criteria: (1) focused on herbal plants as therapeutic agents for depression and/or anxiety; (2) examined their action through inhibitory neurotransmission or related mechanisms within the CNS; (3) were published as full-text articles in peer-reviewed journals; and (4) contained complete and accessible data. Exclusion criteria included duplicate publications, studies outside the herbal or neuropharmacological scope, articles not addressing inhibitory mechanisms, those published more than five years prior, and papers with incomplete or inaccessible data.

Literature Screening Results

After data collection through Google Scholar and Pubmed, 31 articles were obtained that can be used as the basis for this literature review (Table 1). Each article discusses related designs that can be used to identify inhibitory pathways of different plants in treating depression and anxiety. Depression and anxiety are common mental disorders, and herbal treatment is a safer and more efficient alternative. By identifying inhibitory pathways of different plants, the review aims to provide a comprehensive overview of phytopharmacotherapy for these disorders. The review aims to minimize side effects and optimize effectiveness, making it a safer and more effective treatment option.

Epidemiology

Based on WHO data, one in eight people, or approximately 970 million individuals worldwide, experience mental disorders that contribute to global mortality and disability, with numbers continuing to increase each year. Among these, 280 million people suffer from depression and 301 million from anxiety (2). The Indonesian Ministry of Health reported that the prevalence of depression and anxiety in Indonesia in 2023 was 1.4%, with the highest rates among young people (15–24 years old) at 2%. According to the Indonesia National Adolescent Mental Health Survey (2022), around 2.45 million adolescents (5.5%) were found to experience mental disorders (1).

Countries with high incomes and advanced economies tend to have higher stress levels but, conversely, lower happiness percentages. This is attributed to demanding lifestyles that increase the prevalence of depression and anxiety, particularly among individuals engaged in activities such as gambling or online gaming (3). An unhealthy lifestyle, characterized by low physical activity and the frequent consumption of processed foods high in sodium, can elevate blood pressure, triggering stress and anxiety. Similarly, high sugar intake can cause elevated blood glucose levels that contribute to anxiety. Such lifestyle factors catalyze psychological responses that manifest as mood and affect disturbances, ultimately leading to depression and anxiety (4).

Depression and anxiety remain the most commonly experienced mental disorders worldwide, with high prevalence rates that contribute to increased global mortality risk. Individuals affected by these conditions have been shown to face nearly double the risk of death compared to those without mental disorders, placing depression and anxiety among the top ten causes of mortality after cardiovascular diseases and suicide (31).

Depression

Depression, or Major Depressive Disorder (MDD), is the most common mental health condition globally, affecting an estimated 320 million people. It is characterized by persistent mood disturbances accompanied by feelings of worthlessness, anhedonia, social withdrawal, sleep disorders, and suicidal ideation. The pathophysiology of MDD is multifactorial, involving dysregulation of monoamine neurotransmitters (serotonin, norepinephrine, and dopamine), dysfunction in GABAergic signaling, reduced neurogenesis and synaptic plasticity, and increased neurodegeneration. A crucial contributing factor is the stress-induced hyperactivity of the hypothalamic–pituitary–adrenal (HPA) axis, where prolonged elevation of glucocorticoids diminishes synaptic plasticity and promotes neuronal death in the hippocampus. This process leads to synaptic structural alterations and disruption of HPA axis negative feedback regulation. In this context, phytocomponents from various herbal plants have shown promising potential to address the multifactorial nature of depression through multiple and complementary mechanisms (32, 33).

Several herbal plants directly target neurotransmitter imbalances and enhance neuroplasticity to counteract glucocorticoid-induced neurotoxicity. Lemon balm (Melissa officinalis), through its active compound rosmarinic acid, effectively inhibits monoamine oxidase (MAO) and potentially modulates GABA-T receptors, while protecting neurons from oxidative stress and apoptosis, thereby mitigating glucocorticoid-related cellular damage (19). The sensitive plant (Mimosa pudica L.), rich in quercetin, also inhibits MAO and modulates GABA-A receptors, while its saponins activate CREB to enhance BDNF expression and neurogenesis, offsetting the suppression of neuronal growth and synaptic plasticity caused by glucocorticoids (21). Likewise, α-terpineol from mandarin orange (Citrus reticulata) demonstrates antidepressant potential through high affinity for dopamine D2 receptors (DRD2), modulating the dopaminergic system disrupted in depression (24). Asiatic pennywort (Centella asiatica L.), via asiaticoside, increases serotonin (5-HT) levels and stimulates CREB phosphorylation to induce BDNF expression, effectively reversing neurogenesis decline (7). Bay leaf (Laurus nobilis L.), through its flavonoid content, enhances acetylcholine levels via acetylcholinesterase (AChE) inhibition while providing antioxidant neuroprotection (10).

Other herbs act through the regulation of the HPA axis and suppression of neuroinflammation. Banana (Musa sapientum L.), due to its tryptophan content, lowers cortisol levels and stabilizes HPA axis function, addressing the hormonal dysregulation fundamental to depression (12). Lavender (Lavandula angustifolia), rich in linalool, demonstrates a targeted mechanism by reversing corticosterone-induced suppression of hippocampal neurogenesis, directly mitigating one of the core pathogenic effects of glucocorticoids (15). Clerodendrum serratum L., through luteolin, inhibits HPA axis hyperactivity by reducing corticotropin-releasing hormone (CRH) expression and adrenocorticotropic hormone (ACTH) secretion, providing a direct hormonal regulatory effect (27). Similarly, brown algae (Saccharina japonica), through dieckol, modulates the TLR4/NLRP3 signaling pathway to suppress neuroinflammation, oxidative stress, and apoptosis, thus maintaining neuronal homeostasis under glucocorticoid-induced stress (29, 30). In addition, hibiscus (Hibiscus sabdariffa), rich in flavonoids, exhibits anti-inflammatory and neuroprotective effects by directly scavenging reactive oxygen species and enhancing endogenous antioxidant enzymes such as superoxide dismutase (SOD) and catalase, promoting overall neuroprotection and calming effects on the central nervous system (13).

Anxiety

Anxiety, commonly manifesting as Generalized Anxiety Disorder (GAD), is a neurocircuitry dysfunction characterized by an imbalance between inhibitory neurotransmitters such as GABA and excitatory neurotransmitters like glutamate. This neurochemical imbalance is further aggravated by elevated levels of pro-inflammatory cytokines, leading to neuronal disruption and impaired synaptic plasticity. Within this complex interplay of neurochemical and neuroinflammatory mechanisms, numerous herbal plants exhibit therapeutic potential by directly modulating these critical pathways to relieve anxiety symptoms (34, 35).

Several phytocompounds exert anxiolytic effects through modulation of the GABAergic system. Sunflower (Helianthus annuus L.), through its active constituents Heliannone A, B, C, and triterpenoids, modulates GABA-A receptors by enhancing chloride (Cl⁻) ion permeability and reducing neuronal hyperactivity, a mechanism analogous to that of benzodiazepines (25). Similarly, the sensitive plant (Mimosa pudica L.), rich in quercetin, also modulates GABA-A receptors, thereby strengthening GABAergic neurotransmission and producing calming effects (22). Lemon balm (Melissa officinalis), via rosmarinic acid, inhibits GABA-T receptors, increasing synaptic GABA availability and promoting relaxation (20).

Other herbal agents mitigate anxiety by targeting neuroinflammation and oxidative stress. Brown algae (Saccharina japonica), through the polyphenol dieckol, regulates the TLR4/NLRP3 signaling pathway to suppress neuroinflammation, oxidative stress, and apoptosis, effectively counteracting the pro-inflammatory cytokine cascade associated with anxiety (29, 30). Hibiscus (Hibiscus sabdariffa), rich in flavonoids, exerts dual antioxidant actions—directly neutralizing reactive oxygen species and enhancing endogenous antioxidant enzymes such as superoxide dismutase (SOD) and catalase—thus reducing inflammation and promoting central nervous system (CNS) calmness (13). Bay leaf (Laurus nobilis L.), through its flavonoids and phenolic acids, provides similar neuroprotection by reducing oxidative damage to brain cells (10).

Additional herbal species demonstrate diverse anxiolytic mechanisms. Bay leaf also contributes to anxiolysis through the inhibition of acetylcholinesterase (AChE), thereby elevating acetylcholine levels and improving neurotransmission (11). Lavender (Lavandula angustifolia), containing linalool, promotes neurogenesis and dendritic branching, reversing stress-induced suppression of neuroplasticity and restoring healthy neuronal connectivity (15). Chamomile (Chamaemelum nobile), with apigenin, modulates neurotransmitters, suppresses inflammation, and protects against oxidative stress (9). Rose (Rosa damascena), rich in flavonoids and citronellol, provides anxiolytic and sedative effects via inhaled essential oils that influence emotional centers in the brain and enhance GABAergic inhibition of excessive neural activity (17). Peppermint (Mentha piperita L.) contains menthol and menthone, where menthol acts as a GABA-A receptor agonist to inhibit neuronal overactivity while its antinociceptive properties contribute to pain relief (16).

Herbal Medicine as an Alternative Therapy for Treating Depression and Anxiety

Focusing on neurotransmitter system modulation, which fundamentally governs the brain’s chemical balance, various herbal plants demonstrate complex mechanisms that target neurotransmitter degradation, receptor modulation, and reuptake processes (see Figure 2). Inhibiting monoamine oxidase (MAO) prevents the breakdown of monoamine neurotransmitters and is a primary mechanism of several herbal compounds. Lemon balm (Melissa officinalis), through its key constituent rosmarinic acid (RA), has been shown to cross the blood–brain barrier and directly inhibit MAO enzyme activity. This action, validated in vivo by Ghazizadeh et al. (2020), significantly elevates serotonin and norepinephrine levels in the brain, thereby contributing to antidepressant and anxiolytic effects (19). Similarly, the sensitive plant (Mimosa pudica L.), containing quercetin, effectively inhibits both MAO-A and MAO-B, preventing the degradation of serotonin, dopamine, and norepinephrine (21, 22). Clerodendrum serratum L., rich in luteolin, exhibits a dual mechanism by reducing MAO-A activity while simultaneously inhibiting monoamine transporters, including SERT, NET, and DAT. This dual action may offer superior efficacy in increasing neurotransmitter availability compared with conventional antidepressants such as SSRIs, SNRIs, or MAOIs, which generally act on a single target (27).

In the GABAergic modulation pathway—central to neuronal inhibition in the CNS—sunflower (Helianthus annuus L.) exhibits notable anxiolytic activity. Its bioactive compounds, Heliannone A–C and triterpenoids, modulate GABA-A receptors, enhancing chloride (Cl⁻) ion influx and inducing neuronal hyperpolarization. This reduces excitability and prevents excessive neuronal firing, a mechanism analogous to benzodiazepines (25, 26). Mimosa pudica also modulates GABA-A receptors via quercetin, enhancing synaptic inhibition and producing anxiolytic effects consistent with increased GABAergic activity observed by Mambou et al. (2025) (22). Likewise, lemon balm inhibitsGABA-T through its ursolic and oleanolic acids, augmenting synaptic GABA levels, a mechanism supported by clinical trials by Bano et al. (2023) (20). Rose (Rosa damascena), containing citronellol, facilitates GABA-mediated inhibition of excessive neuronal activity, exerting a calming effect primarily through inhalation that stimulates olfactory–limbic pathways (17, 18). Peppermint (Mentha piperita L.) also demonstrates GABA-A receptor agonism through menthol, promoting neuronal hyperpolarization and central nervous system relaxation (16).

Several plants also influence neurotransmitter reuptake mechanisms. Sunflower compounds, particularly Heliannone, inhibit serotonin and dopamine reuptake, increasing synaptic concentrations and restoring neurotransmission similar to conventional antidepressants (25). Mimosa pudica, through mimosine, inhibits serotonin reuptake via SERT, mirroring the action of SSRIs (21). Bay leaf (Laurus nobilis L.) shows potent cholinergic modulation through acetylcholinesterase (AChE) inhibition by its flavonoids, elevating acetylcholine levels to improve mood and cognitive function. Studies in mouse models have shown significant behavioral improvements at oral doses of 100–200 mg/kg without observable toxicity (10, 11).

Regulation of the hypothalamic–pituitary–adrenal (HPA) axis and enhancement of neuroplasticity are also vital mechanisms. Bananas (Musa sapientum L.), through tryptophan, contribute to lowering cortisol levels and stabilizing the HPA axis, supporting stress adaptation and serotonin synthesis. Although Soeliono et al. (2024) found limited anxiolytic effects in animal models, tryptophan supplementation reduced anxiety-related behaviors through serotonergic modulation (12). In mandarin orange (Citrus reticulata), Tang et al. (2022) demonstrated that extract (CREO) administration significantly reduced immobility in forced swim and tail suspension tests, suggesting antidepressant and neuroprotective activity associated with upregulated glucocorticoid receptor (GR), BDNF, and 5-HT1A expression (36). Similarly, Clerodendrum serratum L. (luteolin) suppresses HPA hyperactivity by reducing hypothalamic CRH and pituitary ACTH, directly limiting cortisol overproduction—an advantage over antidepressants that do not target HPA dysregulation (27).

Enhancing neuroplasticity and neurogenesis via the CREB/BDNF pathway also plays a key role. Lavender (Lavandula angustifolia), with its active compound linalool (LEO), reverses corticosterone-induced suppression of hippocampal neurogenesis, restores neuronal growth, and elevates BDNF and oxytocin levels—yielding antidepressant and anxiolytic effects without direct interference in classical monoamine pathways (15, 37). Asiatic pennywort (Centella asiatica L.), through asiaticoside, activates the cAMP/PKA pathway, inhibits NF-κB and NLRP3 inflammasome activation, reduces neuroinflammation, and increases CREB phosphorylation and BDNF expression (7, 38). Chamaemelum nobile (Roman chamomile), via apigenin, upregulates BDNF and neurotrophic factors, reducing oxidative stress and hormonal imbalance (9). Mimosa pudica saponins activate the CREB pathway, enhancing BDNF-mediated neuroplasticity and long-term antidepressant effects (21, 22). Similarly, Clerodendrum serratum (luteolin) enhances synaptophysin and PSD-95 expression via PI3K/AKT/mTOR signaling, mimicking the synaptic strengthening effects of ketamine while avoiding its side effects (27).

Anti-inflammatory, antioxidant, and neuroprotective mechanisms further reinforce the therapeutic profile of these herbs. Brown algae (Saccharina japonica), through dieckol, modulates TLR4/NLRP3 signaling to inhibit microglial polarization toward the M1 phenotype, reducing neuroinflammation and depression-related pathology (29). Centella asiatica also inhibits NF-κB and NLRP3 activation, reducing cytokines such as IL-1β, IL-6, and TNF-α (7). Chamomile’s apigenin inhibits GSK-3β and activates Nrf2, suppressing pro-inflammatory cytokines and providing neuroprotection (9). Mandarin orange compounds linalool and terpinene-4-ol suppress COX-2 and PGE2, further mitigating inflammation (24). Brown algae extract also reduces oxidative stress and apoptosis, protecting cortical neurons (29). Lemon balm’s RA similarly prevents mitochondrial-mediated apoptosis (19), while bay leaf flavonoids and phenolic acids combat oxidative brain damage (10, 11).

Banana peel extract, rich in polyphenols, flavonoids, and carotenoids, has shown significant antioxidant and neuroprotective effects. Its active enzymes—SOD, catalase, glutathione, and MDA reducers—enhance neuronal resilience and reduce oxidative stress, supporting cognition and mood improvement (39). Chamaemelum nobile (apigenin, luteolin, and rutin) and Ginkgo biloba (terpenoids) further mitigate oxidative stress, protect mitochondria, and prevent ROS formation (9, 40). Mandarin orange (linalool, p-cymene, terpinene-4-ol) also reduces apoptosis and mitochondrial dysfunction, reinforcing its neuroprotective potential (24). Rose (Rosa damascena), with flavonoid-rich essential oils, mitigates oxidative brain stress linked to mood disorders (17, 18). Clerodendrum serratum (luteolin) increases Bcl-2 and decreases caspase-3 activity, offering selective neuroprotection via anti-apoptotic pathways (28).

Other molecular pathways also contribute to antidepressant and anxiolytic efficacy. Chamaemelum nobile (apigenin) targets multiple proteins, including LRRK2, as identified by Jia et al. (2020) through network pharmacology and bioinformatics, indicating roles beyond classical neurotransmission (8). Citrus reticulata, per Khanh et al. (2024), interacts with six key targets—OPRM1, PTGS2, ESR1, SLC6A4, DRD2, and NR3C1—highlighting multi-targeted antidepressant mechanisms (24). Ginkgo biloba flavonoids antagonize platelet-activating factor receptors and chloride channels, preventing ROS formation and optimizing antioxidant expression (40). Clinically, Ginkgo biloba reduces antidepressant dosage requirements and side effects, showing promise as adjunct therapy, particularly for elderly depression and mild anxiety, though further clinical validation is required. Finally, Mentha piperita (peppermint) exhibits antinociceptive and anxiolytic effects comparable to SSRIs in mild neurological conditions, though clinical evidence remains limited (41).

Conclusion

From an in-depth comparative analysis of various herbal plants and their roles in modulating neurotransmitter systems, regulating the hypothalamic–pituitary–adrenal (HPA) axis, enhancing neuroplasticity, and exerting anti-inflammatory, antioxidant, and neuroprotective effects, it is evident that phytotherapy provides a promising multi-target approach for managing depression and anxiety. Plants such as Melissa officinalis (lemon balm), Mimosa pudica, Senna alata, Helianthus annuus (sunflower), Rosa damascena (rose), Laurus nobilis (bay leaf), Mentha piperita (peppermint), Citrus reticulata (mandarin orange), Centella asiatica (Asiatic pennywort), Chamaemelum nobile (Roman chamomile), and Ginkgo biloba demonstrate significant therapeutic potential through diverse mechanisms. These include inhibition of monoamine oxidase (MAO) enzymes, modulation of GABAergic receptors, suppression of neurotransmitter reuptake, regulation of hormonal balance via the HPA axis, enhancement of neurogenesis and synaptic plasticity, and protection against oxidative and inflammatory damage. Collectively, these findings highlight the potential of herbal-based therapy as an effective, safer, and more sustainable complementary or alternative strategy for the prevention and management of depression and anxiety.