NLP Mental Health
A cross-domain NLP study evaluating how depression detection models transfer to suicidal ideation content on social media.
Project overview
This research investigates whether a model trained to detect depression in Twitter text can generalize to a related but linguistically different domain: suicidal ideation.
Three representations—TF-IDF, RoBERTa, and MentalBERT—were tested across six traditional classifiers under the same cross-domain conditions to isolate the impact of feature quality.
The research question
Depression and suicidal ideation are clinically related, but the language people use to describe them can differ considerably. Models that perform well within one dataset may therefore fail when applied to another condition.
The central question was whether contextual transformer embeddings could preserve meaning across this topical domain shift better than sparse lexical features.
Experimental workflow
The workflow keeps training and out-of-domain evaluation strictly separated, allowing transfer performance to be measured without retraining on suicidal-intention labels.
Datasets
Both datasets contain Twitter content, keeping the platform consistent while changing the mental-health condition being detected.
MDDL Depression Dataset
- Size
- 257,178 tweets
- Labels
- Depressed / non-depressed
Twitter Suicidal Intention Dataset
- Size
- 9,119 posts
- Labels
- Suicidal / non-suicidal
Vocabulary shift
Only 20 of the top 50 words were shared between the two datasets. Depression content was dominated by clinical language, while the suicidal-intention dataset contained more direct, action-oriented distress expressions.
This 60% non-overlap explains why models depending on exact word frequency struggled to transfer, even though both datasets came from the same social platform.
disorder · diagnosed · bipolar · PTSD
die · kill · suicide · want · can’t
Cross-domain results
Contextual embeddings produced a dramatic improvement over TF-IDF. KNN showed the clearest reversal, moving from the worst result with TF-IDF to the strongest result with RoBERTa.
Cross-domain F1 score (%)
| Classifier | TF-IDF F1 | RoBERTa F1 | MentalBERT F1 |
|---|---|---|---|
| Naive Bayes | 0.2028 | 0.4347 | 0.4845 |
| Random Forest | 0.0998 | 0.3517 | 0.3660 |
| Decision Tree | 0.0964 | 0.3974 | 0.3703 |
| SVM (Linear) | 0.0146 | 0.4842 | 0.2764 |
| Logistic Regression | 0.0119 | 0.5105 | 0.4249 |
| KNN | 0.0009 | 0.6447 | 0.6280 |
The KNN reversal
With TF-IDF, KNN achieved an F1 score of only 0.0009. Sparse lexical vectors placed linguistically different but semantically related tweets far apart.
With RoBERTa embeddings, KNN reached the best cross-domain F1 score of 0.6447 and recall of 0.5429. Dense contextual representations made nearest neighbours genuinely meaningful.
Key findings
Representation dominates
Feature quality affected transfer performance more strongly than classifier choice.
TF-IDF does not generalize
Vocabulary mismatch caused F1 drops of up to 99.7% under domain shift.
RoBERTa wins
General-purpose RoBERTa outperformed MentalBERT across every evaluated classifier.
Limitations and next steps
The transformers were used as frozen feature extractors, and the best recall remained approximately 0.54. The current system is a research prototype and must not be treated as a deployment-ready mental-health screening tool.
Future work includes end-to-end fine-tuning, domain adaptation, ensemble approaches, and stronger recall-focused evaluation while preserving ethical safeguards.