Social and News Sentiment
Text analysis turns news and social media into sentiment signals. Learn how sentiment is measured, from word lists to language models, the evidence and the pitfalls.
Every day, millions of news articles, social media posts and forum comments discuss companies and markets. Sentiment analysis uses computers to read this text and measure whether it is positive, negative or neutral. Traders use sentiment signals to anticipate price moves, detect shifts in attention and spot crowded trades. The tools range from simple word counts to large language models, and the evidence suggests sentiment can add information, though signals are noisy and decay quickly.
Sources of text#
| Source | Characteristics |
|---|---|
| Professional news | Edited, timely, widely read. See News Feeds |
| Social media (X, Reddit, StockTwits) | Very fast, noisy, retail heavy |
| Earnings call transcripts | Management tone and analyst questions. See Earnings Calls |
| Regulatory filings | Formal language; changes between filings matter |
| Analyst reports | Professional opinions |
| Forums and chat groups | Retail enthusiasm and coordination |
How sentiment is measured#
| Method | How it works | Strengths and weaknesses |
|---|---|---|
| Dictionary (word lists) | Count positive and negative words | Simple, transparent; misses context |
| Finance specific dictionaries | Word lists built for financial text | More accurate in finance |
| Machine learning classifiers | Train models on labelled text | Better context; needs training data |
| Language models | Large models score sentiment or extract meaning | Powerful; costly and can be inconsistent |
Researchers Tim Loughran and Bill McDonald showed in 2011 that general purpose word lists misclassify many financial terms: words like "liability", "tax" or "crude" are negative in everyday dictionaries but neutral in finance. Their finance specific word lists became a standard tool. See Machine Learning in Trading.
The evidence#
- Paul Tetlock (2007) found that high pessimism in a Wall Street Journal column predicted downward pressure on stock prices, followed by a reversal.
- Studies of firm specific news found that negative words in news stories predicted lower earnings and returns.
- Social media studies have found some predictive power for short term returns and volume, but results vary and often weaken after publication.
Building a sentiment signal#
- Collect text with accurate timestamps.
- Link text to companies (entity recognition).
- Score sentiment with an appropriate method.
- Aggregate by company and time, weighting by relevance and source.
- Measure change relative to normal levels.
- Test predictive power out of sample, including costs. See In-Sample vs Out-of-Sample Testing.
Pitfalls#
| Pitfall | Explanation |
|---|---|
| Sarcasm and slang | Social posts often mean the opposite of their words |
| Bots and spam | Fake accounts can manipulate sentiment |
| Pump and dump schemes | Coordinated promotion of small stocks. See Market Manipulation |
| Look ahead bias | Using text not available at the time. See Look-Ahead Bias |
| Fast decay | Sentiment signals often matter for hours or days |
| Crowding | Popular sentiment signals are widely traded |
Frequently asked questions#
What is sentiment analysis in trading?#
Using computer methods to score news, social media and other text as positive, negative or neutral, and using those scores as trading signals.
Does social media sentiment predict stock prices?#
Studies have found some short term predictive power, but signals are noisy, can be manipulated and often fade quickly.
What is the Loughran McDonald dictionary?#
A finance specific list of positive and negative words developed by researchers Tim Loughran and Bill McDonald, widely used to measure sentiment in financial text.
Next, learn how weather drives markets in Weather and Agricultural Data.
3 quick questions on this lesson. Get them all right to finish it.
Turn on JavaScript to take the quiz.
Mentioned in
- Alternative Data ExplainedData and Alternative Data
- Web Traffic, App Downloads and Search TrendsData and Alternative Data
- Job Postings DataData and Alternative Data
- Neural Networks and Deep LearningMachine Learning