Paper, Tags: #sarcasm-detection
Sarcasm depends on context, but most computational approaches treat it as a purely linguistic matter, using information such as lexical cues and their corresponding sentiment as predictive features. We show that by including extra-linguistic information from the context of an utterance on Twitter, we're able to achieve gains in accuracy compared to purely linguistic features in the detection of this complex phenomenon.
We present a series of experiments to discern the effect of extra-linguistic information on the detection of sarcasm, reasoning about features derived not only from the local context of the message itself, but also using information about the author, relationship to audience, and immediate communicative context they both share.
We found that including any aspect of the environment leads to improvements in prediction accuracy, and that users are more likely to tag their message with #sarcasm when they're less familiar with their audience.
We extract tweets containing #sarcasm or #sarcastic from 2013/08-2014/07, and capturing 9.7k tweets in total for positive sarcasm. For negative data, we select an equal number of tweets from user over the same time period who haven't mentioned #sarcasm or #sarcastic in their messages. The total dataset is balanced at 19.5k tweets.
We adopt binary logistic regression with L2 regularization using tenfold cross-validation, split on authors.
4 classes of features:
- those scoped only over the immediate tweet being predicted
- those that reason over the author of that tweet
- those that reason over the addressee of the tweet
- those that consider the interaction between the tweet being predicted and the tweet that it's responding to
The baseline accuracy, using only the majority class in each training fold, is 47.4%.
- Word unigrams (72.4%) and bigrams (69.5%)
- Brown cluster unigrams (72.0%) bigrams (69.1%)
- Unlabeled dependency bigrams, lexicalized (70.3%) and Brown clusters (70.2%)
- Part of speech features (66.0%)
- Pronunciation features (57.5%)
- Capitalization features (57.5%)
- Tweet whole sentiment (55.0%)
- Tweet word sentiment (53.7–54.7%)
- Intensifiers (50.1%)
- Author historical salient terms (81.2%)
- Author historical topics (77.4%)
- Profile information (73.7%)
- Author historical sentiment (70.8%)
- Profile unigrams (66.2%)
- Author historical topics (71.2%), Author historical salient terms (70.0%), Profile unigrams (68.6%), Profile information (66.3%)
- Author/Addressee interactional topics (73.9%)
- Historical communication between author and addressee (61.7%)
- Pairwise Brown features between the original message and the response (71.7%)
- Unigram features of the original message (68.8%)
We consider 5 feature combinations: those with access only to tweet-level information:
- tweet features + response features
- tweet features + audience features
- tweet features + author features
Tweet-only information yields an average accuracy of 75.4% across all ten folds, adding response features pushes this to 77.3%, audience features to 79% and author features to 84.9%. Including all features together yields 85.1%.
All feature classes (response, audience and author) display statistically significant improvements over the tweet-only features that ignore the communicative context. This confirms an effect on the interaction of the author and audience in the recognition of sarcasm, which leads us to ask, who is this audience and what about them is predictive of sarcasm across users?
#sarcasm hashtag is not a natural indicator of sarcasm expressed between friends, but rather serves an important communicative function of signaling the author’s intent to an audience who may not otherwise be able to draw the correct inference about their message (as distinct from close friends who may be able to infer sarcasm without such labels).
Studying sarcasm that does rely on common ground, and doesn't require such explicit markers, will likely need to rely on other forms of supervision.