Net promoter score (NPS) is a tool often used by companies to gauge the loyalty of their customers and how the customers rate the customers service.
When NPS surveys are conducted, it is common to also ask for written feedback from the customers, in the form of a short comment.
I was recently reviewing a set of NPS survey data, which contained both NPS scores and comments, and thought it would be interesting to see how sentiment analysis of the comments tracked with the NPS scores. Presumably, the high the NPS score, the more positive the sentiment would be.
All of the analysis was done in Python in a Jupyter notebook which I have shares here.
The NPS data comes in the form of a CSV and its easily imported using Pandas.
A typical snapshot of the data is shown below.
The first sentiment analysis method I tried was to simply use the AFINN data set to determine the sentiment of each comment. The AFINN data set is a list of words which with sentiment scores between -5 and + 5.
There is a python library called 'afinn' that utilises this data set and provides some simple methods. With the NPS data in a dataframe, it easy to apply the afinn methods to all the comments to calculate a sentiment score for each comment.
There are ore 30,000 records in this NPS data set so to provide an overall view of the sentiment compared to the NPS score, we can use the pandas crosstab method.
And we can visualise this using violin plots.
The left plot show the distribution of the sentiment for each comment with a certain NPS score.
The right plot shows similar information but the size of the 'violin' is scaled by the number of comments in each NPS bucket. It is clear from this that most NPS scores are eight and above.
Looking further at these charts, a few things are apparent:
- there are a lot of comments with a zero sentiment
- there are a lot more comments for high NPS scores, than for lower ones.
- there doesn't appear to be much ofd a trend in increasing sentiment with increasing NPS
A weighted mean will remove the impact of the zero sentiment comments and make the more negative or more positive a comment is have a larger overall impact on sentiment and we can normalise it by the comment counts to get a trend of sentiment
We can see from this that there is a general trend in improving sentiment as NPS scores increase, which is what you would expect.
As an alternative to the AFINN library, we can instead use the python natural language tool kit (NLTK) to help with tokenising text etc, and apply the AFINN data directly.
Using NLTK to tokenise the text, we get slightly different sentiment scores. NLTK separates punctuation from words, which the AFINN library doesn't, which leads to some different scores.
We can see from the chart on the right that the trend is similar in shape to that fond previously, but a little flatter.
We can also use NLTK to tokenise at the sentence level, in case comments have multiple sentences, and then find the average sentiment per sentence and use that to calculate an average comment sentiment.
This gives very similar results to work tokenising and, when we look at the distribution of sentences per comment we can see why. Over 85% of comments contain either zero or one sentences.
Next, we can try using a naive bayes classifier to classify the comments as positive or negative, and then work out the trend in positivity versus NPS.
NLTK contains a naive bayes classifier and, for this situation, we'll try training the classifier using the Movie review data set. These reviews aren't completely relevant to to NPS comments but I think there should be enough correlation in positive words between them that the classifier will be reasonable.
To train the classifier we:
- read in the movies reviews
- tag them as positive or negative
- split them into lists of words in positive or negative groups.
The Naive bayes classifier then calculates the probability distribution of each word.
Splitting the movie data in to a training and test set, the classifier achieves and accuracy of 65% which isn't awesome but we'll continue regardless.
Now that we have a trained classifier, we can apply it to all the comments in our NPS data frame. And, relying once again on the Pandas crosstab method, we get the following distribution of positive and negative sentiment by NPS score.
Plotting this as a ratio of positive to negative sentiment we get this trend which is similar to those before.
For the various types of sentiment analysis applied a trend is apparent with sentiment generally increasing as NPS score increases.
In a future post i'll try some other more advances methods to see how they compare.