Every time someone leaves a review on Amazon, posts a complaint on Twitter, or rates a restaurant on Zomato, they are generating data that contains something far more valuable than the words themselves. They are expressing a feeling. Positive, negative, frustrated, delighted, somewhere in between.
The ability to automatically detect and classify these feelings at scale is called sentiment analysis, and it sits at the center of some of the most commercially valuable applications of natural language processing in use today. Companies spend millions trying to understand how customers feel about their products. Political campaigns monitor public opinion in real time. Customer support teams use it to prioritize angry messages before they escalate.
This guide walks through building a sentiment analysis tool in Python from the ground up. Not a wrapper around someone else’s API. Not a three line scikit-learn tutorial that glosses over everything important. A real implementation where you understand every step and why it is there.
What Sentiment Analysis Actually Is
Sentiment analysis is a natural language processing task that involves classifying text according to the emotional tone it expresses. At its simplest, this means determining whether a piece of text expresses a positive, negative, or neutral sentiment. More sophisticated implementations classify along multiple dimensions, detect specific emotions like anger or sadness, or identify the sentiment expressed toward specific aspects of a product or service rather than the text as a whole.
The core challenge is that human language is deeply contextual and often ambiguous. The sentence this phone is sick means something completely different in a product review written today versus one written twenty years ago. The phrase not bad at all expresses positive sentiment despite containing two negative words. Sarcasm, irony, and understatement create patterns that are easy for a human to decode and difficult for a model to learn.
Understanding these challenges from the beginning shapes how you approach building a sentiment analysis system, because different approaches handle these challenges to different degrees, and the right choice depends on what the system actually needs to do.
The Different Approaches to Sentiment Analysis
Before writing any code, it helps to understand the main approaches available and when each is appropriate.
Lexicon based approaches work by looking up words in a pre-built dictionary that assigns sentiment scores to individual words and then combining those scores to produce an overall sentiment for the text. Tools like VADER, which was specifically designed for social media text, use this approach. Lexicon based methods require no training data, work well when the vocabulary is relatively predictable, and are fast and interpretable. They struggle with context dependent language and sarcasm.
Machine learning approaches train a classifier on labeled examples of text and learn statistical patterns that distinguish different sentiment classes. These approaches require labeled training data but can learn patterns that lexicon based methods miss, including domain specific language and contextual signals.
Deep learning approaches use neural network architectures that can capture much more complex patterns in text, including long range dependencies and subtle contextual signals. Transformer based models like BERT have produced state of the art results on sentiment analysis benchmarks and are increasingly practical to use through fine tuning.
For a from scratch implementation that demonstrates genuine understanding of the full pipeline, a machine learning approach using classical methods is the right starting point, both because it is more transparent than a deep learning approach and because it forces engagement with every step of the process rather than handing off complexity to a pre-built model.
Setting Up the Environment
The following libraries are needed for this implementation:
pip install pandas numpy scikit-learn nltk matplotlib seaborn wordcloudOnce installed, the standard imports for this project are:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
import nltk
import re
import string
nltk.download("stopwords")
nltk.download("punkt")
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenizeUnderstanding and Loading the Data
For this implementation, we will use the IMDB movie review dataset, which contains 50,000 reviews labeled as positive or negative. This dataset is large enough to train a meaningful model, small enough to work with on a standard laptop, and representative enough of real world text to produce useful results.
import pandas as pd
df = pd.read_csv("IMDB_Dataset.csv")
print(df.shape)
print(df.head())
print(df["sentiment"].value_counts())
print(df.isnull().sum())The value counts check confirms whether the dataset is balanced between positive and negative examples. An imbalanced dataset, where one class significantly outnumbers the other, requires different handling than a balanced one, and identifying this early prevents misleading evaluation results later.
Exploratory Data Analysis
Before building any model, understanding the data through visualization reveals patterns that shape every subsequent decision.
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
df["sentiment"].value_counts().plot(kind="bar", ax=axes[0],
color=["steelblue", "coral"],
edgecolor="black")
axes[0].set_title("Sentiment Distribution", fontsize=14)
axes[0].set_xlabel("Sentiment")
axes[0].set_ylabel("Count")
axes[0].set_xticklabels(["Negative", "Positive"], rotation=0)
df["review_length"] = df["review"].apply(len)
sns.histplot(data=df, x="review_length", hue="sentiment",
bins=50, alpha=0.6, ax=axes[1])
axes[1].set_title("Review Length Distribution by Sentiment", fontsize=14)
axes[1].set_xlabel("Review Length (characters)")
axes[1].set_xlim(0, 5000)
plt.tight_layout()
plt.show()
print("Average review length by sentiment:")
print(df.groupby("sentiment")["review_length"].mean())The length distribution plot often reveals that negative reviews tend to be longer than positive ones, a pattern that occurs across many review datasets because dissatisfied customers tend to be more verbose. This kind of exploratory insight, discovered through visualization, sometimes suggests useful features that a basic bag of words model would miss.
A complete guide on data visualization with Matplotlib and Seaborn that covers the visualization techniques used throughout this project is available here: https://www.tuxacademy.org/matplotlib-seaborn-data-visualization-python/
Text Preprocessing
Raw text from real sources contains a significant amount of noise that reduces model performance if left unaddressed. HTML tags, punctuation, numbers, and common words that carry no sentiment information all need to be handled before the text is converted into features.
def preprocess_text(text):
text = re.sub(r"<[^>]+>", " ", text)
text = text.lower()
text = re.sub(r"[^a-zA-Z\s]", "", text)
tokens = word_tokenize(text)
stop_words = set(stopwords.words("english"))
sentiment_stopwords = {"not", "no", "never", "nor", "neither",
"barely", "hardly", "scarcely"}
stop_words = stop_words - sentiment_stopwords
tokens = [token for token in tokens if token not in stop_words]
tokens = [token for token in tokens if len(token) > 2]
return " ".join(tokens)
df["cleaned_review"] = df["review"].apply(preprocess_text)
print("Original review:")
print(df["review"].iloc[0][:200])
print("\nCleaned review:")
print(df["cleaned_review"].iloc[0][:200])The most important decision in this preprocessing function is the handling of negation words. Standard stopword removal would eliminate words like not, no, and never, which carry critical sentiment information. The phrase not good means something completely opposite to good, and removing not before training would make these two phrases indistinguishable to the model. Explicitly preserving negation words in the stopword removal step is one of those small decisions that has a meaningful impact on model performance.
Feature Extraction With TF-IDF
Machine learning models cannot work directly with text. The text needs to be converted into numerical features, and the standard approach for classical machine learning on text is TF-IDF, which stands for Term Frequency-Inverse Document Frequency.
TF-IDF represents each document as a vector of numbers, where each number represents how important a particular word is to that document relative to the entire corpus. A word that appears frequently in a single document but rarely across the corpus gets a high score, because it is likely to be informative about that specific document. A word that appears frequently everywhere, even after stopword removal, gets a lower score.
df["label"] = (df["sentiment"] == "positive").astype(int)
X = df["cleaned_review"]
y = df["label"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training samples: {len(X_train)}")
print(f"Testing samples: {len(X_test)}")
vectorizer = TfidfVectorizer(
max_features=50000,
ngram_range=(1, 2),
min_df=2,
max_df=0.95,
sublinear_tf=True
)
X_train_tfidf = vectorizer.fit_transform(X_train)
X_test_tfidf = vectorizer.transform(X_test)
print(f"Feature matrix shape: {X_train_tfidf.shape}")The ngram_range parameter is worth understanding specifically. Setting it to (1, 2) means the vectorizer considers both individual words and pairs of consecutive words as features. This allows the model to learn that not good is a different signal from good alone, which directly addresses one of the most common failure modes of unigram only models.
The fit_transform call on the training data and the transform only call on the test data is a critical distinction. The vectorizer learns its vocabulary and IDF weights from the training data only. Applying it to the test data without refitting ensures that the test set remains a genuine measure of how the model performs on unseen data.
Training and Comparing Multiple Models
Rather than committing to a single model architecture, training several candidates and comparing their performance is the approach that produces the best results and the most informed model selection decision.
models = {
"Logistic Regression": LogisticRegression(
max_iter=1000, C=1.0, random_state=42
),
"Naive Bayes": MultinomialNB(alpha=0.1),
"Linear SVM": LinearSVC(C=1.0, random_state=42, max_iter=2000)
}
results = {}
for name, model in models.items():
model.fit(X_train_tfidf, y_train)
y_pred = model.predict(X_test_tfidf)
accuracy = accuracy_score(y_test, y_pred)
results[name] = {
"model": model,
"predictions": y_pred,
"accuracy": accuracy
}
print(f"{name}: {accuracy:.4f}")
best_model_name = max(results, key=lambda x: results[x]["accuracy"])
best_model = results[best_model_name]["model"]
print(f"\nBest model: {best_model_name}")Each of these three models has different strengths. Logistic Regression is interpretable and works well with high dimensional sparse data like TF-IDF vectors. Naive Bayes is extremely fast to train and often performs surprisingly well on text classification despite its strong independence assumption. Linear SVM typically achieves the highest accuracy on text classification tasks because it finds the optimal boundary between classes in high dimensional space.
Evaluating the Model Properly
Accuracy alone is not a sufficient evaluation metric, especially when the goal is to understand where the model is making mistakes and why.
best_predictions = results[best_model_name]["predictions"]
print(f"Detailed Classification Report for {best_model_name}:")
print(classification_report(y_test, best_predictions,
target_names=["Negative", "Positive"]))
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
cm = confusion_matrix(y_test, best_predictions)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues",
xticklabels=["Negative", "Positive"],
yticklabels=["Negative", "Positive"], ax=axes[0])
axes[0].set_title(f"Confusion Matrix: {best_model_name}")
axes[0].set_ylabel("Actual")
axes[0].set_xlabel("Predicted")
model_names = list(results.keys())
accuracies = [results[name]["accuracy"] for name in model_names]
colors = ["steelblue" if name != best_model_name else "coral"
for name in model_names]
axes[1].bar(model_names, accuracies, color=colors, edgecolor="black")
axes[1].set_ylim(0.8, 1.0)
axes[1].set_title("Model Accuracy Comparison")
axes[1].set_ylabel("Accuracy")
for i, acc in enumerate(accuracies):
axes[1].text(i, acc + 0.002, f"{acc:.3f}", ha="center", fontsize=11)
plt.tight_layout()
plt.show()The confusion matrix reveals four specific categories of predictions: true positives, true negatives, false positives, and false negatives. Understanding which types of errors the model is making most frequently is essential for deciding whether the model is suitable for its intended use and what improvements might address the most important failure modes.
Understanding What the Model Has Learned
One of the most valuable things you can do after training a text classification model is examine which words most strongly predict each class. For a logistic regression model, this is straightforward through the model coefficients:
if best_model_name == "Logistic Regression":
feature_names = vectorizer.get_feature_names_out()
coefficients = best_model.coef_[0]
top_positive_idx = np.argsort(coefficients)[-20:]
top_negative_idx = np.argsort(coefficients)[:20]
fig, axes = plt.subplots(1, 2, figsize=(16, 6))
top_positive_words = [feature_names[i] for i in top_positive_idx]
top_positive_scores = [coefficients[i] for i in top_positive_idx]
axes[0].barh(top_positive_words, top_positive_scores, color="steelblue")
axes[0].set_title("Top 20 Words Predicting Positive Sentiment")
axes[0].set_xlabel("Coefficient Value")
top_negative_words = [feature_names[i] for i in top_negative_idx]
top_negative_scores = [coefficients[i] for i in top_negative_idx]
axes[1].barh(top_negative_words, top_negative_scores, color="coral")
axes[1].set_title("Top 20 Words Predicting Negative Sentiment")
axes[1].set_xlabel("Coefficient Value")
plt.tight_layout()
plt.show()This visualization often reveals both expected patterns and surprising ones. Words like excellent, outstanding, and brilliant predictably drive positive predictions. But examining which bigrams the model has learned to associate with negative sentiment sometimes reveals domain specific patterns that would not be obvious without looking at the feature weights directly.
Building a Prediction Function
The trained model and vectorizer need to be wrapped in a clean function that handles all the preprocessing steps automatically and returns a prediction in a human readable format:
def predict_sentiment(text, model, vectorizer, threshold=0.5):
cleaned = preprocess_text(text)
features = vectorizer.transform([cleaned])
if hasattr(model, "predict_proba"):
probability = model.predict_proba(features)[0]
prediction = 1 if probability[1] >= threshold else 0
confidence = probability[1] if prediction == 1 else probability[0]
else:
prediction = model.predict(features)[0]
confidence = None
sentiment = "Positive" if prediction == 1 else "Negative"
result = {
"text": text[:100] + "..." if len(text) > 100 else text,
"sentiment": sentiment,
"confidence": f"{confidence:.2%}" if confidence else "N/A"
}
return result
test_reviews = [
"This product is absolutely amazing. Best purchase I have ever made.",
"Terrible quality. Stopped working after two days. Complete waste of money.",
"It is okay. Nothing special but does what it is supposed to do.",
"Not bad at all for the price. Would recommend to a friend.",
"I wanted to love this but it was just disappointing in every way."
]
print("Sentiment Predictions:")
print("-" * 60)
for review in test_reviews:
result = predict_sentiment(review, best_model, vectorizer)
print(f"Text: {result['text']}")
print(f"Sentiment: {result['sentiment']} (Confidence: {result['confidence']})")
print("-" * 60)The third and fourth test examples are intentionally chosen to test edge cases. The phrase okay and nothing special is mildly negative in context but might not be classified as clearly negative by a model trained on more extreme examples. Not bad at all is positive in meaning but contains two negative words. How the model handles these cases reveals its actual capabilities better than testing it on obviously positive or negative text.
Saving and Loading the Model
A trained model that cannot be saved and reloaded is not practical for any real application. Python’s joblib library handles this efficiently for scikit-learn models:
import joblib
joblib.dump(best_model, "sentiment_model.pkl")
joblib.dump(vectorizer, "tfidf_vectorizer.pkl")
print("Model and vectorizer saved successfully.")
loaded_model = joblib.load("sentiment_model.pkl")
loaded_vectorizer = joblib.load("tfidf_vectorizer.pkl")
test_result = predict_sentiment(
"This is a genuinely excellent product.",
loaded_model,
loaded_vectorizer
)
print(f"Loaded model prediction: {test_result['sentiment']}")Saving both the model and the vectorizer separately, and confirming that the loaded versions produce the same predictions as the originals, is important because the vectorizer contains the vocabulary and IDF weights learned from the training data, which are just as essential as the model coefficients for making predictions on new text.
Deploying as a Simple Web Application
A model that runs only in a notebook has limited practical value. Converting it into a simple API that can receive text and return predictions makes it genuinely usable:
from flask import Flask, request, jsonify
import joblib
app = Flask(__name__)
model = joblib.load("sentiment_model.pkl")
vectorizer = joblib.load("tfidf_vectorizer.pkl")
@app.route("/predict", methods=["POST"])
def predict():
data = request.get_json()
if not data or "text" not in data:
return jsonify({"error": "No text provided"}), 400
text = data["text"]
if not text.strip():
return jsonify({"error": "Empty text provided"}), 400
result = predict_sentiment(text, model, vectorizer)
return jsonify(result)
@app.route("/health", methods=["GET"])
def health():
return jsonify({"status": "healthy", "model": "sentiment_classifier"})
if __name__ == "__main__":
app.run(debug=True, port=5000)This Flask application exposes two endpoints. The predict endpoint accepts a POST request containing text and returns the sentiment prediction. The health endpoint provides a simple check that the service is running, which is standard practice for any deployed service. Hosting this on a free platform like Render or Railway produces a live URL that can be included in a portfolio and demonstrated directly in an interview.
Common Mistakes and How to Avoid Them
Fitting the vectorizer on the entire dataset before splitting into train and test sets is one of the most common data leakage mistakes in NLP projects. When the vectorizer is fitted on all available data, IDF weights are influenced by the test set, which means the model has indirectly seen information from the test set during training. Always fit the vectorizer on training data only.
Removing all negation words during preprocessing, as most default stopword lists do, destroys critical sentiment information. The decision to preserve words like not, no, and never in the preprocessing step is one of the most important domain specific adjustments in sentiment analysis.
Evaluating only on accuracy when class distributions are imbalanced produces misleadingly optimistic results. A model that predicts the majority class for every input can achieve high accuracy on an imbalanced dataset while being completely useless. Always examine precision, recall, and F1 score for each class separately.
Using the same dataset that comes pre-split into training and test sets without understanding how the split was created can introduce subtle biases, particularly if the split was done non-randomly in a way that affects generalization.
Extending This Project
The implementation covered here is a complete, working sentiment analysis pipeline. Several extensions would make it significantly more powerful and more impressive in a portfolio context.
Aspect based sentiment analysis goes beyond overall document sentiment to identify the sentiment expressed toward specific aspects of a product or service. A review that says the camera is excellent but the battery life is terrible expresses positive sentiment about one aspect and negative sentiment about another, which an overall sentiment classifier would miss.
Fine tuning a pre trained transformer model like BERT on this same dataset would produce higher accuracy, particularly on the edge cases where the classical approach struggles. Comparing the classical and transformer approaches and documenting the tradeoffs is itself a strong project contribution.
Building a real time dashboard that monitors sentiment in a stream of social media posts or customer reviews and displays trends over time creates a genuinely useful tool rather than a demonstration.
A complete guide on fine tuning pre trained AI models that is directly applicable to the transformer based extension of this project is available here: https://www.tuxacademy.org/how-to-fine-tune-pre-trained-ai-model/
Frequently Asked Questions
What accuracy should I expect from a sentiment analysis model?
On the IMDB dataset with a well implemented TF-IDF and linear classifier pipeline, accuracy in the range of 88 to 92 percent is achievable. Transformer based models can reach 94 to 96 percent on the same dataset. The gap between classical and deep learning approaches is real but smaller than many students assume, and for many practical applications the classical approach is sufficient and significantly simpler to deploy.
Can this approach be applied to languages other than English?
Yes, with modifications. The main changes required are using a tokenizer and stopword list appropriate for the target language, sourcing training data in that language, and potentially handling transliteration if the data contains romanized text. The underlying pipeline structure remains the same.
How much training data is needed for a good sentiment analysis model?
The IMDB dataset with 50,000 examples is more than sufficient for the classical approach. For domain specific applications, meaningful models have been trained on as few as a few thousand labeled examples when the domain is consistent and the labeling is high quality.
Is VADER better than training a custom model?
VADER is faster to deploy and requires no training data, making it a good starting point for quick prototyping or for applications where the text domain is similar to social media. A trained custom model typically outperforms VADER when sufficient domain specific training data is available, particularly for text that differs significantly from social media in style or vocabulary.
How do I handle sarcasm in sentiment analysis?
Sarcasm remains one of the hardest problems in sentiment analysis and is not reliably solved by any current approach. Transformer models handle it better than classical approaches due to their ability to capture longer range context, but no model handles it perfectly. Documenting this limitation honestly in a project is more appropriate than claiming to have solved it.
Comparison Table: Sentiment Analysis Approaches
Approach, Training Data Required, Implementation Complexity, Typical Accuracy, Best Use Case
VADER Lexicon, None, Very Low, 70 to 80 percent, Social media, quick prototyping
Naive Bayes plus TF-IDF, Thousands of examples, Low, 82 to 87 percent, Baseline model, fast training
Logistic Regression plus TF-IDF, Thousands of examples, Low to Medium, 88 to 91 percent, Production classical model
Linear SVM plus TF-IDF, Thousands of examples, Medium, 89 to 92 percent, High accuracy classical model
Fine Tuned BERT, Hundreds to thousands, High, 93 to 96 percent, Maximum accuracy requirements
Why This Project Stands Out in a Portfolio
Sentiment analysis is one of the most commercially deployed NLP applications in the world, which means demonstrating genuine competence with it communicates directly relevant capability to any recruiter at a company that works with text data.
What makes this implementation specifically strong as a portfolio piece is the depth of the pipeline: real preprocessing decisions with documented reasoning, multiple model comparison with proper evaluation, feature importance visualization that demonstrates interpretability awareness, a clean prediction function, model persistence, and a deployed API endpoint. Each of these components represents a skill that has direct value in a real data science role.
A complete guide on the AI project portfolio strategies that hiring managers actually respond to, including how to document and present technical projects effectively, is available here: https://www.tuxacademy.org/ai-projects-final-year-students-impress-recruiters/
Call to Action
Start building data science and NLP skills that employers actually hire for.
If you want structured guidance on building real machine learning projects with industry experienced trainers who have worked with production NLP systems, TuxAcademy offers data science and AI courses built entirely around practical, portfolio worthy project work with placement support.
Website: https://www.tuxacademy.org/
Course: https://www.tuxacademy.org/data-science-course-in-noida-complete-guide/
Email: info@tuxacademy.org
Phone: +91-7982029314
Visit the nearest center or book a free counseling session today.
Our Location
Students searching for a data science or NLP course near Noida Sector 18 or machine learning training in Noida will find TuxAcademy directly accessible from across the NCR region.
TuxAcademy is easily accessible from Noida Sector 18, Noida Sector 22, Noida Sector 27, Noida Sector 34, Noida Sector 44, Noida Sector 50, Noida Sector 51, Noida Sector 52, Noida Sector 58, Noida Sector 62, Vaishali, Vasundhara, Indirapuram, and Sahibabad, making it a convenient choice for data science students across the Noida and Ghaziabad belt.
The institute is also easily reachable from Noida City Centre Metro Station, Sector 52 Metro Station, Amity University Noida, NIET Noida, and the Noida Greater Noida Expressway, with strong connectivity for students coming from across Delhi NCR.
TuxAcademy is a preferred destination for students seeking practical, job oriented training in Data Science, Natural Language Processing, Python Programming, Artificial Intelligence, and Full Stack Development across Noida and NCR.