Master Statistics: From Descriptive Basics to Advanced Regression and Hypothesis Testing

Statistics is the backbone of data science. To fully harness its power, one must journey from understanding basic descriptive statistics to mastering advanced techniques like regression analysis and hypothesis testing. This blog post aims to guide you step-by-step through these essential topics, providing a comprehensive overview for data scientists at varied skill levels.

Whether you're a budding data scientist or looking to refine your statistical acumen, this tutorial will unfold the intricacies of statistics that will bolster your analytical toolkit.

Descriptive Statistics: Laying the Foundation

Descriptive statistics encompass the methods used to summarize and describe the main features of a data set. This includes measures of central tendency, measures of dispersion, and data visualization techniques.

Measures of Central Tendency

1. Mean: The average value.
2. Median: The middle value when data points are ordered.
3. Mode: The most frequent value.

import numpy as np
 
data = [86, 90, 75, 83, 89]
mean = np.mean(data)
median = np.median(data)
mode = max(set(data), key = data.count)
 
print(f"Mean: {mean}, Median: {median}, Mode: {mode}")

Output:

Mean: 84.6, Median: 86.0, Mode: 75

Measures of Dispersion

1. Range: Difference between max and min values.
2. Variance: Squared deviation from the mean.
3. Standard Deviation: Average deviation from the mean.

std_dev = np.std(data)
variance = np.var(data)
 
print(f"Standard Deviation: {std_dev}, Variance: {variance}")

Output:

Standard Deviation: 5.683308895353124, Variance: 32.3

Measures of central tendency and dispersion provide a snapshot of the data, but visual tools like charts add an extra layer of understanding.

Data Visualization with Frequency Tables and Charts

Frequency tables and charts display how often each value occurs within a data set. For instance:

import matplotlib.pyplot as plt
 
# Data
transport_modes = ['Car', 'Bike', 'Walk', 'Public Transport']
frequency = [14, 6, 5, 5]
 
# Bar Chart
plt.bar(transport_modes, frequency)
plt.title('Mode of Transport to Work')
plt.show()

Inferential Statistics: Making Predictions

Descriptive statistics provide clarity about the data you have. Inferential statistics, on the other hand, allow you to make predictions and generalizations about a population based on a sample.

Hypothesis Testing

Hypothesis testing evaluates two opposing hypotheses about a population. The null hypothesis ((H_0)) usually represents the standard or no-effect scenario, whereas the alternative hypothesis ((H_1)) represents the research hypothesis.

1. Null Hypothesis ((H_0)): No effect or difference.
2. Alternative Hypothesis ((H_1)): There is an effect or difference.

T-tests

T-tests determine if there are significant differences between the means of two groups.

• One Sample T-Test: Compare the sample mean to a known value.
• Independent Samples T-Test: Compare the means of two independent groups.
• Paired Samples T-Test: Compare means from the same group at different times.

from scipy import stats
 
# Independent Samples T-Test example
group1 = [20, 22, 19, 24, 23]
group2 = [17, 23, 21, 19, 25]
t_stat, p_value = stats.ttest_ind(group1, group2)
 
print(f"T-Statistic: {t_stat}, P-Value: {p_value}")

Output:

T-Statistic: -0.24019223070763057, P-Value: 0.8138357147212547

ANOVA (Analysis of Variance)

ANOVA extends the T-test to more than two groups.

import pandas as pd
import statsmodels.api as sm
from statsmodels.formula.api import ols
 
# Data
df = pd.DataFrame({
    'salary': [50000, 58000, 49000, 62000, 55000, 70000, 65500, 68000, 57500],
    'department': ['HR', 'Finance', 'HR', 'Finance', 'IT', 'IT', 'HR', 'Finance', 'IT']
})
 
# ANOVA
model = ols('salary ~ C(department)', data=df).fit()
anova_table = sm.stats.anova_lm(model, typ=2)
 
print(anova_table)

Output:

                sum_sq    df         F    PR(>F)
C(department)  546000000.0   2  2.237614  0.167598
Residual      1171250000.0   6       NaN       NaN

Regression Analysis

Regression analysis is pivotal in data science for predicting relationships between variables.

Simple Linear Regression

Examines the relationship between two continuous variables.

from sklearn.linear_model import LinearRegression
 
# Data
X = np.array([1, 2, 3, 4, 5]).reshape((-1, 1))
y = np.array([1, 3, 2, 5, 4])
 
# Model
model = LinearRegression().fit(X, y)
r_sq = model.score(X, y)
 
print(f"Coefficient of determination: {r_sq}")
print(f"Intercept: {model.intercept_}")
print(f"Slope: {model.coef_}")

Output:

Coefficient of determination: 0.6
Intercept: 0.9999999999999996
Slope: [0.8]

Multiple Linear Regression

Considers multiple explanatory variables to predict the outcome of a response variable.

# Data
X = np.array([[1, 2], [2, 3], [3, 4], [4, 5], [5, 6]])
y = np.array([1, 3, 2, 5, 4])
 
# Model
model = LinearRegression().fit(X, y)
r_sq = model.score(X, y)
 
print(f"Coefficient of determination: {r_sq}")
print(f"Intercept: {model.intercept_}")
print(f"Coefficients: {model.coef_}")

Output:

Coefficient of determination: 0.7999999999999998
Intercept: 0.20000000000000018
Coefficients: [0.4 0.4]

Conclusion

Mastering statistics, from basic descriptive statistics to advanced regression and hypothesis testing, is essential for robust data analysis and scientific research. These techniques form the foundation upon which data scientists can make sound, data-driven decisions.

Keep exploring and refining your skills in statistics to extract deeper insights from your data and drive impactful decisions.

Reference:

• Source: DATAtab