📈 Statistical Analysis: Finding Patterns in Data

Python

Statistics

Hypothesis Testing

Correlation

Discover meaningful patterns, test hypotheses, and understand statistical relationships using real data

Welcome to Statistical Analysis! 📊

Statistics is the foundation of data science - it helps us understand data, test hypotheses, and make informed decisions based on evidence. Let’s explore key statistical concepts with hands-on examples!

🎯 What You’ll Learn

Descriptive statistics (mean, median, standard deviation)
Data distributions and visualization
Correlation analysis
Hypothesis testing
Confidence intervals
Statistical significance

The Scenario: Analyzing Student Performance

You’re a data analyst for a university studying factors that affect student success. You have data on study hours, sleep, exercise, and exam scores for 100 students. Let’s uncover the patterns!

Part 0: Setting Up Dependencies

# Install required packages for data analytics

import piplite
await piplite.install(['seaborn', 'matplotlib', 'pandas', 'numpy', 'scipy', 'plotly'])
print("Packages installed successfully!")
print("You can now import and use: seaborn, matplotlib, pandas, numpy, scipy, plotly")

# Test imports

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats as stats

print("All packages imported successfully!")
print(f"NumPy version: {np.__version__}")
print(f"Pandas version: {pd.__version__}")
print(f"Matplotlib version: {plt.matplotlib.__version__}")
print(f"Seaborn version: {sns.__version__}")

Part 1: Creating the Dataset

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
import plotly.graph_objects as go
from scipy import stats

# Set random seed for reproducibility
np.random.seed(42)

# Generate realistic student data
n_students = 100

# Study hours per week (with some correlation to scores)
study_hours = np.random.normal(20, 8, n_students)
study_hours = np.clip(study_hours, 5, 45)  # Keep realistic range

# Sleep hours per night
sleep_hours = np.random.normal(7, 1.5, n_students)
sleep_hours = np.clip(sleep_hours, 4, 10)

# Exercise hours per week
exercise_hours = np.random.normal(4, 2, n_students)
exercise_hours = np.clip(exercise_hours, 0, 12)

# Exam scores (influenced by study, sleep, and exercise)
exam_scores = (50 + 
               study_hours * 1.2 +           # Study helps!
               sleep_hours * 2 +             # Sleep is important
               exercise_hours * 1.5 +        # Exercise helps too
               np.random.normal(0, 8, n_students))  # Random variation
exam_scores = np.clip(exam_scores, 40, 100)

# Class attendance (percentage)
attendance = np.random.normal(85, 12, n_students)
attendance = np.clip(attendance, 50, 100)

# Create DataFrame
df = pd.DataFrame({
    'student_id': range(1, n_students + 1),
    'study_hours': study_hours,
    'sleep_hours': sleep_hours,
    'exercise_hours': exercise_hours,
    'attendance_pct': attendance,
    'exam_score': exam_scores
})

print("🎓 Student Performance Dataset")
print("="*80)
print(df.head(10))
print(f"\n📊 Dataset: {len(df)} students with {len(df.columns)-1} variables")

Dataset Overview

We’re tracking 100 students with 5 variables: - Study Hours/Week: Time spent studying - Sleep Hours/Night: Average sleep duration - Exercise Hours/Week: Physical activity - Attendance %: Class participation - Exam Score: Performance (0-100)

Our goal: Understand what factors predict academic success!

Part 2: Descriptive Statistics

Let’s start with basic statistical summaries:

# Calculate comprehensive statistics
print("📊 Descriptive Statistics")
print("="*80)
print(df.describe().round(2))

# Calculate additional statistics
print("\n📈 Additional Measures:")
print(f"Study Hours - Median: {df['study_hours'].median():.2f}, "
      f"Mode: {df['study_hours'].mode().values[0]:.2f}")
print(f"Exam Scores - Range: {df['exam_score'].min():.1f} to {df['exam_score'].max():.1f}")
print(f"              Variance: {df['exam_score'].var():.2f}")
print(f"              Std Dev: {df['exam_score'].std():.2f}")

# Percentiles
print("\n📊 Exam Score Percentiles:")
for percentile in [25, 50, 75, 90]:
    value = np.percentile(df['exam_score'], percentile)
    print(f"  {percentile}th percentile: {value:.1f}")

💡 Understanding Statistics

Mean: Average value (affected by outliers)
Median: Middle value (robust to outliers)
Std Dev: How spread out the data is
Percentiles: Value below which X% of data falls
- 50th percentile = median
- 25th/75th = quartiles (interquartile range)

Part 3: Visualizing Distributions

Let’s see how our variables are distributed:

# Create distribution plots
fig, axes = plt.subplots(2, 3, figsize=(15, 10))

variables = ['study_hours', 'sleep_hours', 'exercise_hours', 
             'attendance_pct', 'exam_score']
colors = ['#FF6B6B', '#40C9D9', '#6BCB77', '#FFD93D', '#FF7B7C']
titles = ['Study Hours/Week', 'Sleep Hours/Night', 'Exercise Hours/Week',
          'Attendance %', 'Exam Score']

for idx, (var, color, title) in enumerate(zip(variables, colors, titles)):
    row = idx // 3
    col = idx % 3
    ax = axes[row, col]
    
    # Histogram with KDE (kernel density estimate)
    ax.hist(df[var], bins=20, color=color, alpha=0.6, edgecolor='black', density=True)
    
    # Add KDE curve
    df[var].plot(kind='kde', ax=ax, color='darkblue', linewidth=2)
    
    # Add mean line
    mean_val = df[var].mean()
    ax.axvline(mean_val, color='red', linestyle='--', linewidth=2, label=f'Mean: {mean_val:.1f}')
    
    ax.set_title(f'Distribution of {title}', fontweight='bold', fontsize=11)
    ax.set_xlabel(title)
    ax.set_ylabel('Density')
    ax.legend()
    ax.grid(alpha=0.3)

# Remove empty subplot
axes[1, 2].remove()

plt.suptitle('📊 Distribution Analysis of Student Variables', 
             fontsize=16, fontweight='bold', y=0.995)
plt.tight_layout()
plt.show()

🎯 Distribution Insights

Normal (bell-shaped): Most students cluster around average
Skewed: Asymmetric distribution (could indicate issues)
Outliers: Extreme values that stand out
Multimodal: Multiple peaks (different student groups?)

The distributions help us understand typical values and variability!

Part 4: Correlation Analysis

Which variables are related to each other?

# Calculate correlation matrix
correlation_matrix = df[['study_hours', 'sleep_hours', 'exercise_hours', 
                          'attendance_pct', 'exam_score']].corr()

# Create beautiful heatmap
fig, ax = plt.subplots(figsize=(10, 8))

sns.heatmap(correlation_matrix, 
            annot=True, 
            fmt='.3f', 
            cmap='RdYlGn',
            center=0,
            square=True,
            linewidths=2,
            cbar_kws={'label': 'Correlation Coefficient'},
            vmin=-1, vmax=1,
            ax=ax)

plt.title('🔥 Correlation Heatmap: Understanding Relationships', 
          fontsize=14, fontweight='bold', pad=20)
plt.tight_layout()
plt.show()

# Display strongest correlations with exam score
print("\n🎯 Correlations with Exam Score:")
print("="*60)
score_corr = correlation_matrix['exam_score'].sort_values(ascending=False)
for var, corr in score_corr.items():
    if var != 'exam_score':
        strength = 'Strong' if abs(corr) > 0.5 else 'Moderate' if abs(corr) > 0.3 else 'Weak'
        print(f"  {var:20s}: {corr:6.3f} ({strength})")

💡 Interpreting Correlations

+1.0: Perfect positive correlation (both increase together)
0.0: No correlation (no relationship)
-1.0: Perfect negative correlation (one increases, other decreases)
> 0.5: Strong relationship
0.3-0.5: Moderate relationship
< 0.3: Weak relationship

Important: Correlation ≠ Causation! Just because variables are correlated doesn’t mean one causes the other.

Part 5: Scatter Plots with Regression Lines

Let’s visualize the strongest relationships:

# Study hours vs exam scores with regression line
fig = px.scatter(df, 
                 x='study_hours', 
                 y='exam_score',
                 trendline='ols',  # Ordinary Least Squares regression
                 title='📚 Study Hours vs Exam Performance',
                 labels={'study_hours': 'Study Hours per Week',
                         'exam_score': 'Exam Score'},
                 color_discrete_sequence=['#FF6B6B'])

fig.update_traces(marker=dict(size=8, line=dict(width=1, color='black')))
fig.update_layout(
    font=dict(size=12),
    height=500,
    template='plotly_white'
)

fig.show()

# Calculate regression statistics
from scipy.stats import linregress
slope, intercept, r_value, p_value, std_err = linregress(df['study_hours'], df['exam_score'])

print(f"\n📈 Regression Analysis: Study Hours → Exam Score")
print("="*60)
print(f"  Equation: Score = {intercept:.2f} + {slope:.2f} × Study Hours")
print(f"  R-squared: {r_value**2:.3f} ({r_value**2*100:.1f}% of variance explained)")
print(f"  P-value: {p_value:.6f} {'(Statistically significant!)' if p_value < 0.05 else ''}")
print(f"\n  Interpretation: Each additional study hour predicts a {slope:.2f} point increase in exam score!")

Part 6: Hypothesis Testing

Let’s test a hypothesis: Do students who study more than 20 hours/week score significantly higher?

# Split students into two groups
high_study = df[df['study_hours'] > 20]['exam_score']
low_study = df[df['study_hours'] <= 20]['exam_score']

print(f"👥 Group Comparison")
print("="*80)
print(f"High study group (>20 hrs/week): {len(high_study)} students")
print(f"  Mean score: {high_study.mean():.2f}")
print(f"  Std dev: {high_study.std():.2f}")
print(f"\nLow study group (≤20 hrs/week): {len(low_study)} students")
print(f"  Mean score: {low_study.mean():.2f}")
print(f"  Std dev: {low_study.std():.2f}")
print(f"\nDifference in means: {high_study.mean() - low_study.mean():.2f} points")

# Perform independent t-test
t_statistic, p_value = stats.ttest_ind(high_study, low_study)

print(f"\n📊 Statistical Test Results:")
print("="*80)
print(f"  T-statistic: {t_statistic:.3f}")
print(f"  P-value: {p_value:.6f}")
print(f"  Significance level: α = 0.05")

if p_value < 0.05:
    print(f"\n  ✅ Result: STATISTICALLY SIGNIFICANT!")
    print(f"  Conclusion: Students who study >20 hrs/week score significantly higher.")
else:
    print(f"\n  ❌ Result: NOT statistically significant")
    print(f"  Conclusion: The difference could be due to random chance.")

# Visualize group comparison
fig = go.Figure()

fig.add_trace(go.Box(
    y=high_study,
    name='High Study (>20 hrs)',
    marker_color='#6BCB77',
    boxmean='sd'
))

fig.add_trace(go.Box(
    y=low_study,
    name='Low Study (≤20 hrs)',
    marker_color='#FF6B6B',
    boxmean='sd'
))

fig.update_layout(
    title='📦 Exam Score Distribution by Study Group',
    yaxis_title='Exam Score',
    template='plotly_white',
    height=500,
    font=dict(size=12)
)

fig.show()

🎓 Understanding Hypothesis Testing

Null Hypothesis (H₀): There’s no difference between groups
Alternative Hypothesis (H₁): There is a difference

P-value: Probability of observing this difference by chance - If p < 0.05: Reject null hypothesis (difference is real!) - If p ≥ 0.05: Fail to reject null (might be random chance)

This is how scientists prove their findings!

Part 7: Confidence Intervals

What’s the true average exam score for all students?

# Calculate 95% confidence interval
mean_score = df['exam_score'].mean()
std_error = stats.sem(df['exam_score'])  # Standard error of mean
confidence_level = 0.95
degrees_freedom = len(df) - 1

# Calculate confidence interval
confidence_interval = stats.t.interval(confidence_level, 
                                       degrees_freedom,
                                       loc=mean_score,
                                       scale=std_error)

print(f"📊 95% Confidence Interval for Mean Exam Score")
print("="*80)
print(f"  Sample mean: {mean_score:.2f}")
print(f"  Standard error: {std_error:.2f}")
print(f"  95% Confidence Interval: [{confidence_interval[0]:.2f}, {confidence_interval[1]:.2f}]")
print(f"\n  Interpretation: We are 95% confident that the true population mean")
print(f"  exam score falls between {confidence_interval[0]:.2f} and {confidence_interval[1]:.2f}.")

# Visualize confidence interval
fig = go.Figure()

# Add mean point
fig.add_trace(go.Scatter(
    x=[mean_score],
    y=['Mean'],
    mode='markers',
    marker=dict(size=15, color='#FF6B6B'),
    name='Sample Mean',
    showlegend=True
))

# Add confidence interval
fig.add_trace(go.Scatter(
    x=[confidence_interval[0], confidence_interval[1]],
    y=['Mean', 'Mean'],
    mode='lines+markers',
    marker=dict(size=10, color='#40C9D9'),
    line=dict(width=4, color='#40C9D9'),
    name='95% CI',
    showlegend=True
))

fig.update_layout(
    title='🎯 95% Confidence Interval for Mean Exam Score',
    xaxis_title='Exam Score',
    template='plotly_white',
    height=400,
    font=dict(size=12),
    yaxis=dict(showticklabels=False)
)

fig.show()

Part 8: Multiple Regression Analysis

Can we predict exam scores using multiple variables?

from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_squared_error

# Prepare data
X = df[['study_hours', 'sleep_hours', 'exercise_hours', 'attendance_pct']]
y = df['exam_score']

# Fit model
model = LinearRegression()
model.fit(X, y)
predictions = model.predict(X)

# Display results
print(f"📊 Multiple Regression: Predicting Exam Scores")
print("="*80)
print(f"\nRegression Equation:")
print(f"  Score = {model.intercept_:.2f}")
for feature, coef in zip(X.columns, model.coef_):
    sign = '+' if coef >= 0 else '-'
    print(f"          {sign} {abs(coef):.2f} × {feature}")

print(f"\n📈 Model Performance:")
print(f"  R-squared: {r2_score(y, predictions):.3f}")
print(f"  RMSE: {np.sqrt(mean_squared_error(y, predictions)):.2f} points")

print(f"\n💡 Feature Importance (sorted by absolute coefficient):")
feature_importance = pd.DataFrame({
    'Feature': X.columns,
    'Coefficient': model.coef_,
    'Abs_Coefficient': np.abs(model.coef_)
}).sort_values('Abs_Coefficient', ascending=False)

for idx, row in feature_importance.iterrows():
    print(f"  {row['Feature']:20s}: {row['Coefficient']:6.2f} (impact per unit)")

🎯 Key Findings

Based on our statistical analysis:

Study hours matter: Strong positive correlation with exam scores
Sleep is important: Better sleep → better performance
Exercise helps: Physical activity correlates with academic success
Attendance counts: Class participation predicts higher scores
Multiple factors: Best predictions use all variables together

This is how data science informs educational policy!

🎓 What You’ve Learned

Congratulations! You’ve mastered fundamental statistical concepts:

✅ Descriptive statistics (mean, median, std dev, percentiles)
✅ Distribution analysis and visualization
✅ Correlation analysis to find relationships
✅ Hypothesis testing with t-tests
✅ Confidence intervals for estimation
✅ Multiple regression for prediction

🚀 What’s Next in CMPSC 301?

In the course, you’ll learn advanced statistics: - ANOVA: Comparing multiple groups - Chi-square tests: Analyzing categorical data - Non-parametric tests: When data isn’t normal - Time series analysis: Temporal patterns - Bayesian statistics: Probabilistic reasoning - Experimental design: A/B testing and causality

💡 Try It Yourself!

Experiment with these challenges:

Add more variables: GPA, major, socioeconomic factors
Test other hypotheses: Do exercise and sleep interact?
Outlier analysis: Find and investigate unusual students
Predictive modeling: Build a model to predict struggling students
Real data: Analyze actual student performance data

Ready to become a statistical wizard? Visit JupyterLite!

🌍 Real-World Applications

Statistical analysis is used everywhere: - Medicine: Clinical trials, drug effectiveness - Business: A/B testing, market research - Sports: Player performance, team strategies - Policy: Economic indicators, social programs - Science: Research validation, discoveries

Master these tools in CMPSC 301 and unlock countless opportunities!

--- title: "📈 Statistical Analysis: Finding Patterns in Data" description: "Discover meaningful patterns, test hypotheses, and understand statistical relationships using real data" categories: [Python, Statistics, Hypothesis Testing, Correlation] execute: eval: false echo: true format: html: code-tools: true code-fold: false --- ## Welcome to Statistical Analysis! 📊 Statistics is the foundation of data science - it helps us understand data, test hypotheses, and make informed decisions based on evidence. Let's explore key statistical concepts with hands-on examples! ### 🎯 What You'll Learn - Descriptive statistics (mean, median, standard deviation) - Data distributions and visualization - Correlation analysis - Hypothesis testing - Confidence intervals - Statistical significance --- ## The Scenario: Analyzing Student Performance You're a data analyst for a university studying factors that affect student success. You have data on study hours, sleep, exercise, and exam scores for 100 students. Let's uncover the patterns! --- ## Part 0: Setting Up Dependencies ```python # Install required packages for data analytics import piplite await piplite.install(['seaborn', 'matplotlib', 'pandas', 'numpy', 'scipy', 'plotly']) print("Packages installed successfully!") print("You can now import and use: seaborn, matplotlib, pandas, numpy, scipy, plotly") # Test imports import numpy as np import pandas as pd import matplotlib.pyplot as plt import seaborn as sns import scipy.stats as stats print("All packages imported successfully!") print(f"NumPy version: {np.__version__}") print(f"Pandas version: {pd.__version__}") print(f"Matplotlib version: {plt.matplotlib.__version__}") print(f"Seaborn version: {sns.__version__}") ``` ## Part 1: Creating the Dataset ```{python} import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns import plotly.express as px import plotly.graph_objects as go from scipy import stats # Set random seed for reproducibility np.random.seed(42) # Generate realistic student data n_students = 100 # Study hours per week (with some correlation to scores) study_hours = np.random.normal(20, 8, n_students) study_hours = np.clip(study_hours, 5, 45) # Keep realistic range # Sleep hours per night sleep_hours = np.random.normal(7, 1.5, n_students) sleep_hours = np.clip(sleep_hours, 4, 10) # Exercise hours per week exercise_hours = np.random.normal(4, 2, n_students) exercise_hours = np.clip(exercise_hours, 0, 12) # Exam scores (influenced by study, sleep, and exercise) exam_scores = (50 + study_hours * 1.2 + # Study helps! sleep_hours * 2 + # Sleep is important exercise_hours * 1.5 + # Exercise helps too np.random.normal(0, 8, n_students)) # Random variation exam_scores = np.clip(exam_scores, 40, 100) # Class attendance (percentage) attendance = np.random.normal(85, 12, n_students) attendance = np.clip(attendance, 50, 100) # Create DataFrame df = pd.DataFrame({ 'student_id': range(1, n_students + 1), 'study_hours': study_hours, 'sleep_hours': sleep_hours, 'exercise_hours': exercise_hours, 'attendance_pct': attendance, 'exam_score': exam_scores }) print("🎓 Student Performance Dataset") print("="*80) print(df.head(10)) print(f"\n📊 Dataset: {len(df)} students with {len(df.columns)-1} variables") ``` :::{.callout-note} ## Dataset Overview We're tracking 100 students with 5 variables: - **Study Hours/Week**: Time spent studying - **Sleep Hours/Night**: Average sleep duration - **Exercise Hours/Week**: Physical activity - **Attendance %**: Class participation - **Exam Score**: Performance (0-100) Our goal: Understand what factors predict academic success! ::: --- ## Part 2: Descriptive Statistics Let's start with basic statistical summaries: ```{python} # Calculate comprehensive statistics print("📊 Descriptive Statistics") print("="*80) print(df.describe().round(2)) # Calculate additional statistics print("\n📈 Additional Measures:") print(f"Study Hours - Median: {df['study_hours'].median():.2f}, " f"Mode: {df['study_hours'].mode().values[0]:.2f}") print(f"Exam Scores - Range: {df['exam_score'].min():.1f} to {df['exam_score'].max():.1f}") print(f" Variance: {df['exam_score'].var():.2f}") print(f" Std Dev: {df['exam_score'].std():.2f}") # Percentiles print("\n📊 Exam Score Percentiles:") for percentile in [25, 50, 75, 90]: value = np.percentile(df['exam_score'], percentile) print(f" {percentile}th percentile: {value:.1f}") ``` :::{.callout-tip} ## 💡 Understanding Statistics - **Mean**: Average value (affected by outliers) - **Median**: Middle value (robust to outliers) - **Std Dev**: How spread out the data is - **Percentiles**: Value below which X% of data falls - 50th percentile = median - 25th/75th = quartiles (interquartile range) ::: --- ## Part 3: Visualizing Distributions Let's see how our variables are distributed: ```{python} # Create distribution plots fig, axes = plt.subplots(2, 3, figsize=(15, 10)) variables = ['study_hours', 'sleep_hours', 'exercise_hours', 'attendance_pct', 'exam_score'] colors = ['#FF6B6B', '#40C9D9', '#6BCB77', '#FFD93D', '#FF7B7C'] titles = ['Study Hours/Week', 'Sleep Hours/Night', 'Exercise Hours/Week', 'Attendance %', 'Exam Score'] for idx, (var, color, title) in enumerate(zip(variables, colors, titles)): row = idx // 3 col = idx % 3 ax = axes[row, col] # Histogram with KDE (kernel density estimate) ax.hist(df[var], bins=20, color=color, alpha=0.6, edgecolor='black', density=True) # Add KDE curve df[var].plot(kind='kde', ax=ax, color='darkblue', linewidth=2) # Add mean line mean_val = df[var].mean() ax.axvline(mean_val, color='red', linestyle='--', linewidth=2, label=f'Mean: {mean_val:.1f}') ax.set_title(f'Distribution of {title}', fontweight='bold', fontsize=11) ax.set_xlabel(title) ax.set_ylabel('Density') ax.legend() ax.grid(alpha=0.3) # Remove empty subplot axes[1, 2].remove() plt.suptitle('📊 Distribution Analysis of Student Variables', fontsize=16, fontweight='bold', y=0.995) plt.tight_layout() plt.show() ``` :::{.callout-note} ## 🎯 Distribution Insights - **Normal (bell-shaped)**: Most students cluster around average - **Skewed**: Asymmetric distribution (could indicate issues) - **Outliers**: Extreme values that stand out - **Multimodal**: Multiple peaks (different student groups?) The distributions help us understand typical values and variability! ::: --- ## Part 4: Correlation Analysis Which variables are related to each other? ```{python} # Calculate correlation matrix correlation_matrix = df[['study_hours', 'sleep_hours', 'exercise_hours', 'attendance_pct', 'exam_score']].corr() # Create beautiful heatmap fig, ax = plt.subplots(figsize=(10, 8)) sns.heatmap(correlation_matrix, annot=True, fmt='.3f', cmap='RdYlGn', center=0, square=True, linewidths=2, cbar_kws={'label': 'Correlation Coefficient'}, vmin=-1, vmax=1, ax=ax) plt.title('🔥 Correlation Heatmap: Understanding Relationships', fontsize=14, fontweight='bold', pad=20) plt.tight_layout() plt.show() # Display strongest correlations with exam score print("\n🎯 Correlations with Exam Score:") print("="*60) score_corr = correlation_matrix['exam_score'].sort_values(ascending=False) for var, corr in score_corr.items(): if var != 'exam_score': strength = 'Strong' if abs(corr) > 0.5 else 'Moderate' if abs(corr) > 0.3 else 'Weak' print(f" {var:20s}: {corr:6.3f} ({strength})") ``` :::{.callout-tip} ## 💡 Interpreting Correlations - **+1.0**: Perfect positive correlation (both increase together) - **0.0**: No correlation (no relationship) - **-1.0**: Perfect negative correlation (one increases, other decreases) - **> 0.5**: Strong relationship - **0.3-0.5**: Moderate relationship - **< 0.3**: Weak relationship **Important**: Correlation ≠ Causation! Just because variables are correlated doesn't mean one causes the other. ::: --- ## Part 5: Scatter Plots with Regression Lines Let's visualize the strongest relationships: ```{python} # Study hours vs exam scores with regression line fig = px.scatter(df, x='study_hours', y='exam_score', trendline='ols', # Ordinary Least Squares regression title='📚 Study Hours vs Exam Performance', labels={'study_hours': 'Study Hours per Week', 'exam_score': 'Exam Score'}, color_discrete_sequence=['#FF6B6B']) fig.update_traces(marker=dict(size=8, line=dict(width=1, color='black'))) fig.update_layout( font=dict(size=12), height=500, template='plotly_white' ) fig.show() # Calculate regression statistics from scipy.stats import linregress slope, intercept, r_value, p_value, std_err = linregress(df['study_hours'], df['exam_score']) print(f"\n📈 Regression Analysis: Study Hours → Exam Score") print("="*60) print(f" Equation: Score = {intercept:.2f} + {slope:.2f} × Study Hours") print(f" R-squared: {r_value**2:.3f} ({r_value**2*100:.1f}% of variance explained)") print(f" P-value: {p_value:.6f} {'(Statistically significant!)' if p_value < 0.05 else ''}") print(f"\n Interpretation: Each additional study hour predicts a {slope:.2f} point increase in exam score!") ``` --- ## Part 6: Hypothesis Testing Let's test a hypothesis: Do students who study more than 20 hours/week score significantly higher? ```{python} # Split students into two groups high_study = df[df['study_hours'] > 20]['exam_score'] low_study = df[df['study_hours'] <= 20]['exam_score'] print(f"👥 Group Comparison") print("="*80) print(f"High study group (>20 hrs/week): {len(high_study)} students") print(f" Mean score: {high_study.mean():.2f}") print(f" Std dev: {high_study.std():.2f}") print(f"\nLow study group (≤20 hrs/week): {len(low_study)} students") print(f" Mean score: {low_study.mean():.2f}") print(f" Std dev: {low_study.std():.2f}") print(f"\nDifference in means: {high_study.mean() - low_study.mean():.2f} points") # Perform independent t-test t_statistic, p_value = stats.ttest_ind(high_study, low_study) print(f"\n📊 Statistical Test Results:") print("="*80) print(f" T-statistic: {t_statistic:.3f}") print(f" P-value: {p_value:.6f}") print(f" Significance level: α = 0.05") if p_value < 0.05: print(f"\n ✅ Result: STATISTICALLY SIGNIFICANT!") print(f" Conclusion: Students who study >20 hrs/week score significantly higher.") else: print(f"\n ❌ Result: NOT statistically significant") print(f" Conclusion: The difference could be due to random chance.") # Visualize group comparison fig = go.Figure() fig.add_trace(go.Box( y=high_study, name='High Study (>20 hrs)', marker_color='#6BCB77', boxmean='sd' )) fig.add_trace(go.Box( y=low_study, name='Low Study (≤20 hrs)', marker_color='#FF6B6B', boxmean='sd' )) fig.update_layout( title='📦 Exam Score Distribution by Study Group', yaxis_title='Exam Score', template='plotly_white', height=500, font=dict(size=12) ) fig.show() ``` :::{.callout-note} ## 🎓 Understanding Hypothesis Testing **Null Hypothesis (H₀)**: There's no difference between groups **Alternative Hypothesis (H₁)**: There is a difference **P-value**: Probability of observing this difference by chance - If p < 0.05: Reject null hypothesis (difference is real!) - If p ≥ 0.05: Fail to reject null (might be random chance) This is how scientists prove their findings! ::: --- ## Part 7: Confidence Intervals What's the true average exam score for all students? ```{python} # Calculate 95% confidence interval mean_score = df['exam_score'].mean() std_error = stats.sem(df['exam_score']) # Standard error of mean confidence_level = 0.95 degrees_freedom = len(df) - 1 # Calculate confidence interval confidence_interval = stats.t.interval(confidence_level, degrees_freedom, loc=mean_score, scale=std_error) print(f"📊 95% Confidence Interval for Mean Exam Score") print("="*80) print(f" Sample mean: {mean_score:.2f}") print(f" Standard error: {std_error:.2f}") print(f" 95% Confidence Interval: [{confidence_interval[0]:.2f}, {confidence_interval[1]:.2f}]") print(f"\n Interpretation: We are 95% confident that the true population mean") print(f" exam score falls between {confidence_interval[0]:.2f} and {confidence_interval[1]:.2f}.") # Visualize confidence interval fig = go.Figure() # Add mean point fig.add_trace(go.Scatter( x=[mean_score], y=['Mean'], mode='markers', marker=dict(size=15, color='#FF6B6B'), name='Sample Mean', showlegend=True )) # Add confidence interval fig.add_trace(go.Scatter( x=[confidence_interval[0], confidence_interval[1]], y=['Mean', 'Mean'], mode='lines+markers', marker=dict(size=10, color='#40C9D9'), line=dict(width=4, color='#40C9D9'), name='95% CI', showlegend=True )) fig.update_layout( title='🎯 95% Confidence Interval for Mean Exam Score', xaxis_title='Exam Score', template='plotly_white', height=400, font=dict(size=12), yaxis=dict(showticklabels=False) ) fig.show() ``` --- ## Part 8: Multiple Regression Analysis Can we predict exam scores using multiple variables? ```{python} from sklearn.linear_model import LinearRegression from sklearn.metrics import r2_score, mean_squared_error # Prepare data X = df[['study_hours', 'sleep_hours', 'exercise_hours', 'attendance_pct']] y = df['exam_score'] # Fit model model = LinearRegression() model.fit(X, y) predictions = model.predict(X) # Display results print(f"📊 Multiple Regression: Predicting Exam Scores") print("="*80) print(f"\nRegression Equation:") print(f" Score = {model.intercept_:.2f}") for feature, coef in zip(X.columns, model.coef_): sign = '+' if coef >= 0 else '-' print(f" {sign} {abs(coef):.2f} × {feature}") print(f"\n📈 Model Performance:") print(f" R-squared: {r2_score(y, predictions):.3f}") print(f" RMSE: {np.sqrt(mean_squared_error(y, predictions)):.2f} points") print(f"\n💡 Feature Importance (sorted by absolute coefficient):") feature_importance = pd.DataFrame({ 'Feature': X.columns, 'Coefficient': model.coef_, 'Abs_Coefficient': np.abs(model.coef_) }).sort_values('Abs_Coefficient', ascending=False) for idx, row in feature_importance.iterrows(): print(f" {row['Feature']:20s}: {row['Coefficient']:6.2f} (impact per unit)") ``` :::{.callout-tip} ## 🎯 Key Findings Based on our statistical analysis: 1. **Study hours matter**: Strong positive correlation with exam scores 2. **Sleep is important**: Better sleep → better performance 3. **Exercise helps**: Physical activity correlates with academic success 4. **Attendance counts**: Class participation predicts higher scores 5. **Multiple factors**: Best predictions use all variables together This is how data science informs educational policy! ::: --- ## 🎓 What You've Learned Congratulations! You've mastered fundamental statistical concepts: ✅ **Descriptive statistics** (mean, median, std dev, percentiles) ✅ **Distribution analysis** and visualization ✅ **Correlation analysis** to find relationships ✅ **Hypothesis testing** with t-tests ✅ **Confidence intervals** for estimation ✅ **Multiple regression** for prediction ### 🚀 What's Next in CMPSC 301? In the course, you'll learn advanced statistics: - **ANOVA**: Comparing multiple groups - **Chi-square tests**: Analyzing categorical data - **Non-parametric tests**: When data isn't normal - **Time series analysis**: Temporal patterns - **Bayesian statistics**: Probabilistic reasoning - **Experimental design**: A/B testing and causality --- ## 💡 Try It Yourself! Experiment with these challenges: 1. **Add more variables**: GPA, major, socioeconomic factors 2. **Test other hypotheses**: Do exercise and sleep interact? 3. **Outlier analysis**: Find and investigate unusual students 4. **Predictive modeling**: Build a model to predict struggling students 5. **Real data**: Analyze actual student performance data **Ready to become a statistical wizard?** Visit [JupyterLite](/live/)! --- :::{.callout-note} ## 🌍 Real-World Applications Statistical analysis is used everywhere: - **Medicine**: Clinical trials, drug effectiveness - **Business**: A/B testing, market research - **Sports**: Player performance, team strategies - **Policy**: Economic indicators, social programs - **Science**: Research validation, discoveries Master these tools in CMPSC 301 and unlock countless opportunities! :::