Python Data Science Essentials

## Why Python for Data Science? Python has become the lingua franca of data science thanks to its expressive syntax and a rich ecosystem of specialized libraries. Whether you are cleaning messy CSV files, building statistical models, or producing publication-ready charts, Python provides battle-tested tools that scale from prototype to production. The three pillars of the Python data stack are **NumPy** for numerical computation, **Pandas** for data manipulation, and **Matplotlib / Seaborn** for visualization. Understanding how these libraries interact is the first step toward efficient data workflows. ## Setting Up Your Environment Before writing any code, ensure you have a reproducible environment. Virtual environments isolate your project dependencies and prevent version conflicts. ```bash # Create and activate a virtual environment python -m venv .venv source .venv/bin/activate # Install core data science packages pip install numpy pandas matplotlib seaborn jupyter ``` Using Jupyter notebooks alongside a solid editor like **Visual Studio Code** gives you interactive exploration with the comfort of version control via **GitHub**. ## NumPy: The Foundation NumPy provides the `ndarray`, a homogeneous multi-dimensional array that is orders of magnitude faster than Python lists for numerical operations. ```python import numpy as np # Create arrays from Python lists a = np.array([1, 2, 3, 4, 5]) b = np.array([[1, 2, 3], [4, 5, 6]]) # Vectorized operations eliminate explicit loops print(a * 2) # [2 4 6 8 10] print(a + np.arange(5)) # [1 3 5 7 9] # Statistical functions print(f"Mean: {a.mean()}, Std: {a.std()}") ``` Broadcasting is one of NumPy's most powerful features. It allows arithmetic between arrays of different shapes without copying data. ```python # Broadcasting: add a row vector to each row of a matrix matrix = np.ones((3, 4)) row = np.array([1, 2, 3, 4]) result = matrix + row # Each row of ones is incremented by [1, 2, 3, 4] ``` ## Pandas: Data Manipulation Powerhouse Pandas builds on NumPy to provide labeled data structures and high-level operations that mirror SQL and spreadsheet workflows. ### DataFrames and Series A `DataFrame` is a two-dimensional labeled array. Each column is a `Series`. ```python import pandas as pd # Create a DataFrame from a dictionary df = pd.DataFrame({ "name": ["Alice", "Bob", "Charlie", "Diana"], "age": [28, 35, 42, 31], "department": ["Engineering", "Marketing", "Engineering", "Sales"], "salary": [95000, 72000, 110000, 68000] }) print(df.head()) ``` ### Filtering and Selection Pandas offers multiple ways to select data, each suited to different scenarios. ```python # Boolean indexing seniors = df[df["age"] >= 35] # Query syntax (readable for complex conditions) engineers = df.query("department == 'Engineering' and salary > 100000") # Column selection names_and_salaries = df[["name", "salary"]] ``` ### Grouping and Aggregation The `groupby` mechanism splits data into groups, applies a function, and combines the results — the classic split-apply-combine pattern. ```python # Average salary by department dept_stats = df.groupby("department").agg( avg_salary=("salary", "mean"), headcount=("name", "count") ).reset_index() print(dept_stats) ``` ## Handling Missing Data Real-world datasets are rarely clean. Pandas represents missing values as `NaN` and provides intuitive methods for detection and imputation. ```python import numpy as np # Introduce missing values df.loc[1, "salary"] = np.nan df.loc[3, "age"] = np.nan # Detect missing values print(df.isnull().sum()) # Fill missing values df["salary"] = df["salary"].fillna(df["salary"].median()) # Drop rows with any missing values clean_df = df.dropna() ``` ## Data Visualization Visual exploration reveals patterns that summary statistics alone cannot capture. ```python import matplotlib.pyplot as plt import seaborn as sns # Seaborn statistical plots fig, axes = plt.subplots(1, 2, figsize=(12, 5)) # Distribution plot sns.histplot(df["salary"], kde=True, ax=axes[0], color="steelblue") axes[0].set_title("Salary Distribution") # Scatter plot with hue sns.scatterplot(data=df, x="age", y="salary", hue="department", s=100, ax=axes[1]) axes[1].set_title("Age vs Salary by Department") plt.tight_layout() plt.savefig("analysis_output.png", dpi=150) plt.show() ``` ## Real-World Workflow: Loading and Cleaning a Dataset A typical data science project follows a consistent pipeline: load, explore, clean, transform, analyze, and visualize. ```python # Load a CSV dataset raw = pd.read_csv("sales_data.csv") # Quick exploration print(raw.shape) print(raw.dtypes) print(raw.describe()) # Clean: remove duplicates and standardize column names raw = raw.drop_duplicates() raw.columns = raw.columns.str.strip().str.lower().str.replace(" ", "_") # Transform: create derived columns raw["revenue"] = raw["quantity"] * raw["unit_price"] raw["order_date"] = pd.to_datetime(raw["order_date"]) raw["month"] = raw["order_date"].dt.to_period("M") # Analyze: monthly revenue trend m monthly.plot(kind="line", marker="o", title="Monthly Revenue") plt.ylabel("Revenue ($)") plt.tight_layout() plt.show() ``` ## Performance Tips - Prefer vectorized NumPy/Pandas operations over Python `for` loops. - Use `pd.read_csv()` with `dtypes` parameter to reduce memory on large files. - Leverage `eval()` and `query()` for complex expressions on large DataFrames. - Consider **Polars** or **Dask** when datasets exceed memory. ## Next Steps Once you are comfortable with these essentials, explore **scikit-learn** for machine learning, **Plotly** for interactive dashboards, and **Apache Airflow** for scheduling data pipelines. These tools build directly on the NumPy and Pandas foundations covered here.