Python Environments

Manage Python environments for deep learning with venv, pip, and Conda. Setup isolated environments, handle dependencies, and troubleshoot common package conflicts in PyTorch and TensorFlow.

Choosing an Environment Manager

For deep learning projects, you need isolated Python environments to manage dependencies. Here are your options:

Recommended: venv + pip

Best for most users - More stable and predictable

Pros:

• Built into Python (no extra installation)
• Works seamlessly with pip
• Lighter weight than Conda
• Fewer dependency conflicts
• Faster environment creation

Cons:

• No cross-language packages (R, C++, etc.)
• Manual CUDA/system library management

Alternative: Conda

Best for: Complex scientific computing stacks with system dependencies

Pros:

• Manages system libraries (CUDA, cuDNN, etc.)
• Cross-language support
• Pre-built binaries for many packages

Cons:

• Larger disk usage
• Slower environment solving
• Mixing pip and Conda can cause conflicts
• More complex to troubleshoot

:::tip[Our Recommendation] Use venv + pip unless you specifically need Conda’s system library management. Modern deep learning frameworks (PyTorch, TensorFlow) work great with pip. :::

Quick Start

# Create virtual environment
python3 -m venv ml-env

# Activate it
source ml-env/bin/activate  # Linux/Mac
# or
ml-env\Scripts\activate  # Windows

# Upgrade pip
pip install --upgrade pip

# Install PyTorch with CUDA
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Or TensorFlow
pip install tensorflow[and-cuda]

# Install common ML packages
pip install numpy pandas matplotlib scikit-learn jupyter

deactivate

# Find all venv environments
find ~ -type d -name "bin" -exec test -e "{}/activate" \; -print 2>/dev/null

# Simply delete the folder
rm -rf ml-env

# Create environment with specific Python version
conda create -n ml-env python=3.11

# Activate it
conda activate ml-env

# IMPORTANT: Install conda packages FIRST, then pip packages

# Install CUDA toolkit via Conda (if needed)
conda install cuda -c nvidia

# Install PyTorch via Conda
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

# Or use pip for PyTorch (after conda packages)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

conda deactivate

conda env list

conda env remove -n ml-env

Environment Management Best Practices

Project-Specific Environments

Create one environment per project:

# Project 1
python3 -m venv ~/envs/project1
source ~/envs/project1/bin/activate
pip install torch transformers

# Project 2
python3 -m venv ~/envs/project2
source ~/envs/project2/bin/activate
pip install tensorflow keras

Requirements Files

Track dependencies:

# Save current packages
pip freeze > requirements.txt

# Install from requirements
pip install -r requirements.txt

Better: Use minimal requirements.txt

# requirements.txt - only specify what you actually use
torch>=2.0.0
transformers>=4.30.0
numpy>=1.24.0

Environment Aliases

Add to ~/.bashrc or ~/.zshrc:

# Quick activation aliases
alias ml='source ~/envs/ml-env/bin/activate'
alias cv='source ~/envs/cv-project/bin/activate'
alias nlp='source ~/envs/nlp-project/bin/activate'

Common Issues

Issue: Conda Taking Forever to Solve Environment

Problem: Solving environment: failed with initial frozen solve. Retrying with flexible solve.

Solutions:

# Install mamba (drop-in replacement for conda)
conda install mamba -c conda-forge

# Use mamba instead of conda
mamba install pytorch torchvision -c pytorch

# Create conda environment
conda create -n ml-env python=3.11
conda activate ml-env

# Use pip for packages (much faster)
pip install torch torchvision

Issue: Mixing pip and Conda Broke Environment

Problem: Packages installed but not found, dependency conflicts

Solution: Start fresh with one approach

# Delete broken environment
conda env remove -n broken-env

# Recreate with venv + pip only
python3 -m venv ml-env
source ml-env/bin/activate
pip install -r requirements.txt

Issue: “No module named X” After Installation

Check you’re in the right environment:

# Check which Python is being used
which python

# Should show your environment path, not system Python
# Correct: /home/user/ml-env/bin/python
# Wrong: /usr/bin/python

Issue: Multiple Python Versions Conflicting

Use specific Python version:

# venv with specific Python
python3.11 -m venv ml-env
source ml-env/bin/activate

# Conda with specific Python
conda create -n ml-env python=3.11
conda activate ml-env

Conda-Specific Issues

Check Conda Disk Usage

# Check conda cache size
du -hs ~/.conda

# Clean up old packages
conda clean --all

# Remove unused environments
conda env list
conda env remove -n unused-env

TensorFlow Version Conflicts

# Search for compatible versions
conda search -c conda-forge tensorflow-gpu

# Install specific version
conda install -c conda-forge tensorflow-gpu=2.16.2

Jupyter Integration

See Jupyter Notebooks for:

• Adding environments as Jupyter kernels
• Managing ipykernel
• Jupyter-specific troubleshooting

Quick Reference

venv Commands

# Create
python3 -m venv envname

# Activate
source envname/bin/activate

# Install package
pip install packagename

# Save dependencies
pip freeze > requirements.txt

# Deactivate
deactivate

# Delete
rm -rf envname

Conda Commands

# Create
conda create -n envname python=3.11

# Activate
conda activate envname

# Install package
conda install packagename

# Save dependencies
conda env export > environment.yml

# Deactivate
conda deactivate

# Delete
conda env remove -n envname

Next Steps

• Set up Jupyter notebooks with your environment
• Install NVIDIA drivers for GPU support
• Troubleshoot GPU errors