​

Setting Up and Running Examples

​

Prerequisites

• Python 3.10-3.12
• Git
• uv package manager (recommended)

​

Step 1: Clone the Repository

git clone https://github.com/TristanBester/pycrm.git
cd pycrm

​

Step 2: Set Up the Development Environment

​

Option A: Using uv (Recommended)

# Create and activate virtual environment
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies with development extras
uv pip install -e ".[dev]"

uv sync --extra dev

​

Option B: Using pip

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies with development extras
pip install -e ".[dev]"

pip install uv

​

Step 3: Navigate to Examples Directory

cd examples

​

Step 4: Understanding the Example Structure

​

Introduction Examples (introduction/)

• q_learning.py - Q-learning in a letter world environment
• cq_learning.py - Counterfactual Q-learning
• environment_example.py - Basic environment setup
• core/ - Core components (ground environment, labelling function, reward machine)

​

CRM Examples (crm/)

• Continuous: SAC, TD3, DDPG, and their CRM variants (CSAC, CTD3, CDDPG)
• Discrete: DQN and CDQN
• Tabular: Basic tabular implementations

​

RM Examples (rm/)

• Continuous: SAC, TD3, DDPG
• Discrete: DQN
• Tabular: Basic tabular implementations

​

Step 5: Running Examples

​

Basic Examples

cd introduction

# Run Q-learning example
python q_learning.py

# Run counterfactual Q-learning
python cq_learning.py

# Run environment example
python environment_example.py

​

Advanced Examples

# CRM continuous examples
cd crm/continuous
python sac.py --seed 42

# CRM discrete examples
cd crm/discrete
python dqn.py --seed 42

# RM continuous examples
cd rm/continuous
python sac.py --seed 42

# RM discrete examples
cd rm/discrete
python dqn.py --seed 42

# Run with specific seed for reproducibility
python sac.py --seed 123

# Run multiple experiments (if supported)
python sac.py --seeds 1 2 3 4 5

​

Step 6: Experiment Tracking

# Log in to wandb (first time only)
wandb login

# Run example with wandb tracking
python sac.py --seed 42

​

Step 7: Using SLURM (HPC Systems)

​

SLURM Script Organization

• scripts/examples/crm/ - Scripts for Counting Reward Machine examples
• scripts/examples/rm/ - Scripts for standard Reward Machine examples

• continuous/ - Scripts for continuous control algorithms (SAC, TD3, DDPG, etc.)
• discrete/ - Scripts for discrete control algorithms (DQN, etc.)

​

Running SLURM Scripts

# Navigate to the appropriate scripts directory
cd scripts/examples/crm/continuous

# Run all algorithms in the category
./start.sh

# Or run individual algorithms
sbatch sac.slurm
sbatch td3.slurm
sbatch ddpg.slurm

# For discrete algorithms
cd scripts/examples/crm/discrete
sbatch dqn.slurm

​

Troubleshooting

​

Common Issues

1. Import Errors: Ensure you’re running from the correct directory:
   cd examples/crm/continuous  # Not just cd examples
   
2. Dependency Issues: Make sure all dependencies are installed:
   uv pip install -e ".[dev]"
   
3. Python Version: Ensure you’re using Python 3.10-3.12:
   python --version
   
4. Virtual Environment: Activate the virtual environment:
   source .venv/bin/activate
   

​

Next Steps

• Check the Worked Examples for detailed tutorials
• Explore the Core Concepts documentation
• Try modifying the examples to experiment with different parameters