MLKD Cluster Guide

Welcome to our cluster guide, this document serves as an introduction to our server and how to use it effectively. We use PyTorch to examplify good practices when writing Machine Learning code and dealing with Datasets and GPU but the same topics are applicable to any library (Tensorflow, Jax, etc.).

Index

• 1 - Introduction to the Cluster

  • 1.1 - Authentication and File Server
    • 1.1.1 - Home Directories
    • 1.1.2 - Data Storage
    • 1.1.3 - IO Intensive Operations
  • 1.2 - Computation Machines
  • 1.3 - GPU Monitor

• 2 - Using the Machines

  • 2.1 - Accessing the Machines
    • 2.1.1 - SSH
    • 2.1.2 - SSH Key Authentication
    • 2.1.3 - SSH Config
  • 2.2 - Accessing Remote Files - SSHFS
  • 2.3 - Transferring Files
    • 2.3.1 - SCP
    • 2.3.2 - SFTP
  • 2.4 - Package Environments
    • 2.4.1 - Anaconda Environments
  • 2.5 - Detaching Processes from SSH Sessions
    • 2.5.1 - tmux
    • 2.5.2 - screen
    • 2.5.3 - nohup
  • 2.6 - VS Code
    • 2.6.1 - VS Code Remote SSH
    • 2.6.2 - Other VS Code Extensions
  • 2.7 - Useful Commands
    • 2.7.1 - htop
    • 2.7.2 - nvidia-smi
    • 2.7.3 - nvtop

• 3 - PyTorch Basics

  • 3.1 - Tensor
    • 3.1.1 - Tensor Creation
    • 3.1.2 - Tensor Data Types
    • 3.1.3 - Tensor Operations
    • 3.1.4 - Other PyTorch Operations Over Tensors
    • 3.1.5 - Tensor Indexing
    • 3.1.6 - Tensor Broadcasting
  • 3.2 - Data
    • 3.2.1 - Datset
    • 3.2.2 - DataLoader
  • 3.3 - nn.Module
  • 3.4 - Vectorization
  • 3.5 - Using the GPU
  • 3.6 - Optimizers and Learning Rate Scheduler
    • 3.6.1 - Optimizer
    • 3.6.2 - Learning Rate Scheduler
  • 3.7 - Gradient Computation and Weight Update
    • 3.7.1 - Gradient Computation
    • 3.7.2 - Weight Update

• 4 - Writing Machine Learning Scripts

  • 4.1 - Parsing Arguments and Hyperparameters
  • 4.2 - Logging
  • 4.3 - Saving and Loading Checkpoints
  • 4.4 - Seeds
  • 4.5 - Testing Code with Jupyter Notebooks
  • 4.6 - Type Hinting and Docstrings
  • 4.7 - Git
  • 4.8 - FAQ

• 5 - Extras

  • 5.1 - Huggingface
  • 5.2 - Lightning
  • 5.3 - Git Hook and CI/CD
  • 5.4 - Github Copilot
  • 5.5 - Profiling Tools
  • 5.6 - Debug Tools

• 6 - FAQ