Get started with SageMaker Debugger

• Tensor analysis using Amazon SageMaker Debugger
  • Installing smdebug
  • Configuring the inputs for the training job
  • Get S3 location of tensors
  • Activation outputs
  • Activation Inputs
  • Gradients
  • Check variance across layers
  • Distribution of weights
  • Inputs
  • Layer outputs
  • Comparison
    • Gradients
    • Activation inputs
• Visualizing Debugging Tensors of MXNet training
  • Overview
  • Dependencies
  • Configure and run the training job
  • Get S3 location of tensors
  • Download tensors from S3
  • Visualize

Real-time analysis

• Using SageMaker debugger to monitor autoencoder model training
  • Training MXNet autoencoder model in Amazon SageMaker with debugger
  • Get tensors and visualize model training in real-time
• Using SageMaker debugger to monitor attentions in BERT model training
  • SageMaker training
  • Get tensors and visualize BERT model training in real-time
    • Attention Head View
    • Neuron view
    • Note: Jupyter widgets in JupyterLab
• Using SageMaker debugger to visualize class activation maps in CNNs
  • Customize the smdebug hook
  • Replace in-place operations
  • Download the dataset and upload it to Amazon S3
  • SageMaker training
    • Debugger hook configuration
    • Builtin rule
  • SageMaker training
    • Check rule status
  • Visualize saliency maps in real-time

MXNet

• Debugging Amazon SageMaker training jobs in real time with Debugger
  • Overview
    • Saving tensors
    • Analysis
    • Installing smdebug
  • Training MXNet models in Amazon SageMaker with Debugger
  • Enable Debugger in Estimator object
    • Configuring the inputs for the training job
    • Result
  • Analysis and Visualization
    • Checking on the training job status
    • Retrieving and Analyzing tensors
    • Visualize tensors of a running training job
    • Additional visualizations
• Enable Spot Training with Amazon SageMaker Debugger
  • Saving tensors
  • Analysis
    • Spot Training
  • Configuring the inputs for the training job
    • Training MXNet models in Amazon SageMaker with Amazon SageMaker Debugger
    • Enable Amazon SageMaker Debugger and Spot Training in Estimator object
  • Rule Parameter
    • How Spot Training works with Amazon SageMaker Debugger

PyTorch

• Using SageMaker Debugger and SageMaker Experiments for iterative model pruning
  • Get training dataset
  • Load and save ResNet model
  • Iterative model pruning: step by step
    • Step 0: Create trial and debugger hook coonfiguration
    • Step 1: Start training job
    • Step 2: Get gradients, weights, biases
    • Step 3: Compute filter ranks
    • Step 4 and step 5: Prune low ranking filters and set new weights
    • Step 6: Start next pruning iteration
    • Overall workflow
  • Run iterative model pruning
  • Additional: run iterative model pruning with custom rule
• Using SageMaker Debugger and SageMaker Experiments for iterative model pruning
  • Get training dataset
  • Load and save AlexNet model
  • Iterative model pruning: step by step
    • Step 0: Create trial and debugger hook coonfiguration
    • Step 1: Start training job
    • Step 2: Get gradients, weights, biases
    • Step 3: Compute filter ranks
    • Step 4 and step 5: Prune low ranking filters and set new weights
    • Step 6: Start next pruning iteration
    • Overall workflow
  • Run iterative model pruning
  • Results
  • Additional: run iterative model pruning with custom rule
• Using Amazon SageMaker Debugger with your own PyTorch container
  • How does Amazon SageMaker Debugger work?
    • Concepts
    • Saving tensors
    • Analysis of tensors
  • Setup
  • Bring Your Own PyTorch for training
    • Modifying the training script
    • Result
    • Analysis

TensorFlow

Tensorflow 2.x

• Amazon SageMaker - Tensorflow 2.1
  • Training TensorFlow Keras models with Amazon SageMaker Debugger
    • Amazon SageMaker TensorFlow as a framework
  • Setup
    • Setting up the Estimator
      • Using Amazon SageMaker Rules
  • Result
  • Data Analysis - Interactive Exploration
• Amazon SageMaker - Tensorflow 2.1
  • Training TensorFlow Keras models with Amazon SageMaker Debugger
    • Amazon SageMaker TensorFlow as a framework
  • Setup
    • Setting up the Estimator
      • Using Amazon SageMaker Rules
  • Result
  • Data Analysis - Interactive Exploration

TensorFlow 1.x

• Amazon SageMaker Debugger - Using built-in rule
  • How does Amazon SageMaker Debugger work?
    • Concepts
    • Saving tensors during training
      • With no changes to your training script
      • Orchestrating your script to store tensors
    • Analysis of tensors
  • Setup
  • Training TensorFlow models in SageMaker with Amazon SageMaker Debugger
    • SageMaker TensorFlow as a framework
    • Setting up the Estimator
      • Using Amazon SageMaker Rules
  • Result
  • Data Analysis - Interactive Exploration
• Detect Stalled Training and Stop Training Job Using SageMaker Debugger Rule
  • How the StalledTrainingRule Built-in Rule Works
    • Import SageMaker Python SDK
    • Import SageMaker Debugger classes for rule configuration
    • Create a unique training job prefix
  • Monitoring Training and Rule Evaluation Status
    • Print the training job name
    • Output the current job status and the rule evaluation status
  • Conclusion
• Amazon SageMaker Debugger - Reacting to Cloudwatch Events from Rules
  • How does Amazon SageMaker Debugger work?
    • Concepts
    • Saving tensors during training
      • With no changes to your training script
      • Orchestrating your script to store tensors
    • Analysis of tensors
    • Cloudwatch Events for Rules
      • Lambda Function
      • Create a CloudWatch Rule
  • Monitoring
  • Result
• Amazon SageMaker - Debugging with custom rules
  • How does Amazon SageMaker Debugger work?
    • Concepts
    • Saving tensors during training
      • With no changes to your training script
      • Orchestrating your script to store tensors
    • Analysis of tensors
  • Training TensorFlow Keras models with Amazon SageMaker Debugger
    • Amazon SageMaker TensorFlow as a framework
    • Setting up the Estimator
      • Creating your own custom rule
        • Summary of what the custom rule evaluates
        • Rule class constructor
        • Defining the rule logic to be invoked at each step:
      • Using your custom rule with SageMaker Estimator
  • Result

XGBoost

• Debugging XGBoost Training Jobs with Amazon SageMaker Debugger Using Rules
  • Overview
    • Saving tensors
    • Analysis
  • Training XGBoost models in Amazon SageMaker with Amazon SageMaker Debugger
    • Data preparation
    • Enabling Debugger in Estimator object
      • DebuggerHookConfig
      • Rules
    • Result
    • Check the status of the Rule Evaluation Job
  • Making this a good run
  • Data Analysis - Manual
    • Performance metrics
    • Feature importances
    • SHAP
• Debugging XGBoost training jobs in real time with Amazon SageMaker Debugger
  • Overview
    • Saving tensors
    • Analysis
  • Training XGBoost models in Amazon SageMaker with Amazon SageMaker Debugger
    • Data preparation
    • Enabling Amazon SageMaker Debugger in the estimator object
    • Using Amazon SageMaker Debugger with XGBoost Classification
    • Result
  • Analysis and Visualization
    • Checking on the training job status
    • Retrieving and Analyzing tensors
    • Visualizing confusion matrix of a running training job
• Explainability with Amazon SageMaker Debugger
  • Introduction
    • Saving tensors
    • Analysis
  • Section 1 - Setup
    • 1.1 Import necessary libraries
    • 1.2 AWS region and IAM Role
    • 1.3 S3 bucket and prefix to hold training data, debugger information and model artifact
  • Section 2 - Data preparation
  • Section 3 - Train XGBoost model in Amazon SageMaker with debugger enabled.
    • 3.1 Install the ‘smdebug’ open source library
    • 3.2 Build the XGBoost container
    • 3.3 Enabling Debugger in Estimator object
      • DebuggerHookConfig
      • Rules
    • 3.4 Result
    • 3.5 Check the status of the Rule Evaluation Job
  • Section 4 - Analyze debugger output
    • Retrieving and Analyzing tensors
    • Plot Performance metrics
    • Feature importance
    • SHAP
    • Global explanations
    • Local explanations
      • Force plot
      • Stacked force plot
    • Outliers
  • Conclusion

Bring your own container

• Build a Custom Training Container and Debug Training Jobs with Amazon SageMaker Debugger
  • Step 1: Prepare prerequisites
    • Install the SageMaker Python SDK v2 and the smdebug library
    • [Optional Step] Restart the kernel to apply the update
  • Step 2: Prepare a Dockerfile and register the Debugger hook to you training script
    • Prepare a Dockerfile
    • Prepare a training script
  • Step 3: Create a Docker image, build the Docker training container, and push to Amazon ECR
    • Create a Docker image
    • [Optional Step] Login to access the Deep Learning Containers image repository
    • Build the Docker container and push it to Amazon ECR
  • Step 4: Use Amazon SageMaker to set the Debugger hook and rule configuration
    • Define Debugger hook configuration
    • Select Debugger built-in rules
  • Step 5. Define a SageMaker Estimator object with Debugger and initiate a training job
    • Initiate the training job in the background
    • Print the training job name
    • Output the current job status
  • Step 6: Retrieve output tensors using the smdebug trials class
    • Call the latest Debugger artifact and create a smdebug trial
    • Print the hyperparameter configuration saved as scalar values
    • Print the size of the steps list to check the training progress
  • Step 7: Analyze the training job using the smdebug trial methods and the Debugger rule job status
    • Plot training and validation loss curves in real time
    • Check the rule job summary
  • Notebook Summary and Other Applications