AWS SageMaker Security: Notebook Escape, Training Job Credential Theft, and IAM Hardening

AWS SageMaker has become a common platform for enterprise machine learning workloads, which means it’s also becoming a common pivot point for cloud attackers. The combination of permissive execution environments (Jupyter notebooks running as EC2), high-privilege IAM roles, and broad S3 access creates a reliable path from initial access to significant cloud compromise.

This guide covers the four most impactful SageMaker misconfiguration patterns, with exploitation context and remediation CLI commands.

1. Overprivileged Notebook Execution Roles

SageMaker notebook instances run with an IAM role attached at instance creation. In practice, this role is frequently arn:aws:iam::ACCOUNT:role/AmazonSageMaker-ExecutionRole-* — which AWS creates with AmazonSageMakerFullAccess and sometimes AmazonS3FullAccess attached.

The Exploitation Path

From a notebook terminal, an attacker can directly query the instance metadata service:

# From inside a SageMaker notebook — accessing the execution role credentials
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/
# Returns the role name, then:
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/<ROLE_NAME>
# Returns: AccessKeyId, SecretAccessKey, Token (valid for up to 6 hours)

With AmazonSageMakerFullAccess, those credentials permit:

Training job creation (for compute access)
Model deployment to endpoints
Access to all SageMaker resources in the account
S3 bucket listing and object download across the account if S3FullAccess is attached

Remediation

Apply least-privilege notebook roles scoped to specific S3 prefixes and SageMaker actions:

# Check current notebook instance role
aws sagemaker describe-notebook-instance \
  --notebook-instance-name <INSTANCE_NAME> \
  --query 'RoleArn' --output text

# List policies attached to the execution role
ROLE_NAME=$(aws sagemaker describe-notebook-instance \
  --notebook-instance-name <INSTANCE_NAME> \
  --query 'RoleArn' --output text | cut -d'/' -f2)

aws iam list-attached-role-policies --role-name "$ROLE_NAME"

# Remove AmazonSageMakerFullAccess — replace with custom policy
aws iam detach-role-policy \
  --role-name "$ROLE_NAME" \
  --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess

Replace with a scoped policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sagemaker:CreateTrainingJob",
        "sagemaker:DescribeTrainingJob",
        "sagemaker:StopTrainingJob"
      ],
      "Resource": "arn:aws:sagemaker:*:ACCOUNT_ID:training-job/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::your-ml-bucket",
        "arn:aws:s3:::your-ml-bucket/team-prefix/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["cloudwatch:PutMetricData", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "*"
    }
  ]
}

2. Training Job Credential Exposure via Custom Container

SageMaker training jobs run custom containers on managed EC2 instances, and the training container has access to the same IMDS-provided credentials as the notebook instance. If an attacker can influence the training container image or the training script, they can exfiltrate credentials during training — which often runs on powerful GPU instances with very high-privilege roles.

Detection

Monitor CloudTrail for unusual credential use from SageMaker training jobs:

# Find training jobs that made unusual API calls during execution
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventSource,AttributeValue=sts.amazonaws.com \
  --start-time 2026-07-03T00:00:00Z \
  --end-time 2026-07-04T23:59:59Z \
  --query "Events[?contains(Username, 'SageMaker')].[EventTime,Username,EventName,Resources]" \
  --output table

Remediation: Block IMDS Access from Training Containers

# Disable IMDS v1 and restrict IMDS hop count for new notebook instances
aws sagemaker create-notebook-instance \
  --notebook-instance-name secure-notebook \
  --instance-type ml.t3.medium \
  --role-arn arn:aws:iam::ACCOUNT:role/SageMakerScopedRole \
  --instance-metadata-service-configuration '{
    "MinimumInstanceMetadataServiceVersion": "2"
  }'

# For existing instances — update at next stop/start cycle
aws sagemaker update-notebook-instance \
  --notebook-instance-name <INSTANCE_NAME> \
  --instance-metadata-service-configuration '{
    "MinimumInstanceMetadataServiceVersion": "2"
  }'

IMDSv2 requires a session token obtained via PUT request — scripts that simply curl http://169.254.169.254/... without the session header will fail, blocking the most common credential theft method.

3. Public SageMaker Endpoints

SageMaker inference endpoints are internet-accessible by default unless explicitly configured with VPC attachment and security groups. Model endpoints can expose:

Model architecture and weights (through prediction API probing)
Sensitive inference data if logging is misconfigured
Compute resources (for high-cost GPU instance types)

Audit Public Endpoints

# List all SageMaker endpoints and check for VPC configuration
aws sagemaker list-endpoints \
  --query 'Endpoints[*].[EndpointName,EndpointStatus]' \
  --output table

# Check endpoint configuration for VPC attachment
for EP in $(aws sagemaker list-endpoints --query 'Endpoints[*].EndpointName' --output text); do
  CONFIG=$(aws sagemaker describe-endpoint --endpoint-name "$EP" --query 'EndpointConfigName' --output text)
  VPC=$(aws sagemaker describe-endpoint-config \
    --endpoint-config-name "$CONFIG" \
    --query 'VpcConfig' --output text 2>/dev/null)
  echo "$EP: VPC=${VPC:-NONE}"
done

Remediation: Attach Endpoints to VPC with Resource-Based Policies

# Add resource-based policy to restrict invocation to specific principals
aws sagemaker add-tags \
  --resource-arn arn:aws:sagemaker:REGION:ACCOUNT:endpoint/ENDPOINT_NAME \
  --tags Key=access-tier,Value=internal-only

# Create endpoint in VPC (new endpoint)
aws sagemaker create-endpoint-config \
  --endpoint-config-name secure-config \
  --production-variants '...' \
  --vpc-config '{
    "SecurityGroupIds": ["sg-XXXXXXXX"],
    "Subnets": ["subnet-XXXXXXXX", "subnet-YYYYYYYY"]
  }'

4. SageMaker Studio Domain with Overpermissive User Profiles

SageMaker Studio creates a domain with user profiles, each backed by an EFS volume and an execution role. Misconfigured Studio domains allow:

Cross-user data access if EFS mounts lack user-level isolation
Privilege escalation if user profile roles have iam:PassRole or sagemaker:CreateDomain
Persistent access via Studio’s kernel gateway, which can survive session termination

Audit Studio Domain Configuration

# List Studio domains
aws sagemaker list-domains --query 'Domains[*].[DomainId,DomainName,Status]' --output table

# Check default user settings for the domain
aws sagemaker describe-domain \
  --domain-id <DOMAIN_ID> \
  --query 'DefaultUserSettings.ExecutionRole' --output text

# Check whether cross-user EFS access is possible
aws sagemaker describe-domain \
  --domain-id <DOMAIN_ID> \
  --query 'HomeEfsFileSystemId' --output text

# Verify EFS access points are per-user
aws efs describe-access-points \
  --file-system-id <EFS_ID> \
  --query 'AccessPoints[*].[AccessPointId,PosixUser,RootDirectory]' \
  --output table

SCP to Prevent Dangerous SageMaker Actions

Apply at the AWS Organizations level to prevent researchers from self-escalating:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenySageMakerPrivilegeEscalation",
      "Effect": "Deny",
      "Action": [
        "sagemaker:CreateDomain",
        "sagemaker:UpdateDomain",
        "sagemaker:CreateUserProfile",
        "sagemaker:DeleteDomain"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalArn": [
            "arn:aws:iam::*:role/SageMakerAdminRole",
            "arn:aws:iam::*:role/CloudAdminRole"
          ]
        }
      }
    },
    {
      "Sid": "RequireIMDSv2ForSageMaker",
      "Effect": "Deny",
      "Action": "sagemaker:CreateNotebookInstance",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "sagemaker:MinimumInstanceMetadataServiceVersion": "2"
        }
      }
    }
  ]
}

Detection: GuardDuty and CloudTrail Signals

Enable GuardDuty for the region and watch for:

SageMaker:CredentialAccess/SuspiciousTrainingJob — if your GuardDuty version supports ML threat detection
CloudTrail AssumeRole events from SageMaker principals making unusual cross-service calls
GetObject calls to S3 buckets outside the expected ML data bucket from SageMaker execution roles

# CloudTrail query: SageMaker roles accessing unexpected S3 buckets
aws logs filter-log-events \
  --log-group-name aws-cloudtrail-logs \
  --filter-pattern '{ ($.userIdentity.sessionContext.sessionIssuer.userName = "SageMaker*") && ($.eventName = "GetObject") }' \
  --start-time $(date -d '24 hours ago' +%s000) \
  --query 'events[*].message' --output text | \
  python3 -c "import sys,json; [print(json.loads(l).get('requestParameters',{}).get('bucketName','')) for l in sys.stdin]" | \
  sort | uniq -c | sort -rn

Summary: Priority Remediation Checklist

Finding	Risk	Fix
`AmazonSageMakerFullAccess` on notebook role	Critical	Replace with scoped policy
IMDS v1 enabled on notebook instances	High	Force IMDSv2 via instance config
Public inference endpoints	High	Attach to VPC + resource policy
Studio domain EFS without per-user access points	Medium	Provision per-user EFS access points
`iam:PassRole` in execution role	Critical	Remove — admin operation only

SageMaker environments are trusted by security teams because they’re “just data science tooling,” which is exactly what makes them attractive targets. Apply the same IAM hygiene standards to ML environments that you apply to production workloads — they typically have broader data access and run on more powerful (and expensive) compute.