Ray AI Cluster Security: Hardening Against ShadowRay 2.0 and CVE-2023-48022

ShadowRay 2.0 is a direct consequence of a configuration problem that has been documented since 2023. Ray’s Jobs API and dashboard (port 8265 by default) have no authentication. Any host that can reach port 8265 can submit arbitrary Python code to run across the cluster with the permissions of the Ray worker process. CVE-2023-48022 formalised this as a vulnerability. Three years later, Oligo Security presented at Black Hat USA 2026 with a count of over 200,000 internet-exposed Ray servers, a self-propagating botnet that uses Ray’s own scheduling primitives to replicate across clusters, and evidence of 240GB of exfiltrated AI models, source code, and datasets.

This is not a patching problem. There is no patch that closes the exposure in the same way a CVE patch does — because the “vulnerability” is Ray’s design for trusted internal environments, combined with operators exposing it to the internet. The fix is configuration and network controls.

The Exposure Model

Ray runs a cluster with several network-accessible components:

Component	Default Port	Auth	Risk
Ray Dashboard	8265	None	Read cluster state, submit jobs
Ray Jobs API	8265/api/jobs	None	Submit arbitrary Python, execute as Ray worker
Ray GCS (Global Control Service)	6379	None	Cluster metadata, node discovery
Ray Object Store	8076	None	Read/write distributed object store
Ray Client	10001	None	Direct Python interaction with cluster

The Jobs API is the primary exploit vector. A single HTTP POST to /api/jobs/ with a JSON body containing Python code executes that code across the cluster. No authentication. No token. No configuration change needed by the attacker.

ShadowRay 2.0 extends this by using Ray’s NodeAffinitySchedulingStrategy to distribute malware payloads across every worker node in the cluster after initial access — turning a single API call into full cluster compromise.

Hardening: AWS

Security Group Configuration

The foundational fix: restrict port 8265 and the Ray cluster ports to your VPC CIDR or specific management CIDRs. Nothing else.

resource "aws_security_group_rule" "ray_dashboard_restricted" {
  type              = "ingress"
  from_port         = 8265
  to_port           = 8265
  protocol          = "tcp"
  cidr_blocks       = [var.vpc_cidr]  # Never use 0.0.0.0/0
  security_group_id = aws_security_group.ray_cluster.id
  description       = "Ray dashboard — VPC-internal only"
}

resource "aws_security_group_rule" "ray_gcs_restricted" {
  type              = "ingress"
  from_port         = 6379
  to_port           = 6379
  protocol          = "tcp"
  cidr_blocks       = [var.vpc_cidr]
  security_group_id = aws_security_group.ray_cluster.id
  description       = "Ray GCS — VPC-internal only"
}

# No internet exposure rule — if you find this in your SG, remove it
# NEVER: cidr_blocks = ["0.0.0.0/0"]

IAM Role for Ray Workers

Ray worker nodes typically need S3 access for data and model storage, and may need ECR access to pull container images. Scope their IAM role to the minimum necessary:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::your-ml-data-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": ["ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage"],
      "Resource": "arn:aws:ecr:region:account-id:repository/ray-workers"
    }
  ]
}

The ShadowRay campaign exfiltrates environment variables including AWS keys. If your Ray workers run with an instance profile that has broad permissions (S3 FullAccess, EC2 FullAccess, or worse), a compromised cluster gives attackers lateral movement into your entire cloud tenant.

Critically: do not store AWS access keys as environment variables in Ray job configurations. Use instance profiles. If you must pass credentials to Ray jobs, use AWS Secrets Manager and fetch them at runtime with short-lived tokens.

VPC Endpoint for S3

Route Ray’s S3 traffic through a VPC endpoint to prevent traffic traversing the public internet and to allow S3 bucket policies scoped to specific VPCs:

resource "aws_vpc_endpoint" "s3" {
  vpc_id       = var.vpc_id
  service_name = "com.amazonaws.${var.region}.s3"
  route_table_ids = [var.private_route_table_id]
}

resource "aws_s3_bucket_policy" "ml_data" {
  bucket = var.ml_data_bucket
  policy = jsonencode({
    Statement = [{
      Effect    = "Allow"
      Principal = { AWS = aws_iam_role.ray_worker.arn }
      Action    = ["s3:GetObject", "s3:PutObject"]
      Resource  = "${aws_s3_bucket.ml_data.arn}/*"
      Condition = {
        StringEquals = {
          "aws:sourceVpc" = var.vpc_id
        }
      }
    }]
  })
}

Hardening: Azure

Network Security Group

Equivalent to the AWS security group rule — block port 8265 from anything outside your VNET or specific management subnet:

resource nsgRayCluster 'Microsoft.Network/networkSecurityGroups@2023-04-01' = {
  name: 'nsg-ray-cluster'
  location: resourceGroup().location
  properties: {
    securityRules: [
      {
        name: 'DenyRayDashboardInternet'
        properties: {
          priority: 100
          protocol: 'Tcp'
          access: 'Deny'
          direction: 'Inbound'
          sourceAddressPrefix: 'Internet'
          destinationPortRange: '8265'
          destinationAddressPrefix: '*'
        }
      }
      {
        name: 'AllowRayDashboardVnet'
        properties: {
          priority: 110
          protocol: 'Tcp'
          access: 'Allow'
          direction: 'Inbound'
          sourceAddressPrefix: 'VirtualNetwork'
          destinationPortRange: '8265'
          destinationAddressPrefix: '*'
        }
      }
    ]
  }
}

Managed Identity for Ray Workers

Ray workers on Azure should authenticate to storage and other services via managed identity rather than stored credentials:

# In your Ray task code — use DefaultAzureCredential, not stored keys
from azure.identity import ManagedIdentityCredential
from azure.storage.blob import BlobServiceClient

credential = ManagedIdentityCredential()
blob_client = BlobServiceClient(
    account_url=f"https://{storage_account_name}.blob.core.windows.net",
    credential=credential
)

Hardening: GCP

Firewall Rules

GCP default firewall rules can allow broad ingress. Be explicit:

# firewall-deny-ray-internet.yaml
name: deny-ray-dashboard-from-internet
network: projects/PROJECT_ID/global/networks/VPC_NAME
direction: INGRESS
priority: 500
sourceRanges:
  - "0.0.0.0/0"
targetTags:
  - ray-cluster
denied:
  - IPProtocol: tcp
    ports: ["8265", "6379", "10001"]

Authentication Proxy Pattern

Ray does not natively support authentication on its dashboard or Jobs API. For organisations that need the dashboard accessible outside the VPC (for remote teams), deploy an authenticating reverse proxy in front of it.

Using nginx with OAuth2 Proxy:

server {
    listen 443 ssl;
    server_name ray-dashboard.internal.example.com;
    
    ssl_certificate /etc/ssl/certs/internal.crt;
    ssl_certificate_key /etc/ssl/private/internal.key;

    location /oauth2/ {
        proxy_pass http://oauth2-proxy:4180;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location = /oauth2/auth {
        proxy_pass http://oauth2-proxy:4180;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
    }

    location / {
        auth_request /oauth2/auth;
        error_page 401 = /oauth2/sign_in;
        proxy_pass http://ray-head:8265;
        proxy_set_header Host $host;
    }
}

With oauth2-proxy configured against your identity provider (Entra ID, Google Workspace, Okta), this gates access to the Ray dashboard behind SSO without modifying Ray itself.

Kubernetes Deployments: NetworkPolicy

If you run Ray on Kubernetes (via KubeRay), add a NetworkPolicy to restrict ingress to the Ray head pod:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ray-head-network-policy
  namespace: ray-system
spec:
  podSelector:
    matchLabels:
      ray.io/node-type: head
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ray-system
        - podSelector: {}
      ports:
        - protocol: TCP
          port: 8265
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - protocol: TCP
          port: 8265

This restricts dashboard access to pods within the ray-system namespace and a monitoring namespace, blocking any cross-namespace or external access.

Indicators of Compromise: ShadowRay 2.0

If you suspect a cluster is already compromised, look for:

Cron jobs:

crontab -l  # per user
cat /etc/cron.d/*
# Look for entries downloading from GitHub/GitLab and piping to bash, running every 15 minutes

Systemd services:

systemctl list-units --type=service | grep -E "dns-filter|health-monitor"
# ShadowRay uses these names for persistence services

Hidden binaries:

find /usr/local/bin /usr/bin /tmp -name ".*" -type f
# Leading-dot files are a common ShadowRay hiding pattern

Outbound miner connections:

ss -tnp | grep -E "3333|4444|14444|45560"
# Common XMRig pool ports
netstat -tnp | grep xmrig

Unexpected Ray jobs in queue:

ray job list  # from within the cluster
# Any jobs with names or entrypoints you don't recognise

CloudTrail / Cloud Audit Detection

In AWS, if a compromised Ray cluster exfiltrates credentials from the instance metadata service and uses them externally, CloudTrail will show API calls from unexpected source IPs:

// AWS CloudTrail via Sentinel
AWSCloudTrail
| where UserIdentityArn contains "ray-worker-role"
| where SourceIPAddress !startswith "10."  // your private range
| where SourceIPAddress !startswith "172.16."
| where EventName in ("GetObject", "ListBuckets", "DescribeInstances", "GetCallerIdentity")
| project TimeGenerated, SourceIPAddress, EventName, UserIdentityArn, Resources
| order by TimeGenerated desc