Training Example: Modal – Review the Data, Give Your Score & Compare to the Real AI Evaluation

[H1] AI infrastructure that developers love
Run inference, training, batch processing, and sandboxes with
sub-second cold starts, instant autoscaling, and a developer
experience that feels local. Get Started Contact Us
[H2] The production cloud for AI.
Modal SDK
[H3] Your cloud environment, in code.
Stay in Python, ship to the cloud. Composable primitives that specify everything from logic to hardware in one place. AI-native runtime
[H3] Built for speed, at any scale.
Engineered from the ground up for heavy AI workloads, with super-fast autoscaling and containers that boot instantly. Elastic cloud capacity
[H3] Autoscale from 0 to 1000+ GPUs, instantly.
Modal routes workloads across clouds and regions in real time. Get the GPUs you need in seconds, with no commitments or capacity planning. Production ready
[H3] Out-of-the-box observability.
Integrated logging and full visibility into every function, sandbox, and container. The observability tools to build robust, production-ready applications. Workloads
[H2] Build full-scale AI systems.
[H4] Inference
Deploy and scale inference for LLMs, audio, image/video generation.
[H4] Training
Fine-tune open-source models on single or multi-node clusters instantly.
[H4] Sandboxes
Programmatically scale secure, ephemeral environments for running untrusted code.
[H2] Engineered for inference.
From the proxy layer to the GPU scheduler, every part of Modal's stack is optimized for how inference workloads actually behave. Learn More
[H4] LLM Inference
Run any model or inference engine on H100s, A100s, A10Gs and more. Scale to zero between requests, burst to handle demand.
[H4] Multi-modal Inference
Image generation, video, audio, embeddings, or a model your team built from scratch. Any framework, any hardware config.
[H4] Batch and Async Inference
Run evals, embeddings, re-ranking, and dataset generation at scale. Thousands of GPUs, fully parallel, no job orchestration to manage.
[H4] Online inference
Sub-10ms overhead latency from anywhere with our globally distributed compute. Out-of-the-box support for token streaming, WebRTC, WebSocket.
[H2] Built for the full training loop.
From single-GPU fine-tuning to parallel hyperparameter sweeps to multi-node runs, Modal handles all of your coding infrastructure in a single code file. Learn More
[H4] Fine-tuning
SFT, LoRA, full fine-tunes on B200s, H100s, A100s and more. Any framework, any architecture, single or multi-GPU.
[H4] Reinforcement Learning
Thousands of concurrent trajectories, running in parallel. The only platform where sandboxes and training infrastructure are native to the same stack.
[H4] Multi-node training
Access up to 128 B200s with 3200 Gbps Infiniband networking, gang-scheduled with just a single line of code.
[H4] Parallel hyperparameter sweeps
Launch hundreds of experiments simultaneously with a few lines of code. Scale to the hardware you need, back to zero when you're done.
[H2] Designed to scale agents.
From interactive coding agents to long-running RL rollouts, Modal Sandboxes are the execution layer AI systems need: isolated, flexible, and built to scale. Learn More
[H4] Coding agents
Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products demand.
[H4] Background agents
Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.
[H4] RL rollouts
Spin up hundreds of thousands of concurrent rollout environments in seconds. Fast enough to keep your GPU inference resources saturated across every episode.
[H4] GPU-accelerated research
H100s, A100s, A10Gs available on demand. Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved capacity.
[H2] Global GPU infrastructure
Any GPU, any time Globally distributed across clouds Automated fleet health Scales with demand Any GPU, any time Globally distributed across clouds Automated fleet health Scales with demand
[H2] Security and governance
Team controls Battle-tested isolation SOC2 & HIPAA Data residency controls Learn More Team controls Battle-tested isolation SOC2 & HIPAA Data residency controls Learn More
[H2] Empowering teams of all sizes to ship at scale
Learn More
[IMG: Decagon]
65% Latency reduction
[IMG: Runway]
Real-time, multi-node inference for Runway Characters Real-time robot control running on Modal with 10–15 ms latency. 4 months faster to launch
[IMG: Chai Discovery]
ML‑driven molecular design Powering AI app generation at scale “We’re actively saving 2 engineers’ worth of ongoing time”
[IMG: Reducto]
3x latency decrease for document processing “Modal makes it easy to write code that runs on 100s of GPUs in parallel, transcribing podcasts in a fraction of the time.”
[H2] Built with Modal
All examples Audio Transcription LLM Inference Coding Agents Computational Biology Image and Video Inference
[H3] Transcribe speech in batches with Whisper
Turn audio bytes into text at scale
[H3] Voice chat with LLMs
Build an interactive voice chat app
[H3] Transcribe speech with Kyutai STT
Stream transcripts at the speed of speech
[H3] Make music
Turn prompts into music with ACE-Step
[H3] Fine-tune Whisper on domain vocab
Improve Whisper transcription accuracy on specialized vocabularies with fine-tuning
[H3] Deploy a TTS API with Chatterbox
Serve text-to-speech with Chatterbox to generate natural audio from text

[IMG: Modal]
[IMG: modal logo]
[H2] Sign up for Modal
Get started with $30 free monthly compute! Continue with GitHub
[IMG: Google]
Continue with Google Continue with SSO Already have an account? Log in By proceeding, you agree to our terms of service.
[H2] Scale inferencewith Modal
Serve custom or open source AI models with sub‑second cold starts and access to all the latest GPUs. Instant access to
thousands of GPUs Get started
for free Python, Typescript,
and Go SDKs
[H2] Run trainingwith Modal
Scale single-node or multi-node GPU
experiments instantly. Instant access to
thousands of GPUs Get started
for free Python, Typescript,
and Go SDKs
[H2] Batch processingwith Modal
Spawn millions of parallel jobs in seconds. Powered by Modal's hyper-elastic compute infrastructure. Instant access to
thousands of GPUs Get started
for free Python, Typescript,
and Go SDKs
[H2] Deploy sandboxeswith Modal
Dynamic access to secure dev environments
for agents and AI-generated code. Instant access to
thousands of GPUs Get started
for free Python, Typescript,
and Go SDKs

IntroductionCustom container images Defining ImagesUsing existing container imagesFast pull from registryGPUs and other resources GPU accelerationUsing CUDA on ModalConfiguring CPU, memory, and diskScaling out Scaling outInput concurrencyBatch processingJob queuesDynamic batchingMulti-node clusters (Beta)Deployment Apps, Functions, and entrypointsManaging deploymentsInvoking deployed functionsContinuous deploymentRunning untrusted code in FunctionsModal Sandboxes SandboxesRunning commandsNetworking and securityFile accessSnapshotsDocker in Sandboxes (Alpha)Modal NotebooksSecrets and environment variables SecretsEnvironment variablesScheduling and cron jobsWeb Functions Web FunctionsStreaming endpointsWeb Function URLsRequest timeoutsProxy Auth TokensNetworking TunnelsProxies (Beta)Cluster networkingData sharing and storage Passing local dataVolumesStoring model weightsCloud bucket mountsDictsQueuesDataset ingestionPerformance Cold start performanceMemory SnapshotsHigh-performance LLM inferenceReliability and robustness Failures and retriesPreemptionTimeoutsGPU healthTroubleshootingSecurity and privacy Security and privacyAudit logsIntegrations Using OIDC to authenticate with external servicesConnecting Modal to your Datadog accountConnecting Modal to your OpenTelemetry providerOkta SSOCustom SAML SSOSlack notifications (Beta)Workspace & account settings WorkspacesEnvironmentsModal user account setupService usersRole-Based Access Control (RBAC)BillingOther topics Feature maturityJavaScript/Go SDKsModal 1.0 migration guideFile and project structureDeveloping and debuggingDeveloping Modal code with LLMsJupyter notebooksAsynchronous API usageGlobal variablesRegion selectionContainer lifecycle hooksParametrized functionsS3 Gateway endpointsGPU Metrics Copy page
[H1] Security and privacy at Modal
The document outlines Modal’s security and privacy commitments.
[H2] Application security (AppSec)
AppSec is the practice of building software that is secure by design, secured
during development, secured with testing and review, and deployed securely. We build our software using memory-safe programming languages, including Rust
(for our worker runtime and storage infrastructure) and Python (for our API
servers and Modal client). Software dependencies are audited by Github’s Dependabot. We make decisions that minimize our attack surface. Most interactions with
Modal are well-described in a gRPC API, and occur through modal, our open-source command-line tool
and Python client library. We have automated synthetic monitoring test applications that continuously
check for network and application isolation within our runtime. We use HTTPS for secure connections. Modal forces HTTPS for all services using
TLS (SSL), including our public website and the Dashboard to ensure secure
connections. Modal’s client library connects
to Modal’s servers over TLS and verify TLS certificates on each connection. All user data is encrypted in transit and at rest. All public Modal APIs use TLS 1.3, the latest and
safest version of the TLS protocol. Internal code reviews are performed using a modern, PR-based development
workflow (Github), and engage external penetration testing firms to assess our
software security.
[H2] Corporate security (CorpSec)
CorpSec is the practice of making sure Modal employees have secure access to
Modal company infrastructure, and also that exposed channels to Modal are
secured. CorpSec controls are the primary concern of standards such as SOC2. Access to our services and applications is gated on a SSO Identity Provider
(IdP). We mandate phishing-resistant multi-factor authentication (MFA) in all
enrolled IdP accounts. We regularly audit access to internal systems. Employee laptops are protected by full disk encryption using FileVault2, and
managed by Secureframe MDM.
[H2] Network and infrastructure security (InfraSec)
InfraSec is the practice of ensuring a hardened, minimal attack surface for
components we deploy on our network. Modal uses logging and metrics observability providers, including Datadog and
Sentry.io. Compute jobs at Modal are containerized and virtualized using gVisor, the sandboxing technology
developed at Google and used in their Google Cloud Run and Google
Kubernetes Engine cloud services. We conduct annual business continuity and security incident exercises.
[H2] Vulnerability remediation
Security vulnerabilities directly affecting Modal’s systems and services will be
patched or otherwise remediated within a timeframe appropriate for the severity
of the vulnerability, subject to the public availability of a patch or other
remediation mechanisms. If there is a CVSS severity rating accompanying a vulnerability disclosure, we
rely on that as a starting point, but may upgrade or downgrade the severity
using our best judgement.
[H3] Severity timeframes
Critical: 24 hours High: 1 week Medium: 1 month Low: 3 months Informational: 3 months or longer
[H2] Shared responsibility model
Modal prioritizes the integrity, security, and availability of customer data. Under our shared responsibility model, customers also have certain responsibilities regarding data backup, recovery, and availability. Data backup: Customers are responsible for maintaining backups of their data. Performing daily backups is recommended. Customers must routinely verify the integrity of their backups. Data recovery: Customers should maintain a comprehensive data recovery plan that includes detailed procedures for data restoration in the event of data loss, corruption, or system failure. Customers must routinely test their recovery process. Availability: While Modal is committed to high service availability, customers must implement contingency measures to maintain business continuity during service interruptions. Customers are also responsible for the reliability of their own IT infrastructure. Security measures: Customers must implement appropriate security measures, such as encryption and access controls, to protect their data throughout the backup, storage, and recovery processes. These processes must comply with all relevant laws and regulations.
[H2] SOC 2
We have successfully completed a System and Organization Controls (SOC) 2 Type 2
audit. Go to our Security Portal to request access to the report.
[H2] HIPAA
HIPAA, which stands for the Health Insurance Portability and Accountability Act, establishes a set of standards that protect health information, including individuals’ medical records and other individually identifiable health information. HIPAA guidelines apply to both covered entities and business associates—of which Modal is the latter if you are processing PHI on Modal. Modal’s services can be used in a HIPAA compliant manner. It is important to note that unlike other security standards, there is no officially recognized certification process for HIPAA compliance. Instead, we demonstrate our compliance with regulations such as HIPAA via the practices outlined in this doc, our technical and operational security measures, and through official audits for standards compliance such as SOC 2 certification. To use Modal services for HIPAA-compliant workloads, a Business Associate Agreement (BAA) should be established with us prior to submission of any PHI. This is available on our Enterprise plan. Contact us at security@modal.com to get started. At the moment, Volumes v1, Images (excluding Filesystem and Directory Snapshots), Memory Snapshots, and user code are out of scope of the commitments within our BAA, so PHI should not be used in those areas of the product. Volumes v2 are HIPAA compliant.
[H2] PCI
Payment Card Industry Data Security Standard (PCI) is a standard that defines
the security and privacy requirements for payment card processing. Modal uses Stripe to securely process transactions and
trusts their commitment to best-in-class security. We do not store personal
credit card information for any of our customers. Stripe is certified as “PCI
Service Provider Level 1”, which is the highest level of certification in the
payments industry.
[H2] Bug bounty program
Keeping user data secure is a top priority at Modal. We welcome contributions
from the security community to identify vulnerabilities in our product and
disclose them to us in a responsible manner. We currently run a private bug
bounty program through HackerOne. If you have found a vulnerability and
wish to participate, please send an email to security@modal.com with your
HackerOne username or email and we will invite you to the program.
[H2] Data privacy
Modal will never access or use: your source code. the inputs (function arguments) or outputs (function return values) to your Modal Functions. any data you store in Modal, such as in Images or Volumes. App logs and metadata are stored on Modal. Modal will not access this data
unless permission is granted by the user to help with troubleshooting.
[H3] Data retention
Different Modal products have different retention policies for the data they
handle. The table below summarizes how long each type of data is retained. DataProductRetention Inputs and outputsFunctions (.remote, .spawn, .map, Web Functions, Scheduled Functions)Up to 7 days, then deletedRequest and response payloadsModal Inference endpointsNot stored — proxied directly to your containerApp and container logsFunctions, SandboxesPlan-dependent: 1 day on Starter, 30 days on Team, configurable on Enterprise (see pricing)Audit logsWorkspace (Enterprise)Per Enterprise contract (see Audit logs)FilesVolumes, ImagesPersistent until you delete themMemory snapshotsFunction memory snapshots, Sandbox memory snapshots7 days after creationFilesystem snapshotsSandbox filesystem snapshotsPersistent until you delete them (stored as Images)Directory snapshotsSandbox directory snapshots30 days after last creation or useEntriesDicts7 days after last read or writePartitionsQueuesConfigurable per-partition TTL (default 24 hours)App, Function, and container metadataAll productsStored for the lifetime of your account
[H4] Function inputs and outputs
Function inputs and outputs are stored encrypted at rest. Small payloads
(≤ 2 MiB) are stored inline in our metadata store; larger payloads are stored
in object storage. Both are deleted within a maximum TTL of 7 days.
[H4] Modal Inference endpoints
Modal Inference endpoints are zero data retention: request and response payloads
are never written to disk and pass through Modal’s infrastructure only as
in-flight network traffic. Inference endpoints terminate TLS at Modal’s edge
proxy and forward requests directly to your containers over an internal tunnel.
[H2] Questions?
Email us! Security and privacy at ModalApplication security (AppSec)Corporate security (CorpSec)Network and infrastructure security (InfraSec)Vulnerability remediationSeverity timeframesShared responsibility modelSOC 2HIPAAPCIBug bounty programData privacyData retentionFunction inputs and outputsModal Inference endpointsQuestions?

[IMG: Applied Compute]
[H3] Reinforcement learning at scale with Modal
[H3] How Ramp built a full context background coding agent on Modal
Customers use Modal for Language Models Image, Video, 3D Audio Processing Fine-Tuning Batch Processing Sandboxed Code Computational Bio Coding Agents Language Models Image, Video, 3D Audio Processing Fine-Tuning Batch Processing Sandboxed Code Computational Bio Coding Agents
[IMG: DoorDash]
“As we scale agentic commerce for local businesses, we need a highly efficient path to production… We’re excited to evaluate Claude Managed Agents for this next step, building on our AI infrastructure with Modal.”
[IMG: Cognition]
“Modal powers both our reinforcement learning infrastructure and production inference. Millions of sandboxes on one end, real-time serving on the other.”
[IMG: Decagon]
65% Latency reduction
[IMG: Runway]
Real-time, multi-node inference for Runway Characters Real-time robot control running on Modal with 10–15 ms latency. 4 months faster to launch
[IMG: Chai Discovery]
ML‑driven molecular design Powering AI app generation at scale “We’re actively saving 2 engineers’ worth of ongoing time”
[IMG: Reducto]
3x latency decrease for document processing “Modal makes it easy to write code that runs on 100s of GPUs in parallel, transcribing podcasts in a fraction of the time.”