Modal llms.txt

Modal is a platform for running Python code in the cloud with minimal configuration, especially for serving AI models and high-performance batch processing. It supports fast prototyping, serverless APIs, scheduled jobs, GPU inference, distributed volumes, and sandboxes.

Important notes:

• Modal's primitives are embedded in Python and tailored for AI/GPU use cases, but they can be used for general-purpose cloud compute.
• Modal is a serverless platform, meaning you are only billed for resources used and can spin up containers on demand in seconds.

You can sign up for free at [https://modal.com] and get $30/month of credits.

Guide

• Introduction
• Custom container images
  • Defining Images
  • Using existing container images
  • Fast pull from registry
• GPUs and other resources
  • GPU acceleration
  • Using CUDA on Modal
  • Configuring CPU, memory, and disk
• Scaling out
  • Scaling out
  • Input concurrency
  • Batch processing
  • Job queues
  • Dynamic batching
  • Multi-node clusters (Beta)
• Deployment
  • Apps, Functions, and entrypoints
  • Managing deployments
  • Invoking deployed functions
  • Continuous deployment
  • Running untrusted code in Functions
• Modal Sandboxes
  • Sandboxes
  • Running commands
  • Networking and security
  • File access
  • Snapshots
  • Pricing and resources
  • Docker in Sandboxes (Alpha)
  • VM Sandboxes (Alpha)
• Modal Notebooks
• Secrets and environment variables
  • Secrets
  • Environment variables
• Scheduling and cron jobs
• Web Functions
  • Web Functions
  • Streaming endpoints
  • Web Function URLs
  • Request timeouts
  • Proxy Auth Tokens
• Networking
  • Tunnels
  • Proxies (Beta)
  • Cluster networking
• Data sharing and storage
  • Passing local data
  • Volumes
  • Storing model weights
  • Cloud bucket mounts
  • Dicts
  • Queues
  • Dataset ingestion
• Performance
  • Cold start performance
  • Memory Snapshots
  • High-performance LLM inference
• Reliability and robustness
  • Failures and retries
  • Preemption
  • Timeouts
  • GPU health
  • Troubleshooting
• Security and privacy
  • Security and privacy
  • Customer-supplied encryption keys
  • Audit logs
• Integrations
  • Using OIDC to authenticate with external services
  • Connecting Modal to your Datadog account
  • Connecting Modal to your OpenTelemetry provider
  • Okta SSO
  • Custom SAML SSO
  • Slack notifications (Beta)
• Workspace & account settings
  • Workspaces
  • Environments
  • Modal user account setup
  • Service users
  • Role-Based Access Control (RBAC)
  • Billing
• Other topics
  • Feature maturity
  • JavaScript/Go SDKs
  • Modal 1.0 migration guide
  • File and project structure
  • Developing and debugging
  • Developing Modal code with LLMs
  • Jupyter notebooks
  • Asynchronous API usage
  • Global variables
  • Region selection
  • Container lifecycle hooks
  • Parametrized functions
  • Dynamic function configuration
  • S3 Gateway endpoints
  • GPU Metrics

Examples

• Featured
• Getting started
  • Hello, world
  • Simple web scraper
  • Serving Web Functions
• Large language models (LLMs)
  • Deploy an OpenAI-compatible LLM service with vLLM
  • Cut Ministral 3 cold start times by 10x with snapshots
  • Maximize tokens per second in batch processing with vLLM
  • Serve an ultra-low-latency chatbot with SGLang
  • Deploy Nemotron 3
  • Efficient LLM Finetuning with Unsloth
  • Run a multimodal RAG chatbot to answer questions about PDFs
  • Fine-tune an LLM to replace your CEO
  • Deploy a stateless MCP with FastMCP
• Images, video, & 3D
  • Edit images with Flux Kontext
  • Fine-tune Wan2.1 video models on your face
  • Run Flux fast with torch.compile
  • Fine-tune Flux with LoRA
  • Animate images with LTX-Video
  • Generate video clips with LTX-Video
  • Run Stable Diffusion with a CLI, API, and web UI
• Audio
  • Deploy a Moshi voice chatbot
  • Stream transcripts at the speed of speech using Kyutai STT
  • Make music with ACE-Step
  • Generate speech with Chatterbox
  • Run high throughput batched transcription with Whisper
  • Fine-tune Whisper to recognize new words
• Real-time communication (WebRTC)
  • Serverless WebRTC
  • WebRTC quickstart with FastRTC
• Computational biology
  • Design protein binders with ESMFold2
  • Fold proteins with Chai-1
  • Build a protein-folding dashboard
  • Fold proteins with Boltz-2
• Modal Sandboxes
  • Run a background coding agent with OpenCode
  • Build a scalable AI coding platform
  • Create GIFs from Slack using the Claude Agent SDK
  • Run a LangGraph agent's code in a secure GPU sandbox
  • Control a sandboxed computer with an LLM
  • Build a stateful, sandboxed code interpreter
  • Run Node.js, Ruby, and more in a Sandbox
  • Speed up Sandbox starts with warm pools
• Reinforcement Learning
  • Train a model to solve math problems using GRPO and verl
  • Train a model to solve coding problems using GRPO and TRL
• Embeddings
  • Embed millions of documents with TEI
  • Turn satellite images into vectors and store them in MongoDB
• Parallel processing and job scheduling
  • Deploy a Hacker News Slackbot
  • Run a Document OCR job queue
  • Serve a Document OCR web app
• Training models from scratch
  • Train an SLM with early-stopping grid search over hyperparameters
  • Run long, resumable training jobs
• Hosting popular libraries
  • YOLO: Fine-tune and serve computer vision models
  • Blender: Build a 3D render farm
  • Streamlit: Run and deploy Streamlit apps
  • SQLite: Publish explorable data with Datasette
  • Algolia: Build docsearch with a crawler
• Connecting to other APIs
  • Discord: Deploy and run a Discord Bot
  • Google Sheets: Sync databases and APIs to a Google Sheet
  • OpenAI: Run a RAG Q&A chatbot
  • Tailscale: Add Modal Apps to your VPN
  • Prometheus: Publish custom metrics with Pushgateway
• Managing data
  • Mount S3 buckets in Modal apps
  • Build your own data warehouse with DuckDB, DBT, and Modal
  • Create a LoRA Playground with Modal, Gradio, and S3
• Miscellaneous

API Reference

• Changelog
• Application construction
  • App: The main unit of deployment for code on Modal
  • App.function: Decorator for registering a function with an App
  • App.cls: Decorator for registering a class with an App
• Serverless execution
  • Function: A serverless function backed by an autoscaling container pool
  • Cls: A serverless class supporting parametrization and lifecycle hooks
• Extended Function configuration
  • Class parametrization
    • parameter: Used to define class parameters, akin to a Dataclass field
  • Lifecycle hooks
    • enter: Decorator for a method that will be executed during container startup
    • exit: Decorator for a method that will be executed during container shutdown
    • method: Decorator for exposing a method as an invokable function
  • Web integrations
    • fastapi_endpoint: Decorator for exposing a simple FastAPI-based endpoint
    • asgi_app: Decorator for functions that construct an ASGI web application
    • wsgi_app: Decorator for functions that construct a WSGI web application
    • web_server: Decorator for functions that construct an HTTP web server
  • Function semantics
    • batched: Decorator that enables dynamic input batching
    • concurrent: Decorator that enables input concurrency
  • Scheduling
    • Cron: A schedule that runs based on cron syntax
    • Period: A schedule that runs at a fixed interval
  • Exception handling
    • Retries: Function retry policy for input failures
• Sandboxed execution
  • Sandbox: An interface for restricted code execution
  • ContainerProcess: An object representing a sandboxed process
  • FileIO: A handle for a file in the Sandbox filesystem
• Container configuration
  • Image: An API for specifying container images
  • Secret: A pointer to secrets that will be exposed as environment variables
• Data primitives
  • Persistent storage
    • Volume: Distributed storage supporting highly performant parallel reads
    • CloudBucketMount: Storage backed by a third-party cloud bucket (S3, etc.)
    • NetworkFileSystem: Shared, writeable cloud storage (superseded by modal.Volume)
  • In-memory storage
    • Dict: A distributed key-value store
    • Queue: A distributed FIFO queue
• Networking
  • Proxy: An object that provides a static outbound IP address for containers
  • forward: A context manager for publicly exposing a port from a container