Monitoring and Debugging

This section covers how to monitor and debug Ray applications and clusters.

See Getting Help if your problem is not solved by the troubleshooting guide.

• Overview
  • Dashboard (Web UI)
  • Application Logging
  • Exceptions
  • Accessing Ray States
  • Debugger
  • Monitoring Cluster State and Resource Demands
  • Metrics
  • Profiling
  • Tracing
• Ray Dashboard
  • Getting Started
  • Views
    • Node View
    • Jobs View
    • Actors View
    • Logs view
    • Metrics View
    • Event View
  • Advanced Usage
  • Viewing built-in dashboard API metrics
    • Debugging ObjectStoreFullError and Memory Leaks
    • Running Behind a Reverse Proxy
  • References
    • Node View
    • Jobs View
    • Actors
    • Logs
• Monitoring Ray States
  • Getting Started
  • Key Concepts
  • Summary
    • E.g., Summarize all actors
    • E.g., Summarize all tasks
    • E.g., Summarize all objects
  • List
    • E.g., List all nodes
    • E.g., List all placement groups
    • E.g., List local referenced objects created by a process
    • E.g., List alive actors
    • E.g., List running tasks
    • E.g., List non-running tasks
    • E.g., List running tasks that have a name func
    • E.g., List tasks with more details
  • Get
    • E.g., Get a task info
    • E.g., Get a node info
  • Logs
    • E.g., Get all retrievable log file names from a head node in a cluster
    • E.g., Get a particular log file from a node
    • E.g., Stream a log file from a node
    • E.g., Stream log from an actor with actor id
    • E.g., Stream log from a pid
  • Failure Semantics
    • Query Failures
    • Data Truncation
    • Garbage Collected Resources
  • API Reference
• Ray Debugger
  • Getting Started
  • Running on a Cluster
  • Debugger Commands
  • Stepping between Ray tasks
  • Post Mortem Debugging
  • Debugging APIs
• Logging
  • Driver logs
  • Job logs
  • Worker logs
  • Disabling logging to the driver
  • Customizing Actor logs prefixes
  • How to set up loggers
  • How to use structured logging
    • Logging directory structure
    • Log rotation
  • Redirecting Ray logs to stderr
• Metrics
  • Getting Started
  • Grafana
    • Deploying Grafana
  • System Metrics
    • Metrics Semantics and Consistency
  • Application-level Metrics
  • Configurations
    • Customize prometheus export port
    • Alternate Prometheus host location
    • Alternate Grafana host location
  • Troubleshooting
    • Mac does not trust the developer when installing prometheus or grafana
    • Grafana dashboards are not embedded in the Ray dashboard
• Profiling
  • Visualizing Tasks in the Ray Timeline
  • Python CPU Profiling in the Dashboard
  • Profiling Using Python’s CProfile
    • Profiling Ray Actors with cProfile
  • Profiling (Internal)
• Tracing
  • Getting Started
  • Tracing Startup Hook
    • Tracer Provider
    • Remote Span Processors
    • Additional Instruments
      • Custom Traces
• Troubleshooting Failures
  • Crashes
• Troubleshooting Hangs
  • Observing Ray Work
• Troubleshooting Performance
  • No Speedup
• Ray Gotchas
  • Environment variables are not passed from the driver to workers
  • Filenames work sometimes and not at other times
  • Placement groups are not composable
  • Outdated Function Definitions
• Getting Help
  • Ray Community
    • Discourse Forum
    • Bugs or Feature Requests
    • StackOverflow
    • Slack
• Debugging (internal)
  • Starting processes in a debugger
  • Backend logging
  • Backend event stats
  • Callback latency injection
• Profiling (internal)
  • Getting a stack trace of Ray C++ processes
  • Installation
  • Launching the to-profile binary
  • Memory Profiling
  • Visualizing the CPU profile
  • Running Microbenchmarks
  • References