System Status

This incident has been resolved.

We've freed up additional room in the SIN and AMS regions and are monitoring capacity.

We've freed up additional room in the SIN and AMS regions and are monitoring capacity.

We are currently investigating capacity issues in SIN and AMS regions that are affecting: - Machine Create and Start events - Deployments, due to affected, degraded Remote Builders - Sprite startup from cold state

This may also affect: - Remote builders in AMS and SIN regions, which could currently be experiencing degraded performance or failures. - Sprites starting from a cold state, which may experience failures in starting

We are currently investigating elevated errors when creating and starting machines in the SIN and AMS regions. Choosing other regions to create or deploy may help in the meantime

This incident has been resolved.

With the additional capacity we've brought online, machine start failure rates in IAD have now recovered. We'll continue to monitor IAD capacity.

We've brought some additional capacity online in IAD and are seeing improvements, and we're continuing to work on adding more and freeing up additional room.

We're continuing to evaluate our options for increasing short-term capacity in the IAD region.

We're currently investigating capacity issues in IAD that is preventing machine starts (machine creates are currently unaffected). This may result in deploys failing to complete (even for apps outside of the IAD region). As a workaround, using legacy Fly builders explicitly located in another region (i.e., `FLY_REMOTE_BUILDER_REGION=lhr fly deploy --depot=false --recreate-builder`) may help in the meantime.

This incident has been resolved.

We've implemented a fix and have seen error rates for machine creates in ORD drop off. We're continuing to monitor the results.

We've identified the cause of this increased failure rate and a fix is in progress. We are seeing most creates in ORD succeed at this time, though failure rate is still above baseline.

We are continuing to investigate this issue. We are seeing 408 errors decreasing in ORD, though still above baseline.

We are currently investigating elevated errors creating machines in the ORD (Chicago, Illinois) region. Users may see `failed to launch VM: request returned non-2xx status: 408` errors when creating, updating, or scaling machines in ORD. Existing, already running machines in the ORD region continue to run as normal.

This incident has been resolved.

Some Managed Postgres clusters in FRA region are still unreachable, we are investigating this issue.

Apps and Managed Postgres clusters in FRA region should be back online at this time. We are monitoring for any further issues.

We are investigating network issues in FRA region. Apps and/or Managed Postgres clusters in the region may be inaccessible at this time.

This incident is resolved, Grafana logs are now working properly.

We've deployed a fix and are monitoring the results. Logs are now be visible on Grafana.

Using the Logs panel in Grafana at https://fly-metrics.net/ will show a 502 error from the backend and won't show any logs. You can use `fly logs` or the live log viewer directly on https://fly.io/dashboard to view streaming logs for the time being.

Using the Logs panel in Grafana at https://fly-metrics.net/ will show a 502 error from the backend and won't show any logs. You can use `fly logs` or the live log viewer directly on https://fly.io/dashboard to view streaming logs for the time being.