# Pandium Status - Incident History ## API incident due to cloud provider maintenance ### May 16, 2026 12:50 EDT **Resolved** - This incident has been resolved and we are continuing to monitor. We apologize for the inconvenience. ### May 16, 2026 12:40 EDT **Identified** - Our US environments are experiencing instability stemming from a maintenance task by an upstream provider. We are working to mitigate and will update shortly. ## API incident in EU region ### May 16, 2026 12:04 EDT **Resolved** - This issue has been resolved and we are continuing to monitor. Apologies for the inconvenience. ### May 16, 2026 11:55 EDT **Update** - A fix has been implemented and we are monitoring. ### May 16, 2026 11:30 EDT **Identified** - The issue has been identified and we are working on a fix. ### May 16, 2026 11:28 EDT **Investigating** - We are currently investigating an issue and will update when we know more. ## EU environments experiencing instability ### January 22, 2026 16:17 EST **Resolved** - This incident has been resolved. ### January 22, 2026 13:53 EST **Monitoring** - A fix has been implemented and we are monitoring to assure peak efficiency. ### January 22, 2026 12:47 EST **Update** - We are continuing to monitor recovery and will update as soon as we have further details. ### January 22, 2026 11:13 EST **Update** - We are continuing to monitor the fix as we recover. ### January 22, 2026 10:35 EST **Update** - We are currently implementing a fix for the issue. ### January 22, 2026 09:59 EST **Identified** - An issue has been identified and we are working on a fix. ### January 22, 2026 09:51 EST **Investigating** - We are currently investigating an incident affecting our EU environments. Logging into the Integration Hub and runs are affected. We are actively investigating and will update when we know more. ## Services inaccessible ### June 12, 2025 17:21 EDT **Resolved** - This issue has been resolved and all services are caught up and fully operational. We will continue to monitor to ensure stability. ### June 12, 2025 16:50 EDT **Monitoring** - All systems are back online and we are monitoring the recovery of the backlog of jobs. ### June 12, 2025 15:43 EDT **Update** - EU environments have recovered and we are continuing to monitor. US environments remain affected by the outage. ### June 12, 2025 15:20 EDT **Update** - Runs are firing with delays in our EU environment. We are monitoring. ### June 12, 2025 14:37 EDT **Update** - The issue with our cloud hosting provider is ongoing. We are continuing to monitor. ### June 12, 2025 14:13 EDT **Identified** - All services are inaccessible or responding slowly due to an issue with our cloud hosting provider. We will update as soon as we know more. ## Some runs firing with delays ### March 21, 2025 10:24 EDT **Resolved** - This issue has been resolved and runs are processing normally. ### March 21, 2025 10:00 EDT **Identified** - We have identified the issue and deployed a fix. ### March 21, 2025 09:36 EDT **Investigating** - Some runs in the US region are firing with delays. We are investigating. ## Runs processing with delays ### November 1, 2024 19:13 EDT **Resolved** - This issue has been resolved. ### November 1, 2024 19:09 EDT **Monitoring** - A fix has been implemented and we are monitoring recovery. ### November 1, 2024 19:03 EDT **Identified** - The issue has been identified and a fix is being implemented. ### November 1, 2024 19:02 EDT **Update** - The issue with runs has been identified and we are implementing a fix. No other services are affected. ### November 1, 2024 18:49 EDT **Investigating** - Runs for some US-Based accounts are processing with delays. We are investigating. ## Runs slowed ### October 1, 2024 15:30 EDT **Resolved** - This incident has been resolved. ### October 1, 2024 15:25 EDT **Investigating** - We are currently investigating an issue in which some runs are firing with delays. ## Some runs processing with delays ### September 27, 2024 10:47 EDT **Resolved** - This incident has been resolved and we are continuing to monitor. ### September 27, 2024 09:35 EDT **Identified** - The issue has been identified and we are working on a fix. ### September 27, 2024 09:15 EDT **Investigating** - We are investigating. ## Runs slowed ### September 26, 2024 15:52 EDT **Resolved** - This incident has been resolved. ### September 26, 2024 15:12 EDT **Monitoring** - A fix has been implemented and we are monitoring. ### September 26, 2024 13:35 EDT **Identified** - Runs are starting with delays as we investigate the underlying cause. ## Runs degraded ### September 21, 2024 21:35 EDT **Resolved** - The underlying service issue causing instability has been resolved and runs are firing. We are continuing to monitor for the next few hours and will open a new incident if necessary. ### September 21, 2024 21:09 EDT **Monitoring** - A fix has been implemented and runs are recovering. We are closely monitoring and will update when fully resolved. ### September 21, 2024 19:35 EDT **Update** - The Pandium Integration Hub is fully operational however jobs are still running with delays. We will provide an update as soon as possible. ### September 21, 2024 16:03 EDT **Update** - We are continuing to work on a fix for this issue. ### September 21, 2024 14:36 EDT **Identified** - We have identified that there is an issue with our underlying cloud hosting provider and we are working with them to implement a fix. They have escalated this issue and we will provide an update as soon as possible. ## API responses exceeding error limits ### July 25, 2024 02:39 EDT **Resolved** - Continuing to monitor. All systems green. ### July 25, 2024 02:27 EDT **Monitoring** - We have increased replicas and are seeing a reduction in error counts. Monitoring. ### July 25, 2024 02:12 EDT **Investigating** - API responses exceeding error limits. We are investigating. ## Degraded Performance ### July 14, 2024 17:42 EDT **Resolved** - Scheduled run runs have caught up, and we have not experienced any more issues with control plane. ### July 14, 2024 13:37 EDT **Monitoring** - A fix has been implemented. We are now monitoring recovery. ### July 14, 2024 12:35 EDT **Update** - The underlying issue has been identified. Ectd cluster of the control plane is having a fit. Working on mitigation. ### July 14, 2024 11:53 EDT **Identified** - Issue is with the control plane of underlying hosting provider. We are working with them on mitigation. ### July 14, 2024 03:06 EDT **Investigating** - Underlying Control Plane under high load. Investigating. Run and API are running slow. ## Infrastructure Upgrades ### December 17, 2022 11:23 EST **Completed** - Maintenance is complete! Have a good weekend y'all! ### December 17, 2022 11:20 EST **Verifying** - Verification is currently underway for the maintenance items. ### December 17, 2022 09:00 EST **In Progress** - Scheduled maintenance is currently in progress. We will provide updates as necessary. ### December 16, 2022 15:50 EST **Scheduled** - We will be doing scheduled upgrades in production between 9am-1pm EST on Saturday 12/17. During this time, scheduled jobs will continue to run, however there may be intermittent slowness in the UI. If this occurs, we expect this to resolve in under 1 minute.