Rss icon
Sep 23 2014
Sep 22 2014
Sep 21 2014
Sep 20 2014
Sep 19 2014
Sep 18 2014
Sep 17 2014
Droplet Events Temporarily Disabled
Posted on 2014-09-17 15:00:00 UTC

We have temporarily disabled droplet events as we work to resolve issues impacting event processing. We apologize for any issue that this causes for you.

Latest Update

The issue has been resolved by our team, and droplet events have been re-enabled. Thanks for your patience, and if you have any follow up questions please contact our support team.

Read More

Updates

The issue has been resolved by our team, and droplet events have been re-enabled. Thanks for your patience, and if you have any follow up questions please contact our support team.

resolved 2014-09-17 15:08:21 UTC
Delayed Events
Posted on 2014-09-17 05:31:00 UTC

We are currently working to resolve event processing issues which may cause droplet related actions (create, destroy, etc) to be delayed. We'll provide updates as we find more information and resolve the issue.

Latest Update

After continued monitoring, the issue is now marked as resolved.

Read More

Updates

We believe this issue is fully resolved and are currently monitoring this further to ensure there are no additional issues.

monitoring 2014-09-17 06:13:15 UTC

After continued monitoring, the issue is now marked as resolved.

resolved 2014-09-17 07:13:07 UTC
Sep 16 2014
Sep 15 2014
Sep 14 2014
Sep 13 2014
NYC2: Minimal Impact Network Maintenance (2014-09-13 02:00 UTC)
Posted on 2014-09-13 01:34:00 UTC

Start: 2014-09-13 02:00 UTC (2014-09-12 20:00 EDT)
End: 2014-09-13 08:00 UTC

We will be performing some network hardware and provider maintenance in NYC2.

Expected Impact:

There may be a brief moment of latency or packet loss lasting 1-3 minutes each as we transition the network to the new hardware. Shared private networking will have no impact.

Resolution:

As part of our continued evaluation of recent outages, we have located a couple parts that have the potential to cause a future interruption. We will be proactively replacing the parts to ensure the continued stability of nyc2.

Approximately 30 minutes prior to starting any work, we will post and update our progress on the www.digitaloceanstatus.com website and keep it updated with our progress.

If you have any questions, do not hesitate to contact support.

Latest Update

We are done.

Read More

Updates

At this time we have completed all our changes and are now closely monitoring.

monitoring 2014-09-13 05:57:35 UTC

We are done.

resolved 2014-09-13 07:09:34 UTC
Sep 12 2014
Sep 11 2014
Droplet Creates Temporarily Disabled
Posted on 2014-09-11 19:54:00 UTC

At this time we are investigating an issue causing new droplet creates to fail. We have temporarily disabled droplet creates as we work to resolve the issue. We apologize for any issue that this causes for you.

Latest Update

At this time the issue has been resolved by our team, and droplet creates have been re-enabled. Thanks for your patience, and if you have any follow up questions please contact our support team.

Read More

Updates

Our engineering team has identified the issue and is working to deploy a fix. We appreciate your patience and will be providing an update soon.

update 2014-09-11 20:22:39 UTC

At this time the issue has been resolved by our team, and droplet creates have been re-enabled. Thanks for your patience, and if you have any follow up questions please contact our support team.

resolved 2014-09-11 20:47:34 UTC
Sep 10 2014
Sep 09 2014
Sep 08 2014
Event Processing Delays
Posted on 2014-09-08 16:21:00 UTC

We are currently working to resolve event processing issues which may cause droplet related actions to be delayed. We'll provide updates as we know more.

Latest Update

At this time event processing times have returned to normal. Thanks for your patience.

Read More

Updates

We have identified the source of the processing delays and are currently working to resolve it. We expect this to be resolved soon.

investigating 2014-09-08 16:29:55 UTC

At this time event processing times have returned to normal. Thanks for your patience.

resolved 2014-09-08 17:15:02 UTC
Sep 07 2014
Sep 06 2014
Sep 05 2014
Sep 04 2014
Event Processing Delays
Posted on 2014-09-04 10:57:00 UTC

We are currently working to resolve event processing delays for all droplet events across the cloud. This will affect droplet creation, power cycling, destroys, and other related events. We will provide updates as we have more information to share.

Latest Update

The event backlog has caught up completely and all systems are operational.

Read More

Updates

We have identified the source of the event processing delays and are working to repair the problem. We'll continue to post updates as we have them.

update 2014-09-04 11:33:04 UTC

We have resolved the underlying cause of the event processing delays and are continuing to monitor our event queue until everything has finished catching up.

monitoring 2014-09-04 12:02:14 UTC

The event backlog has caught up completely and all systems are operational.

resolved 2014-09-04 13:49:37 UTC
Sep 03 2014
Dashboard and API Problems
Posted on 2014-09-03 03:49:00 UTC

We are currently investigating problems with our cloud dashboard and API. We will provide additional information when we have more to share.

Latest Update

Dashboard and API performance has returned to normal.

Read More

Updates

Dashboard and API performance has returned to normal.

resolved 2014-09-03 04:00:45 UTC
Sep 02 2014
Sep 01 2014
Dashboard and API errors
Posted on 2014-09-01 21:05:00 UTC

We are currently investigating intermittent problems with the dashboard and API. We will post additional updates as we have more information to share.

Latest Update

The error rates for our sites have fallen back to normal levels. We will continue to monitor closely but believe the problems have been resolved.

Read More

Updates

The error rates for our sites have fallen back to normal levels. We will continue to monitor closely but believe the problems have been resolved.

resolved 2014-09-01 21:11:22 UTC
Aug 31 2014
Aug 30 2014
Aug 29 2014
Event latency across all regions
Posted on 2014-08-29 23:51:00 UTC

We are currently investigating event latency affecting droplets across all regions.

We will provide updates as we have more information.

Latest Update

We've resolved the issue that was causing event latency. We're sorry for the inconvenience.

Read More

Updates

We appreciate your patience. Our engineering team is still working to resolve the issue which is causing event latency across all regions.

investigating 2014-08-30 00:17:21 UTC

The primary issue is resolved. Our engineering team are currently investigating some smaller related issues.

investigating 2014-08-30 01:08:56 UTC

We've resolved the issue that was causing event latency. We're sorry for the inconvenience.

resolved 2014-08-30 01:18:53 UTC
DNS update issues
Posted on 2014-08-29 13:45:00 UTC

We are currently investigating issues affecting updates to DNS. We will provide status updates as we have more information.

Latest Update

We've identified an issue with a 3rd party provider, which has been immediately resolved. We're continuing to work with them to fully understand the problem.

Latest Update

We've resolved the issue that was causing DNS update issues. We're sorry for the inconvenience.

Read More

Updates

We've resolved the issue that was causing DNS update issues. We're sorry for the inconvenience.

investigating 2014-08-30 00:19:58 UTC
Aug 28 2014
Aug 27 2014
Community site offline
Posted on 2014-08-27 02:30:00 UTC

Our community site is currently offline. We are working on a resolution, expect an update soon.

Latest Update

We've resolved the issue that was causing the site to be offline, and it's operational again. We're sorry for the inconvenience.

Read More

Updates

We've resolved the issue that was causing the site to be offline, and it's operational again. We're sorry for the inconvenience.

resolved 2014-08-27 02:43:29 UTC
Aug 26 2014
Aug 25 2014
Aug 24 2014
Aug 23 2014
Aug 22 2014
LON1: Minimal Impact Network Maintenance (2014-08-23 00:00 UTC)
Posted on 2014-08-22 23:49:00 UTC

Start: 2014-08-23 00:00 UTC
End: 2014-08-23 05:00 UTC

We will be upgrading the software on our core routers to ensure added stability to the region.

Expected Impact:

There may be several brief periods of latency lasting 1-3 minutes each as we take a core router out of service, upgrade it and return it to service. We have full redundancy in this area and will only be doing 1 at a time to ensure full up time as the work is completed. We have tested the upgrade in our labs and consider it to be low risk.

Resolution:

Our analysis of recent outages in other regions has identified several software bugs. We are proactively updating our core routers to ensure increased stability and reliability of LON1.

Approximately 30 minutes prior to starting any work, we will post and update our progress on the www.digitaloceanstatus.com website and keep it updated with our progress.

If you have any questions, do not hesitate to contact support.

Thanks,
DigitalOcean Networking Team.

Latest Update

We have completed our upgrades.

Read More

Updates

We have completed our upgrades.

resolved 2014-08-23 01:13:35 UTC
Aug 21 2014
Aug 20 2014
SGP1: Minimal Impact Network Maintenance
Posted on 2014-08-20 15:50:00 UTC

Start: 2014-08-20 16:00 UTC
End: 2014-08-20 20:00 UTC

We will be upgrading the software on our core routers to ensure added stability to the region.

Expected Impact:

There may be several brief periods of latency lasting 1-3 minutes each as we take a core router out of service, upgrade it and return it to service. We have full redundancy in this area and will only be doing 1 at a time to ensure full up time as the work is completed. We have tested the upgrade in our labs and consider it to be low risk.

Resolution:

Our analysis of recent outages in other regions has identified several software bugs. We are proactively updating our core routers to ensure increased stability and reliability of SGP1.

Approximately 30 minutes prior to starting any work, we will post and update our progress on the www.digitaloceanstatus.com website and keep it updated with our progress.

If you have any questions, do not hesitate to contact support.

Thanks,
DigitalOcean Networking Team

Latest Update

We have completed our upgrades.

Read More

Updates

We have completed our upgrades.

resolved 2014-08-20 16:55:01 UTC
AMS2 Events Temporarily Disabled
Posted on 2014-08-20 13:34:00 UTC

At this time events in AMS2 have been temporarily disabled as our engineering team diagnoses an issue causing event processing delays. We will update you shortly with more information and appreciate your patience.

Latest Update

Events have been re-enabled in AMS2. Thanks for your patience.

Read More

Updates

We are continuing to investigate the issue which caused event processing delays. Events remain disabled as we investigate further. Thanks for your patience, we'll provide an update soon.

update 2014-08-20 13:59:58 UTC

We appreciate your patience. Our engineering team is still working to resolve the issue which caused droplet creation issues in AMS2.

update 2014-08-20 15:18:53 UTC

Events have been re-enabled in AMS2. Thanks for your patience.

resolved 2014-08-20 15:51:21 UTC
Aug 19 2014
Aug 18 2014
SFO1: Minimal impact network maintenance (2014-08-18 04:00 UTC)
Posted on 2014-08-18 03:35:00 UTC

Start: 2014-08-18 04:00 UTC
End: 2014-08-18 08:00 UTC

We will be upgrading the software on our core routers to ensure added stability to the region.

Expected Impact:

There may be several brief periods of latency lasting 1-3 minutes each as we take a core router out of service, upgrade it and return it to service. We have full redundancy in this area and will only be doing 1 at a time to ensure full up time as the work is completed. We have tested the upgrade in our labs and consider it to be low risk.

Resolution:

Our analysis of recent outages in other regions has identified several software bugs. We are proactively updating our core routers to ensure increased stability and reliability of SFO1.

Approximately 30 minutes prior to starting any work, we will post and update our progress on the www.digitaloceanstatus.com website and keep it updated with our progress.

If you have any questions, do not hesitate to contact support.

Thanks,
DigitalOcean Networking Team.

Latest Update

We have completed all work.

Read More

Updates

We are completed with our upgrades and will be monitoring the network closely for any problem.

monitoring 2014-08-18 05:19:56 UTC

We have completed all work.

resolved 2014-08-18 06:43:15 UTC
Aug 17 2014
Aug 16 2014
Partial power failure in NYC1 region
Posted on 2014-08-16 12:40:00 UTC

We are currently investigating connectivity issues affecting some droplets in our NYC 1 region. We will provide status updates as we have more information.

Latest Update

This time the NYC1 region remains stable with the exception of 5 hypervisor servers which we are working to restore. At this time we are discontinuing public status updates for this issue since it is resolved for the overwhelming majority of customers.

We will be contacting the customers with droplets on those hypervisors directly as we work to bring them back online.

If you are still having problems with your droplets and have not heard from us, please contact [email protected] so that we can troubleshoot further.

Read More

Updates

We continue to investigate limited connectivity issues with some droplets in the NYC1 region. We will update this status as more information becomes available.

update 2014-08-16 13:15:19 UTC

Our engineering and network teams are are currently recovering from a partial power outage in the NYC1 region. We do not have an ETA at this time, but will share more information as it becomes available.

update 2014-08-16 13:28:26 UTC

Our teams are continuing to work and resolve the issues related to a partial power outage in NYC1. We appreciate your patience and will provide a further update as one is available.

update 2014-08-16 13:58:02 UTC

At this time the vast majority of nodes in the NYC1 region have been returned to service. We continue to work to restore service to a few hypervisor nodes which are still having issues.

update 2014-08-16 15:38:27 UTC

This time the NYC1 region remains stable with the exception of 5 hypervisor servers which we are working to restore. At this time we are discontinuing public status updates for this issue since it is resolved for the overwhelming majority of customers.

We will be contacting the customers with droplets on those hypervisors directly as we work to bring them back online.

If you are still having problems with your droplets and have not heard from us, please contact [email protected] so that we can troubleshoot further.

resolved 2014-08-16 16:26:53 UTC
Aug 15 2014
Connectivity issues in AMS2
Posted on 2014-08-15 20:05:00 UTC

We are currently investigating connectivity issues affecting some droplets in our AMS2 region. We will provide status updates as we have more information.

Latest Update

Everything continues to behave normally. At this point, we continue to be stable and will discontinue status updates. We appreciate your support.

Read More

Updates

The connectivity problems have been resolved for several minutes but we continue to monitor the situation closely.

monitoring 2014-08-15 20:18:51 UTC

After further investigation, it appears that the issue is not fully resolved for all droplets. We continue to investigate and are working to restore service to all affected droplets as quickly as possible.

investigating 2014-08-15 20:39:19 UTC

We continue to work to resolve connectivity problems affecting a very small percentage of droplets. At this time, we are preparing some changes to our core network that we believe will help to resolve the problem. We are hopeful that we can restore service to the affected droplets within the next 30 minutes.

update 2014-08-15 21:02:01 UTC

We continue to be stable after the core networking changes we made, but we're going to monitor closely for a little longer before we close this status event. Thank you for your patience.

monitoring 2014-08-15 22:41:56 UTC

Everything continues to behave normally. At this point, we continue to be stable and will discontinue status updates. We appreciate your support.

resolved 2014-08-15 22:58:15 UTC
Aug 14 2014
Aug 13 2014
Aug 12 2014
Aug 11 2014
Aug 10 2014
Aug 09 2014
Scheduled database maintenance
Posted on 2014-08-09 11:59:00 UTC

We have begun the scheduled database maintenance planned for this morning. As a reminder, we anticipate up to 30 minutes of downtime for the control panel and API while we perform this work. Expect further updates as we have them.

Latest Update

We've successfully completed database maintenance. Thanks for your patience!

Read More

Updates

We're continuing to work on the planned maintenance. It's taking longer than expected, but we should be back online soon.

update 2014-08-09 12:31:21 UTC

Work is continuing on the planned maintenance. Things are nearly wrapped up, we're just finishing verification of the changes.

update 2014-08-09 12:55:12 UTC

We've completed our scheduled maintenance for this morning. We're continuing to monitor our systems and will post a final update.

update 2014-08-09 13:01:30 UTC

We've successfully completed database maintenance. Thanks for your patience!

resolved 2014-08-09 13:18:27 UTC
Aug 08 2014
Aug 07 2014
Aug 06 2014
Aug 05 2014
Aug 04 2014
Aug 03 2014
Aug 02 2014
Aug 01 2014
Jul 31 2014
NYC2 Power Maintenance July 31st, 2014
Posted on 2014-07-31 21:48:00 UTC

Maintenance Start: July 31st 17:00 EDT
Maintenance End: August 1st 05:00 EDT

One of our datacenter providers will be doing a power maintenance which will include de-energizing power feeds one at a time.
All of our equipment has redundant power sources therefore we do not expect this maintenance to affect any customers.
We will have staff on site to monitor the electricians progress.
Support staff will be monitoring the hypervisors as well.

Should you notice any issues with your droplets during this time please don't hesitate to contact support.

Thank you,
DigitalOcean Support

Latest Update

All circuits are re-energized and we are completed. No impact to any droplets.

Read More

Updates

All circuits are re-energized and we are completed. No impact to any droplets.

resolved 2014-08-01 05:39:06 UTC
Network Maintenance
Posted on 2014-07-31 04:20:00 UTC

We are currently preparing to perform a network maintenance in order to return the NYC2 region to full redundancy. While we do not expect this to cause any problems, but we are prepared to roll back quickly if there are any issues.

We will keep you posted as we have new information to share.

Latest Update

We have completed our maintenance window. You may have seen a brief period of up to 60 seconds of additional latency as the network reconverged. At this time all systems are working as expected and we have restored redundancy in the NYC2 region's core network.

Thank you for your patience and support.

Read More

Updates

We have just begun the first steps in our maintenance procedure. Things are going well so far. We will continue to report as we have additional information.

monitoring 2014-07-31 04:37:16 UTC

We have now restored redundancy in the network core for the NYC2 region. We will continue to monitor things very closely.

monitoring 2014-07-31 04:55:41 UTC

We have completed our maintenance window. You may have seen a brief period of up to 60 seconds of additional latency as the network reconverged. At this time all systems are working as expected and we have restored redundancy in the NYC2 region's core network.

Thank you for your patience and support.

resolved 2014-07-31 05:11:03 UTC
Jul 30 2014
Jul 29 2014
NYC2 Connection to Control Panel & Droplets
Posted on 2014-07-29 17:16:00 UTC

We have identified a slow connection to our NYC2 data center. Our team is working to resolve this and we will update you soon.

Some customers may have no connection at this time.

Latest Update

At this time the network is stable and we have removed the problematic core switch from the network. We will continue to work to fully understand and resolve the problems with that switch. Once we are comfortable that the device has been stabilized we will schedule a maintenance window to return it to production in the next couple of days to restore redundancy to our network. We are sincerely sorry for the disruption.

Read More

Updates

Due to this issue our Support Team is currently unable to reply to support tickets. Our team is standing by and will reply as soon as possible. We're sorry for the trouble and thank you for your patience.

update 2014-07-29 17:21:52 UTC

Our team is still actively working to resolve the issue in NYC2, we appreciate your patience and will provide an update soon.

update 2014-07-29 17:44:53 UTC

Our team is continuing to work to resolve the issue impacting connection to NYC2. Thank you for your patience.

update 2014-07-29 18:05:26 UTC

We are currently working to recover from a routing engine failure in our core switching infrastructure in the NYC2 region. All of our networking and site reliability team members are working hard to isolate the problem and we have engaged our vendor for additional support.

update 2014-07-29 18:38:28 UTC

We are seeing connectivity restored to our NYC2 data center. We will continue to monitor connectivity and events closely.

If you continue to see any connectivity issues please open a support ticket and our team will investigate.

monitoring 2014-07-29 19:06:28 UTC

At this time the network is stable and we have removed the problematic core switch from the network. We will continue to work to fully understand and resolve the problems with that switch. Once we are comfortable that the device has been stabilized we will schedule a maintenance window to return it to production in the next couple of days to restore redundancy to our network. We are sincerely sorry for the disruption.

resolved 2014-07-29 19:27:33 UTC
Jul 28 2014
Jul 27 2014
Jul 26 2014
Jul 25 2014
Jul 24 2014
Jul 23 2014