Rss icon
Aug 27 2014
Community site offline
Posted on 2014-08-27 02:30:00 UTC

Our community site is currently offline. We are working on a resolution, expect an update soon.

Latest Update

We've resolved the issue that was causing the site to be offline, and it's operational again. We're sorry for the inconvenience.

Read More

Updates

We've resolved the issue that was causing the site to be offline, and it's operational again. We're sorry for the inconvenience.

resolved 2014-08-27 02:43:29 UTC
Aug 26 2014
Aug 25 2014
Aug 24 2014
Aug 23 2014
Aug 22 2014
LON1: Minimal Impact Network Maintenance (2014-08-23 00:00 UTC)
Posted on 2014-08-22 23:49:00 UTC

Start: 2014-08-23 00:00 UTC
End: 2014-08-23 05:00 UTC

We will be upgrading the software on our core routers to ensure added stability to the region.

Expected Impact:

There may be several brief periods of latency lasting 1-3 minutes each as we take a core router out of service, upgrade it and return it to service. We have full redundancy in this area and will only be doing 1 at a time to ensure full up time as the work is completed. We have tested the upgrade in our labs and consider it to be low risk.

Resolution:

Our analysis of recent outages in other regions has identified several software bugs. We are proactively updating our core routers to ensure increased stability and reliability of LON1.

Approximately 30 minutes prior to starting any work, we will post and update our progress on the www.digitaloceanstatus.com website and keep it updated with our progress.

If you have any questions, do not hesitate to contact support.

Thanks,
DigitalOcean Networking Team.

Latest Update

We have completed our upgrades.

Read More

Updates

We have completed our upgrades.

resolved 2014-08-23 01:13:35 UTC
Aug 21 2014
Aug 20 2014
SGP1: Minimal Impact Network Maintenance
Posted on 2014-08-20 15:50:00 UTC

Start: 2014-08-20 16:00 UTC
End: 2014-08-20 20:00 UTC

We will be upgrading the software on our core routers to ensure added stability to the region.

Expected Impact:

There may be several brief periods of latency lasting 1-3 minutes each as we take a core router out of service, upgrade it and return it to service. We have full redundancy in this area and will only be doing 1 at a time to ensure full up time as the work is completed. We have tested the upgrade in our labs and consider it to be low risk.

Resolution:

Our analysis of recent outages in other regions has identified several software bugs. We are proactively updating our core routers to ensure increased stability and reliability of SGP1.

Approximately 30 minutes prior to starting any work, we will post and update our progress on the www.digitaloceanstatus.com website and keep it updated with our progress.

If you have any questions, do not hesitate to contact support.

Thanks,
DigitalOcean Networking Team

Latest Update

We have completed our upgrades.

Read More

Updates

We have completed our upgrades.

resolved 2014-08-20 16:55:01 UTC
AMS2 Events Temporarily Disabled
Posted on 2014-08-20 13:34:00 UTC

At this time events in AMS2 have been temporarily disabled as our engineering team diagnoses an issue causing event processing delays. We will update you shortly with more information and appreciate your patience.

Latest Update

Events have been re-enabled in AMS2. Thanks for your patience.

Read More

Updates

We are continuing to investigate the issue which caused event processing delays. Events remain disabled as we investigate further. Thanks for your patience, we'll provide an update soon.

update 2014-08-20 13:59:58 UTC

We appreciate your patience. Our engineering team is still working to resolve the issue which caused droplet creation issues in AMS2.

update 2014-08-20 15:18:53 UTC

Events have been re-enabled in AMS2. Thanks for your patience.

resolved 2014-08-20 15:51:21 UTC
Aug 19 2014
Aug 18 2014
SFO1: Minimal impact network maintenance (2014-08-18 04:00 UTC)
Posted on 2014-08-18 03:35:00 UTC

Start: 2014-08-18 04:00 UTC
End: 2014-08-18 08:00 UTC

We will be upgrading the software on our core routers to ensure added stability to the region.

Expected Impact:

There may be several brief periods of latency lasting 1-3 minutes each as we take a core router out of service, upgrade it and return it to service. We have full redundancy in this area and will only be doing 1 at a time to ensure full up time as the work is completed. We have tested the upgrade in our labs and consider it to be low risk.

Resolution:

Our analysis of recent outages in other regions has identified several software bugs. We are proactively updating our core routers to ensure increased stability and reliability of SFO1.

Approximately 30 minutes prior to starting any work, we will post and update our progress on the www.digitaloceanstatus.com website and keep it updated with our progress.

If you have any questions, do not hesitate to contact support.

Thanks,
DigitalOcean Networking Team.

Latest Update

We have completed all work.

Read More

Updates

We are completed with our upgrades and will be monitoring the network closely for any problem.

monitoring 2014-08-18 05:19:56 UTC

We have completed all work.

resolved 2014-08-18 06:43:15 UTC
Aug 17 2014
Aug 16 2014
Partial power failure in NYC1 region
Posted on 2014-08-16 12:40:00 UTC

We are currently investigating connectivity issues affecting some droplets in our NYC 1 region. We will provide status updates as we have more information.

Latest Update

This time the NYC1 region remains stable with the exception of 5 hypervisor servers which we are working to restore. At this time we are discontinuing public status updates for this issue since it is resolved for the overwhelming majority of customers.

We will be contacting the customers with droplets on those hypervisors directly as we work to bring them back online.

If you are still having problems with your droplets and have not heard from us, please contact support@digitalocean.com so that we can troubleshoot further.

Read More

Updates

We continue to investigate limited connectivity issues with some droplets in the NYC1 region. We will update this status as more information becomes available.

update 2014-08-16 13:15:19 UTC

Our engineering and network teams are are currently recovering from a partial power outage in the NYC1 region. We do not have an ETA at this time, but will share more information as it becomes available.

update 2014-08-16 13:28:26 UTC

Our teams are continuing to work and resolve the issues related to a partial power outage in NYC1. We appreciate your patience and will provide a further update as one is available.

update 2014-08-16 13:58:02 UTC

At this time the vast majority of nodes in the NYC1 region have been returned to service. We continue to work to restore service to a few hypervisor nodes which are still having issues.

update 2014-08-16 15:38:27 UTC

This time the NYC1 region remains stable with the exception of 5 hypervisor servers which we are working to restore. At this time we are discontinuing public status updates for this issue since it is resolved for the overwhelming majority of customers.

We will be contacting the customers with droplets on those hypervisors directly as we work to bring them back online.

If you are still having problems with your droplets and have not heard from us, please contact support@digitalocean.com so that we can troubleshoot further.

resolved 2014-08-16 16:26:53 UTC
Aug 15 2014
Connectivity issues in AMS2
Posted on 2014-08-15 20:05:00 UTC

We are currently investigating connectivity issues affecting some droplets in our AMS2 region. We will provide status updates as we have more information.

Latest Update

Everything continues to behave normally. At this point, we continue to be stable and will discontinue status updates. We appreciate your support.

Read More

Updates

The connectivity problems have been resolved for several minutes but we continue to monitor the situation closely.

monitoring 2014-08-15 20:18:51 UTC

After further investigation, it appears that the issue is not fully resolved for all droplets. We continue to investigate and are working to restore service to all affected droplets as quickly as possible.

investigating 2014-08-15 20:39:19 UTC

We continue to work to resolve connectivity problems affecting a very small percentage of droplets. At this time, we are preparing some changes to our core network that we believe will help to resolve the problem. We are hopeful that we can restore service to the affected droplets within the next 30 minutes.

update 2014-08-15 21:02:01 UTC

We continue to be stable after the core networking changes we made, but we're going to monitor closely for a little longer before we close this status event. Thank you for your patience.

monitoring 2014-08-15 22:41:56 UTC

Everything continues to behave normally. At this point, we continue to be stable and will discontinue status updates. We appreciate your support.

resolved 2014-08-15 22:58:15 UTC
Aug 14 2014
Aug 13 2014
Aug 12 2014
Aug 11 2014
Aug 10 2014
Aug 09 2014
Scheduled database maintenance
Posted on 2014-08-09 11:59:00 UTC

We have begun the scheduled database maintenance planned for this morning. As a reminder, we anticipate up to 30 minutes of downtime for the control panel and API while we perform this work. Expect further updates as we have them.

Latest Update

We've successfully completed database maintenance. Thanks for your patience!

Read More

Updates

We're continuing to work on the planned maintenance. It's taking longer than expected, but we should be back online soon.

update 2014-08-09 12:31:21 UTC

Work is continuing on the planned maintenance. Things are nearly wrapped up, we're just finishing verification of the changes.

update 2014-08-09 12:55:12 UTC

We've completed our scheduled maintenance for this morning. We're continuing to monitor our systems and will post a final update.

update 2014-08-09 13:01:30 UTC

We've successfully completed database maintenance. Thanks for your patience!

resolved 2014-08-09 13:18:27 UTC
Aug 08 2014
Aug 07 2014
Aug 06 2014
Aug 05 2014
Aug 04 2014
Aug 03 2014
Aug 02 2014
Aug 01 2014
Jul 31 2014
NYC2 Power Maintenance July 31st, 2014
Posted on 2014-07-31 21:48:00 UTC

Maintenance Start: July 31st 17:00 EDT
Maintenance End: August 1st 05:00 EDT

One of our datacenter providers will be doing a power maintenance which will include de-energizing power feeds one at a time.
All of our equipment has redundant power sources therefore we do not expect this maintenance to affect any customers.
We will have staff on site to monitor the electricians progress.
Support staff will be monitoring the hypervisors as well.

Should you notice any issues with your droplets during this time please don't hesitate to contact support.

Thank you,
DigitalOcean Support

Latest Update

All circuits are re-energized and we are completed. No impact to any droplets.

Read More

Updates

All circuits are re-energized and we are completed. No impact to any droplets.

resolved 2014-08-01 05:39:06 UTC
Network Maintenance
Posted on 2014-07-31 04:20:00 UTC

We are currently preparing to perform a network maintenance in order to return the NYC2 region to full redundancy. While we do not expect this to cause any problems, but we are prepared to roll back quickly if there are any issues.

We will keep you posted as we have new information to share.

Latest Update

We have completed our maintenance window. You may have seen a brief period of up to 60 seconds of additional latency as the network reconverged. At this time all systems are working as expected and we have restored redundancy in the NYC2 region's core network.

Thank you for your patience and support.

Read More

Updates

We have just begun the first steps in our maintenance procedure. Things are going well so far. We will continue to report as we have additional information.

monitoring 2014-07-31 04:37:16 UTC

We have now restored redundancy in the network core for the NYC2 region. We will continue to monitor things very closely.

monitoring 2014-07-31 04:55:41 UTC

We have completed our maintenance window. You may have seen a brief period of up to 60 seconds of additional latency as the network reconverged. At this time all systems are working as expected and we have restored redundancy in the NYC2 region's core network.

Thank you for your patience and support.

resolved 2014-07-31 05:11:03 UTC
Jul 30 2014
Jul 29 2014
NYC2 Connection to Control Panel & Droplets
Posted on 2014-07-29 17:16:00 UTC

We have identified a slow connection to our NYC2 data center. Our team is working to resolve this and we will update you soon.

Some customers may have no connection at this time.

Latest Update

At this time the network is stable and we have removed the problematic core switch from the network. We will continue to work to fully understand and resolve the problems with that switch. Once we are comfortable that the device has been stabilized we will schedule a maintenance window to return it to production in the next couple of days to restore redundancy to our network. We are sincerely sorry for the disruption.

Read More

Updates

Due to this issue our Support Team is currently unable to reply to support tickets. Our team is standing by and will reply as soon as possible. We're sorry for the trouble and thank you for your patience.

update 2014-07-29 17:21:52 UTC

Our team is still actively working to resolve the issue in NYC2, we appreciate your patience and will provide an update soon.

update 2014-07-29 17:44:53 UTC

Our team is continuing to work to resolve the issue impacting connection to NYC2. Thank you for your patience.

update 2014-07-29 18:05:26 UTC

We are currently working to recover from a routing engine failure in our core switching infrastructure in the NYC2 region. All of our networking and site reliability team members are working hard to isolate the problem and we have engaged our vendor for additional support.

update 2014-07-29 18:38:28 UTC

We are seeing connectivity restored to our NYC2 data center. We will continue to monitor connectivity and events closely.

If you continue to see any connectivity issues please open a support ticket and our team will investigate.

monitoring 2014-07-29 19:06:28 UTC

At this time the network is stable and we have removed the problematic core switch from the network. We will continue to work to fully understand and resolve the problems with that switch. Once we are comfortable that the device has been stabilized we will schedule a maintenance window to return it to production in the next couple of days to restore redundancy to our network. We are sincerely sorry for the disruption.

resolved 2014-07-29 19:27:33 UTC
Jul 28 2014
Jul 27 2014
Jul 26 2014
Jul 25 2014
Jul 24 2014
Jul 23 2014
Jul 22 2014
NYC2 Connectivity and Event Latency
Posted on 2014-07-22 17:43:00 UTC

We are currently investigating network connectivity issues affecting some droplets in the NYC2 datacenter. These connectivity problems are also causing problems when accessing our cloud control panel and API.

We will provide updates as we have more information.

Latest Update

Read More

Updates

Connectivity to the affected droplets has been restored and our control panel and API are behaving normally. We will continue to monitor the situation very closely.

monitoring 2014-07-22 17:54:38 UTC

resolved 2014-07-23 14:49:44 UTC
Jul 21 2014
NYC2 Connection to Control Panel & Droplets
Posted on 2014-07-21 22:10:00 UTC

We have identified a slow connection to our Control Panel which may be impacting web and API users. Our team is working to resolve this and we will update you soon.

Latest Update

At this point the cloud control panel and API continues to be stable and all droplet communications are working as expected.

Read More

Updates

This issue is only impacting droplets in NYC2. We will provide an update as it becomes available.

update 2014-07-21 22:21:10 UTC

There is networking issue which is affecting droplet and control panel connectivity in NYC2. Our team is working to resolve this and we will update you soon.

update 2014-07-21 22:25:24 UTC

Due to this issue our Support Team is currently unable to reply to support tickets. Our team is standing by and will reply as soon as possible. We're sorry for the trouble and thank you for your patience.

update 2014-07-21 22:42:18 UTC

Our team is still actively working to resolve the issue in NYC2, we appreciate your patience and will provide an update soon.

update 2014-07-21 22:56:02 UTC

Our team is continuing to work to resolve the issue impacting NYC2. Thank you for your patience.

update 2014-07-21 23:52:44 UTC

Event processing and the cloud dashboard are currently responsive, but we continue to troubleshoot networking issues for some droplets in NYC2.

update 2014-07-22 00:14:59 UTC

At this time connection to control panel and droplets have returned to normal status. We are continuing to monitor the network. If you are still experiencing a service interruption please open a support ticket.

monitoring 2014-07-22 00:21:32 UTC

At this point the cloud control panel and API continues to be stable and all droplet communications are working as expected.

resolved 2014-07-22 01:03:19 UTC
Jul 20 2014
Jul 19 2014
Jul 18 2014
Jul 17 2014
Jul 16 2014
Jul 15 2014
Jul 14 2014
Event Processing Delays
Posted on 2014-07-14 17:50:00 UTC

We are currently working to resolve event processing delays for all droplet events across the cloud. This will affect droplet creation, power cycling, destroys, and other related events. We will provide updates as we have more information to share.

Latest Update

Event processing has been restored.

Read More

Updates

Event processing has been restored.

resolved 2014-07-14 17:55:34 UTC
Jul 13 2014
Jul 12 2014
Jul 11 2014
Jul 10 2014
Jul 09 2014
Jul 08 2014
Network Latency - All Regions
Posted on 2014-07-08 14:02:00 UTC

We are currently investigating higher than normal latency in all regions. We will provide updates as we have more information.

Latest Update

At this time our network latency is back to normal in all regions.

Read More

Updates

Latency is back to normal in our SFO, NYC2, and AMS2 datacenters. We are still working on the remaining regions.

investigating 2014-07-08 14:32:29 UTC

At this time our network latency is back to normal in all regions.

resolved 2014-07-08 14:57:32 UTC
Network Problems AMS1 & NYC2
Posted on 2014-07-08 03:48:00 UTC

We are currently investigating networking problems affecting our AMS1 and NYC2 datacenters. This may cause latency and packet loss reaching droplets in those regions as well and problems accessing services such as the DigitalOcean API or control panel.

Latest Update

Read More

Updates

Network performance has stabilized but we continue to monitor the situation very closely. Thank you for your patience.

monitoring 2014-07-08 04:24:59 UTC

resolved 2014-07-08 04:47:17 UTC
Jul 07 2014
Jul 06 2014
Jul 05 2014
Jul 04 2014
Event Processing Problems
Posted on 2014-07-04 18:30:00 UTC

We are currently working to resolve some problems with our event processing systems. All droplet actions like creates, deletes, and power on/off are affected. All hands are on deck to resolve this as quickly as possible.

Latest Update

Event processing continues to work normally so we are going to discontinue tracking this as a status event. Thanks again for your patience.

Read More

Updates

We continue to work to resolve the event processing problems. Our team knows where the problem lies and is working as fast as possible to restore service. We will post additional updates as we have more information to share.

update 2014-07-04 18:56:35 UTC

Our event system is once again processing events, but we have a backlog to work through. We expect this to drain very quickly and we will report back when event latency is back to normal.

Thank you for your patience.

update 2014-07-04 19:05:41 UTC

Event latency has largely returned to normal at this time. We will continue to monitor the situation very closely.

monitoring 2014-07-04 19:11:10 UTC

Event processing continues to work normally so we are going to discontinue tracking this as a status event. Thanks again for your patience.

resolved 2014-07-04 19:32:43 UTC
Jul 03 2014
Slow Event Processing
Posted on 2014-07-03 17:21:00 UTC

We're experiencing slow event processing resulting in a backlog of events such as creates, destroys, or power-cycles. Our engineers are working to resolve the issue.

Latest Update

Event processing speed is back to normal. Thank you for your patience.

Read More

Updates

Event processing speed is back to normal. Thank you for your patience.

resolved 2014-07-03 17:47:09 UTC
Event Processing: AMS2
Posted on 2014-07-03 02:05:00 UTC

We're experiencing slow event processing in AMS2 resulting in a backlog of events. Events have been temporarily disabled in AMS2 while our engineers investigate the issue.

Latest Update

Read More

Updates

Event processing has been enabled for some time, but we continue to monitor to ensure that things are fully back to normal.

monitoring 2014-07-03 03:46:32 UTC

resolved 2014-07-03 04:41:03 UTC
Jul 02 2014
Jul 01 2014
Jun 30 2014
Jun 29 2014
Jun 28 2014
Jun 27 2014