Discussion in   General Discussions   started     9 years ago   August 10, 2015, 11:06:34 PM   by   Evolution

Unexpected Downtime

Evolution
Offline
1290 Posts
Topic :   Unexpected Downtime
9 years ago  August 10, 2015, 11:06:34 PM

A short time ago, our Chicago router crashed and was rebooted by a watchdog system that we have running. This led to several minutes of connectivity loss for this location. The cause for this is likely the same as prior Chicago router reboots -- making it a probable hardware issue. We will be investigating and following up with our hardware vendor.

Update @ Aug 10th, 3:32pm CDT: This happened again a short while ago. We are working on getting a new part delivered from our hardware vendor as quickly as possible.


Admin Dante - Owner - UO Evolution Custom Ultima Online Shard

Website - http://www.uoevolution.com

Forum - http://www.uoevo.com/forum

Wiki - http://www.uoevo.com/wiki

Discord - http://www.discord.gg/JwEBhPH

Evolution
Offline
1290 Posts
#1 Re :   Unexpected Downtime
9 years ago  August 10, 2015, 11:07:14 PM

Our Chicago router has crashed twice today, leading to several minutes of downtime on each occasion. We are relatively certain that this is a hardware problem with one of the CPUs in the router and we have asked our hardware vendor for a replacement.

Uptime is very important to us. For the time being, in order to make sure that services will not have further unscheduled interruptions, we will be reactivating our old router in Chicago and moving our Internap connections back over to it, taking the new one offline entirely until its faulty CPU has been replaced. This will cause another brief interruption for your services sometime this evening, but it should be shorter than the others.

Update @ 9:17pm CDT: We are still waiting for the facility to move us back to our old router. In the meantime, the new router has crashed again. We are continuing to wait, so we can get back onto the stable router.

Update @ 10:23pm CDT: We continue to wait anxiously for the facility to get around to doing the critically important work of switching us back to our backup router. The broken router has again crashed, just now.


Admin Dante - Owner - UO Evolution Custom Ultima Online Shard

Website - http://www.uoevolution.com

Forum - http://www.uoevo.com/forum

Wiki - http://www.uoevo.com/wiki

Discord - http://www.discord.gg/JwEBhPH

Evolution
Offline
1290 Posts
#2 Re :   Unexpected Downtime
9 years ago  August 11, 2015, 11:08:26 AM

The new router was replaced last night and is stable. This morning, we've had a few large denial of service attacks against servers at the location which can cause very brief packet loss as the null is applied.  The ISP is aware and is mitigating the attacks.

Sorry for the disruptions over the past 2 days.  The issue will be resolved shortly


Admin Dante - Owner - UO Evolution Custom Ultima Online Shard

Website - http://www.uoevolution.com

Forum - http://www.uoevo.com/forum

Wiki - http://www.uoevo.com/wiki

Discord - http://www.discord.gg/JwEBhPH

Evolution
Offline
1290 Posts
#3 Re :   Unexpected Downtime
9 years ago  August 26, 2015, 03:00:10 PM


Expected Downtime

Between 2:30am and 4:30am CDT on Thursday, August 27, we will be cutting back over to our newer router in Chicago, following hardware and software upgrades that are likely to have resolved the problem that it had which caused crashes. This adjustment will lead to a few minutes of connectivity loss for your service.
We will be carefully monitoring for problems following the switch. If the new router crashes again, we will immediately change back to the backup router, and then follow it up by entirely replacing the new router.


Admin Dante - Owner - UO Evolution Custom Ultima Online Shard

Website - http://www.uoevolution.com

Forum - http://www.uoevo.com/forum

Wiki - http://www.uoevo.com/wiki

Discord - http://www.discord.gg/JwEBhPH