As of 1:29pm Shardtime the Chicago data center down, I will keep you informed as we know more information.  Once it comes back up we will see what needs to be done.

Thanks and I do apologize in advance.

Thanks for the heads up Kane, and keeping everyone posted.
With this down time maybe I will actually accomplish something at work today.... maybe

thanks for the heads up Kane was wondering why i could not log in. hehe .. ps hope i dont lose my newly made champ axe lol.

NOOOOOOOOooooooooooooo............  What ever will I do now Kane? It is so dark here.

I started to go outside... but the door handle felt warm to the touch. I read somewhere that meant it was unsafe to open the door.

KG: How long ago was the axe made? 

Looks like the same issue as before, Dante is already on the phone with them.  It is effecting the entire datacenter and not just us.  I will repeat the samething I did earlier, as soon as the shard comes up I will pull the save from before this happened in case the unexpected happens.

I am already having withdrawls kane. My body is shaking and I see spots and halos. What to do? I need an intervention I think. Calgone, take me away.

Our Chicago location stopped responding to queries from our webserver at approximately 1:28pm CDT. We are currently investigating and have contacted the site to have an on-site engineer physically inspect our router.

Update @ 2:07pm CDT: We are still working to get this resolved ASAP. The local facility's phone system is having a problem that causes it to hang up as soon as we pick up the phone, which has so far delayed this at least another 15 minutes, as they are unable to reach us. We are asking them to try other methods.

Update @ 2:48pm CDT: The facility was finally able to contact us and rebooted the router, which immediately brought back up the location. We are continuing to investigate, but this initially appears to be an OS-related crash. We will likely need to prep and schedule an OS upgrade in the near future, to occur during an early-morning maintenance window.

Update @ 4:24pm CDT: The router appears to have stopped responding again. We are having the site reboot it, which should be a faster operation this time. We are not yet ready with a new OS version to try, unfortunately, or we would immediately apply that. We are still working on it and are evaluating other options as we continue to try to determine why this has happened.

Just 1 year from wipe and probs again LOL I hope everything is safe now

We have been working on the issue all night and our ISP and the Data Center sent this update.

Sorry again for the interruptions, we are doing our best to resolve this issue

"Dante": Update @ 5:24pm CDT: The router was brought online with a reboot at approximately 4:55pm CDT, after which we rebooted it again with the latest (and untested) kernel. We are continuing to test and monitor.

We have also examined the hardware logs, which say that one of the CPUs in the router had a high temperature. It is not clear whether this happened after the machine stopped responding, or whether it is the reason that it did stop responding. We currently see low temperatures for both CPUs and all fans are working, but to be safe, we have bumped up fan speeds to maximum levels.

"Dante": Update @ 3:26am CDT on 5/21: This router has crashed again.

We have contacted the facility to have them shift us back to the older router that we have in Chicago. We will then have to reconfigure it in a few ways, which may lead to some further downtime.

After we have the new router offline, we can do more testing on it, and replace parts as necessary -- at this point, this seems most likely to be a hardware issue.

"Dante": Update @ 3:57am CDT: We are back up with the old router. We are continuing to investigate the new problem with the new router. Our current theory is a faulty CPU temperature sensor (a problem we've never seen before) or just a plain bad CPU.

Router Update[/size]

Between 2am CDT and 4am CDT on Monday, June 1, we plan to switch back to our newer router in Chicago. We have made adjustments to its configuration and run extensive CPU and memory tests on it since we took it offline on May 20, and it seems to be stable, but to be safe, we have also developed a tool that will automatically reboot it if it becomes unresponsive again; this means that if downtime occurs, it will only last 3-5 minutes instead of the longer previous outages (if this system kicks in, we will of course then migrate back away from it).[/size]As a result of the maintenance on Monday, your service should see less than three minutes of connectivity loss.