Discussion in   Shard Updates   started     11 years ago   February 23, 2014, 12:02:59 AM   by   Kyn

02-22-2014 Upstream failure

Kyn
Offline
336 Posts
Topic :   02-22-2014 Upstream failure
11 years ago  February 23, 2014, 12:02:59 AM


0-22-2014 @ 11:01 PM


We had been effectively blocking an extremely large DDoS attack when one of Internap's providers dropped, causing attack traffic to be forced over other links in our data center and saturating Internap's network. We are currently working with Internap to address this, but in the meantime, we are seeing high latencies and packet loss to many UO Evolution Players, as well as client disconnections.


A very large NTP reflection DDoS attack against our Teamspeak server in Chicago caused Internap to null-route the TS3 server.. We immediately noticed this and contacted Internap to remove the null, and we worked with them to mitigate the attack, restoring services.


Internap continues to develop and tweak its systems and network to handle large attacks, and we provide constant feedback and do everything that we can to help on this end.


11:53 PM Update

As I described earlier, our TS3 server in Chicago is currently null-routed and therefore offline, due to a massive attack in Chicago and concurrent issue with one of Internap's upstreams. We will need to leave this in place until the upstream brings back up its link to Internap. They have not yet provided an ETA on this but it is a serious problem that they are working to resolve ASAP. Doing my best to expedite the process. Sorry guys..


02:23 AM Update

Teamspeak is now running properly.

Keldon
Offline
223 Posts
#1 Re :   02-22-2014 Upstream failure
11 years ago  February 23, 2014, 12:21:23 AM

Kyn,


Thank you for your hard work on such a crappy situation!

Kyn
Offline
336 Posts
#2 Re :   02-22-2014 Upstream failure
11 years ago  February 23, 2014, 12:21:58 AM

Kyn,


Thank you for your hard work on such a crappy situation!


It's becoming ridiculous.

Blackbeard
Offline
15 Posts
#3 Re :   02-22-2014 Upstream failure
11 years ago  February 23, 2014, 12:45:14 AM

Just curious, but are these the same people DDoSing us over and over?

Keldon
Offline
223 Posts
#4 Re :   02-22-2014 Upstream failure
11 years ago  February 23, 2014, 12:51:23 AM

Just curious, but are these the same people DDoSing us over and over?


from what I have heard it is different people hitting the data center (not us specifically) each time.  This has caused a lot of head ache for Kyn as each time he has to null route a new source in order to keep our server and team speak online.

Blackbeard
Offline
15 Posts
#5 Re :   02-22-2014 Upstream failure
11 years ago  February 23, 2014, 01:06:50 AM

Ah okay. It makes more sense that its not specifically targeting us and we are just included in the attack on the provider. Well it does stink that  I wanted to get some farming done tonight but crap happens I guess. I would like to send another donation to our server in the form of a hug >.< Thank you Kyn for all the time you spend trying to defend us from the evils of the real world and I know theres probably 100 other things you would rather be doing but selflessly you are fighting for all of us to have a fun few hours of play time before bed. <3 you Kyn and the rest of the Evo family!

Kyn
Offline
336 Posts
#6 Re :   02-22-2014 Upstream failure
11 years ago  February 23, 2014, 01:11:50 AM

Ah okay. It makes more sense that its not specifically targeting us and we are just included in the attack on the provider. Well it does stink that  I wanted to get some farming done tonight but crap happens I guess. I would like to send another donation to our server in the form of a hug >.< Thank you Kyn for all the time you spend trying to defend us from the evils of the real world and I know theres probably 100 other things you would rather be doing but selflessly you are fighting for all of us to have a fun few hours of play time before bed. <3 you Kyn and the rest of the Evo family!


Shard is available, just our TS server is down at the moment. We got hit with some kind of reflection attack I've never seen before. Averaging about 30-50 GBPS I've managed to null route most of this crap and I'm now working on updating the firewall to filter out the noise packets so our users can start connecting again. I don't have an ETA yet but I'm going hard!

Malachai
Offline
34 Posts
#7 Re :   02-22-2014 Upstream failure
10 years ago  February 24, 2014, 06:05:51 AM

This right here is why I have come to call UOE my home and dont see myself moving anytime soon.
The dedication and selflessness of everyone involved makes me extremely happy to be here.
 
Dante and Kyn you guys are so full of awesomesauce I can taste it all the way from Sweden.
 

Warboss
Offline
47 Posts
#8 Re :   02-22-2014 Upstream failure
10 years ago  March 17, 2014, 06:05:20 PM

Why do you keep making kids mad to ddos you?