Investigating "Socket was Closed"

Since late February/early March I've found myself plagued by "socket was closed" whenever I try to play EVE Online, other games seem unaffected and I certainly have no issues with World of Warships. 

It's been a major issue as the usual symptoms means I cannot login more than one character at a time without disconnections, usually a random character, and when you're doing JF runs it plays havoc with whats going on as you don't know if your cyno character is online or has disappeared let alone if your JF pilot is sitting on the undock with a bumper nearby (although rare, I always jumped when I knew the locals OR local was empty). 

Thankfully you can reconnect to a disconnected cyno if its still active, but a few times I've lost connection as I've jumped and it certainly raised the blood pressure as I scrambled to relogin a disconnected character.

Diagnosing the Home Network

I have some familiarity with network connectivity investigation so the first thing I did was ensure that my computer and it's network connection were not at fault. 

My PC is connected via a 10m CAT6 cable to my routers 100/1000 port but I also have a wireless card built into my motherboard, this meant I could test both wired and WiFi connections. I first replaced the CAT6 cable, no change, I then tested WiFi, again no change to socket disconnections. The disconnections occurred at much the same frequency whether I was wired or wireless. As far as I'm concerned no issues there.

The Router Configuration

Next was to check the router configuration and to test my network in a semi-isolation mode. I was able to speak to one of the Network Administrators at my work who provided me with feedback on my router configuration and extra suggested things that I could look into that I had not considered.

The check of the router configuration yielded nothing of consequence, I did turn off features that had no need of (i.e. QOS, Guest Wireless, old port forwarding rules) and then I turned off or disconnected all devices on my network but my computer and then tried EVE Online. It looked promising but logging during the logging in of a second character the socket was closed on the Character Selection screen. Closing and reloading with a different character still resulted in a socket disconnection, just appearing to be at a random time.

One thing I did learn for certain is that home grade network equipment really is shockingly bad from a management features and point of view, I'll be looking to invest in a commercial grade router and switch in the near future to replace the piece of sh!t router I use now. 

Calling the ISP for the First Time

At this point I was confident that the issue lay outside of my network and so I called my ISP. 

I called them again because I couldn't get through and then again, again and again. Every time I called it said the delay to speak to someone was "greater than an hour" and even with the call back option I really didn't have the time at night to wait. I emailed and called over the course of two more days and eventually managed to get an email back from a Level 1 Support person.

I explained the problem and was told that they'd look into the issue, a few more minutes on hold and I was told my "connection had been reset". Wow, really what does that mean. I eventually managed to get the L1SP to explain that what he'd done is reset my ADSL connection at the local exchange. Turns out my ISP has space in the exchange of the Australian Communications company (Telstra) and he'd reset the connection as they usually find this works.

I wasn't happy with this, he wasn't investigating at all in my eyes, but a reset was worth seeing if it worked. I left it 24 hours then tested and found that it hadn't fixed the issue because when I loaded EVE Online the next night the very first thing that happened to me was a socket disconnection at the loading screen of the first character I loaded.

Network Diagnostics

To say I was a little pissed off would be right. I wasn't ready to go orbital but so far nothing pointed to an issue I could get fixed. I was sure the issue was outside my network so I needed to do some network diagnostics.

I could have used external utilities but the new EVE Launcher has some basic tools built in and I thought it was a good idea to use these as I could provide them to CCP saying I used their own tools to get the results.

So I clicked on the menu icon in the Launcher and selected the Network Diagnostics menu item.

EVE Launcher Network Diagnostics

I ran the Basic, Traceroute and Download test tests over several days at roughly the same time of night (~8pm) (the results seemed to be fairly consistent with little variation in the actual results):

Pinging dm794883twbxj.cloudfront.net [52.84.206.22] with 32 bytes of data:

Reply from 52.84.206.22: bytes=32 time=18ms TTL=55

Reply from 52.84.206.22: bytes=32 time=17ms TTL=55

Reply from 52.84.206.22: bytes=32 time=17ms TTL=55

Reply from 52.84.206.22: bytes=32 time=17ms TTL=55

Pinging client.eveonline.com [87.237.34.202] with 32 bytes of data:

Reply from 87.237.34.202: bytes=32 time=346ms TTL=239

Reply from 87.237.34.202: bytes=32 time=362ms TTL=239

Reply from 87.237.34.202: bytes=32 time=350ms TTL=239

Reply from 87.237.34.202: bytes=32 time=345ms TTL=239

Wow. A large difference in ping times there, what about the Traceroute:

Tracing route to client.eveonline.com [87.237.34.202]

over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  [MY ROUTER] 

  2    11 ms    14 ms    13 ms  [ISP SWITCH #1] 

  3    19 ms    16 ms    17 ms  [ISP SWITCH #2] 

  4    17 ms    30 ms    18 ms  [ISP SWITCH #3]  

  5   173 ms   173 ms   225 ms  [ISP SWITCH #4] 

  6   177 ms   176 ms   176 ms  sjo-b21-link.telia.net [62.115.33.97] 

  7   250 ms   250 ms   250 ms  ash-bb4-link.telia.net [80.91.245.70] 

  8   323 ms   322 ms   322 ms  ldn-bb2-link.telia.net [62.115.141.95] 

  9   323 ms   323 ms   322 ms  ldn-b1-link.telia.net [62.115.113.227] 

 10   361 ms   367 ms   363 ms  ccp-ic-316224-ldn-b1.c.telia.net [62.115.144.151] 

 11   355 ms   348 ms   354 ms  srv202-c.ccp.cc [87.237.34.202] 

Beejesus! Look at those pings, they match what we've seen with the ping. Seems we're okay on my ISP's switches until the last one when it seems to go absolutely haywire and increases by at least a ten times. Telia seems that my ping times are terrible in their network so lets pass this on to my ISP and see if they can do anything.

Calling the ISP for the Second Time

I found out that I no longer had a case open with my ISP and it had been closed as resolved after the last contact I had with them. So I called them and gave up when I was told the wait time was, again, "greater than one hour". So I emailed them to create a new ticket and included the results of my diagnostics along with all the symptoms and details I had.

I was very surprised when I got an email 24 hours later from, what I later found out, was a fellow EVE player who happened to work in Support for my ISP. 

Over the course of a few days I answered questions, provided information and waited. I did find out:

"... it seems that a lot of people (all over the world) are getting routed through TeliaSonera - and according to some Forum posts they are "Known game killers"."

and my traffic was being routed through the US to the EVE Online servers in London and this routing was unfortunately not under the control of my ISP.

I was impressed with the help I got this time from my ISP, no solution but the person I was speaking to was at least trying and he asked me to try a few more things that my ISP Network Operations Centre people wanted because they could find no issue with EVE Online and me.

So I went back to searching the EVE Online Forums (:shudder:) as well as Google.

Epiphany

Whilst searching the forums I found a thread and a link to a Reddit post that gave me some good information. I did find another user who posted Ping Plotter output that showed he had the same ISP as me and his results were a close match with me.

I did find one piece of information popping up whenever I searched was the suggestion to change the port number that EVE Online uses from 26000 to 3724. At this point I was looking for ANYTHING that might work so I tried it.

For one whole week after I changed the Port Number I did not have a single socket disconnection on any character, even when multiple characters were open at the same time. It was joyous, I could finally get into the backlog of things I could not do previously. Jumps were made, sometimes mulitple jumps to/from the same cyno in one cycle. Cargo was delivered, goods were bought, moved and then used to construct new things.

Oh it was joyous.

Then it stopped working. Just stopped. It was like a switch, it was working when I went to bed but when I logged in the next day it no longer worked. New Eden is a fickle Mistress and her husband must have found out I was using her because the pain returned.

Reaching the Fifth Stage of Anger

I reached out again to my ISP as well as trying to think of solutions or things I could try but it was obvious this was out of my control. So I continued playing, limiting my play time to what I knew I could do and then only doing JF runs once a week when I know it would be safest.

Well I thought it would be safest, two weeks ago I lost a freighter in high sec to a socket disconnection, a failure to emergency warp and an inability to login again in time to save my freighter. 

This was the last straw.