Investigating "Socket was Closed"

Since late February/early March I've found myself plagued by "socket was closed" whenever I try to play EVE Online, other games seem unaffected and I certainly have no issues with World of Warships. 

It's been a major issue as the usual symptoms means I cannot login more than one character at a time without disconnections, usually a random character, and when you're doing JF runs it plays havoc with whats going on as you don't know if your cyno character is online or has disappeared let alone if your JF pilot is sitting on the undock with a bumper nearby (although rare, I always jumped when I knew the locals OR local was empty). 

Thankfully you can reconnect to a disconnected cyno if its still active, but a few times I've lost connection as I've jumped and it certainly raised the blood pressure as I scrambled to relogin a disconnected character.

Diagnosing the Home Network

I have some familiarity with network connectivity investigation so the first thing I did was ensure that my computer and it's network connection were not at fault. 

My PC is connected via a 10m CAT6 cable to my routers 100/1000 port but I also have a wireless card built into my motherboard, this meant I could test both wired and WiFi connections. I first replaced the CAT6 cable, no change, I then tested WiFi, again no change to socket disconnections. The disconnections occurred at much the same frequency whether I was wired or wireless. As far as I'm concerned no issues there.

The Router Configuration

Next was to check the router configuration and to test my network in a semi-isolation mode. I was able to speak to one of the Network Administrators at my work who provided me with feedback on my router configuration and extra suggested things that I could look into that I had not considered.

The check of the router configuration yielded nothing of consequence, I did turn off features that had no need of (i.e. QOS, Guest Wireless, old port forwarding rules) and then I turned off or disconnected all devices on my network but my computer and then tried EVE Online. It looked promising but logging during the logging in of a second character the socket was closed on the Character Selection screen. Closing and reloading with a different character still resulted in a socket disconnection, just appearing to be at a random time.

One thing I did learn for certain is that home grade network equipment really is shockingly bad from a management features and point of view, I'll be looking to invest in a commercial grade router and switch in the near future to replace the piece of sh!t router I use now. 

Calling the ISP for the First Time

At this point I was confident that the issue lay outside of my network and so I called my ISP. 

I called them again because I couldn't get through and then again, again and again. Every time I called it said the delay to speak to someone was "greater than an hour" and even with the call back option I really didn't have the time at night to wait. I emailed and called over the course of two more days and eventually managed to get an email back from a Level 1 Support person.

I explained the problem and was told that they'd look into the issue, a few more minutes on hold and I was told my "connection had been reset". Wow, really what does that mean. I eventually managed to get the L1SP to explain that what he'd done is reset my ADSL connection at the local exchange. Turns out my ISP has space in the exchange of the Australian Communications company (Telstra) and he'd reset the connection as they usually find this works.

I wasn't happy with this, he wasn't investigating at all in my eyes, but a reset was worth seeing if it worked. I left it 24 hours then tested and found that it hadn't fixed the issue because when I loaded EVE Online the next night the very first thing that happened to me was a socket disconnection at the loading screen of the first character I loaded.

Network Diagnostics

To say I was a little pissed off would be right. I wasn't ready to go orbital but so far nothing pointed to an issue I could get fixed. I was sure the issue was outside my network so I needed to do some network diagnostics.

I could have used external utilities but the new EVE Launcher has some basic tools built in and I thought it was a good idea to use these as I could provide them to CCP saying I used their own tools to get the results.

So I clicked on the menu icon in the Launcher and selected the Network Diagnostics menu item.

EVE Launcher Network Diagnostics

I ran the Basic, Traceroute and Download test tests over several days at roughly the same time of night (~8pm) (the results seemed to be fairly consistent with little variation in the actual results):

Pinging dm794883twbxj.cloudfront.net [52.84.206.22] with 32 bytes of data:

Reply from 52.84.206.22: bytes=32 time=18ms TTL=55

Reply from 52.84.206.22: bytes=32 time=17ms TTL=55

Reply from 52.84.206.22: bytes=32 time=17ms TTL=55

Reply from 52.84.206.22: bytes=32 time=17ms TTL=55

Pinging client.eveonline.com [87.237.34.202] with 32 bytes of data:

Reply from 87.237.34.202: bytes=32 time=346ms TTL=239

Reply from 87.237.34.202: bytes=32 time=362ms TTL=239

Reply from 87.237.34.202: bytes=32 time=350ms TTL=239

Reply from 87.237.34.202: bytes=32 time=345ms TTL=239

Wow. A large difference in ping times there, what about the Traceroute:

Tracing route to client.eveonline.com [87.237.34.202]

over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  [MY ROUTER] 

  2    11 ms    14 ms    13 ms  [ISP SWITCH #1] 

  3    19 ms    16 ms    17 ms  [ISP SWITCH #2] 

  4    17 ms    30 ms    18 ms  [ISP SWITCH #3]  

  5   173 ms   173 ms   225 ms  [ISP SWITCH #4] 

  6   177 ms   176 ms   176 ms  sjo-b21-link.telia.net [62.115.33.97] 

  7   250 ms   250 ms   250 ms  ash-bb4-link.telia.net [80.91.245.70] 

  8   323 ms   322 ms   322 ms  ldn-bb2-link.telia.net [62.115.141.95] 

  9   323 ms   323 ms   322 ms  ldn-b1-link.telia.net [62.115.113.227] 

 10   361 ms   367 ms   363 ms  ccp-ic-316224-ldn-b1.c.telia.net [62.115.144.151] 

 11   355 ms   348 ms   354 ms  srv202-c.ccp.cc [87.237.34.202] 

Beejesus! Look at those pings, they match what we've seen with the ping. Seems we're okay on my ISP's switches until the last one when it seems to go absolutely haywire and increases by at least a ten times. Telia seems that my ping times are terrible in their network so lets pass this on to my ISP and see if they can do anything.

Calling the ISP for the Second Time

I found out that I no longer had a case open with my ISP and it had been closed as resolved after the last contact I had with them. So I called them and gave up when I was told the wait time was, again, "greater than one hour". So I emailed them to create a new ticket and included the results of my diagnostics along with all the symptoms and details I had.

I was very surprised when I got an email 24 hours later from, what I later found out, was a fellow EVE player who happened to work in Support for my ISP. 

Over the course of a few days I answered questions, provided information and waited. I did find out:

"... it seems that a lot of people (all over the world) are getting routed through TeliaSonera - and according to some Forum posts they are "Known game killers"."

and my traffic was being routed through the US to the EVE Online servers in London and this routing was unfortunately not under the control of my ISP.

I was impressed with the help I got this time from my ISP, no solution but the person I was speaking to was at least trying and he asked me to try a few more things that my ISP Network Operations Centre people wanted because they could find no issue with EVE Online and me.

So I went back to searching the EVE Online Forums (:shudder:) as well as Google.

Epiphany

Whilst searching the forums I found a thread and a link to a Reddit post that gave me some good information. I did find another user who posted Ping Plotter output that showed he had the same ISP as me and his results were a close match with me.

I did find one piece of information popping up whenever I searched was the suggestion to change the port number that EVE Online uses from 26000 to 3724. At this point I was looking for ANYTHING that might work so I tried it.

For one whole week after I changed the Port Number I did not have a single socket disconnection on any character, even when multiple characters were open at the same time. It was joyous, I could finally get into the backlog of things I could not do previously. Jumps were made, sometimes mulitple jumps to/from the same cyno in one cycle. Cargo was delivered, goods were bought, moved and then used to construct new things.

Oh it was joyous.

Then it stopped working. Just stopped. It was like a switch, it was working when I went to bed but when I logged in the next day it no longer worked. New Eden is a fickle Mistress and her husband must have found out I was using her because the pain returned.

Reaching the Fifth Stage of Anger

I reached out again to my ISP as well as trying to think of solutions or things I could try but it was obvious this was out of my control. So I continued playing, limiting my play time to what I knew I could do and then only doing JF runs once a week when I know it would be safest.

Well I thought it would be safest, two weeks ago I lost a freighter in high sec to a socket disconnection, a failure to emergency warp and an inability to login again in time to save my freighter. 

This was the last straw.

In Between

My jump freighter pilots are currently, stuck, in a mid-point system as one has had their account expire and the other I can't be bothered to login as he can't do anything anyway.

 

Sound bitter do I. Not really, I'm fine. It's not like you have to login to play EVE Online anyway is it.

 

Wanted: One working socket for my Kingdom!

 

I signed up to Discord last week at the urging of my CEO and, as he said, they've been able to speak to me more in a week of that than the past 6 months in-game. Whilst I appreciate Discord and actually am liking it I'm still unable to do much of anything within EVE Online.

 

I tried to jump out two JF's of material on Sunday and managed to get to my first mid-point before my second mid-point cyno character kept closing as he was logging in. The only way I was able to get him to the character selection screen was to close all other clients..

 

Yay I was able to login. Well no, I need the others logged in too in order to do things except as soon as you login another character the "dice of fate are rolled" and you either lose one at a time or all your characters that are logged in.

 

Back to the start I go, do not collect 200 ISK.

 

I gave up in the end and just quit EVE whilst swearing loudly enough my youngest son came and told me to stop saying naughty words. At which point I loaded World of Warships, an online game that I have no issues with connecting to or playing, and spending an enjoyable 40 minutes pummeling enemy ships in my North Carolina.

 

Got my first Citadel on the notoriously-hard-to-citadel Tirpitz at 17km. At the end of my 40 minutes I found I had enough XP (thanks to a 3x XP bonus for first win of the day per ship) to unlock the Tier 9 Iowa class battleship which made me very, very happy.

 

Reaching Out

 

I got frustrated on Thursday or Friday (I can't remember) enough that I decided to do something that I normally would stop myself doing but I needed to do something seeing as CCP Support weren't exactly talkative to me.

 

So I sent CCP Falcon a message on Reddit enquiring how to escalate a support ticket as I hadn't had a response in 7 days and I really needed some help.

 

Kudos to the Gentleman, he responded and whilst he didn't commit to help or promise anything his quick response was enough that I'm not longer stressing about this. I'm annoyed and want it fixed, yes. But I'm no longer stressing about it or letting it bother me.

 

It's out of my control and, after CCP Falcon said so, that Support are dealing with some major issues at the moment that are taking up their time I'm content to sit back and let things take their course. I'm still updating the ticket every couple of days with research I've done and on Friday I spent some time getting to see how Logging worked in the EVE client.

 

Was that a mistake that was. I eventually got to bed around 3am after staring at so many log entries that I swear I was dreaming about get-resource calls in my sleep :)

 

That said, I was able to replicate the disconnections whilst running in Logging mode and I uploaded 6MB of log data to my ticket for someone (hopefully) to check out.

 

Ending with a Laugh

 

NosyGamer posted a funny this morning. He's off to Iceland for Fanfest but before he left he visited r/eve on Reddit. Oh boy what a mistake, as he so well puts it:

The EVE Online sub-reddit is capable of sucking the enjoyment of EVE out of a person faster than an officer-fit Bhaalgorn can drain the capacitor from an Atron.

 

It does occasionally provide entertainment or decent information but it is a horrible place full of trolls, idiots and even more trolls. Since they started the Propaganda tagging with the World War Bee stuff I've avoided the place and I think my IQ went up by at least a few points :)

7 Days

My support petition for help with "the socket was closed" has been open for 7 days and has not been acknowledged or answered by CCP.

 

24 hours left before I have to choose to renew or cancel my subscriptions.

 

Really I don't think I'll have any impact if I do close them but I'm really disappointed in the lack of communication.

 

My day job is working as a Help Desk Officer and it's a point of pride that I will triage or acknowledge any tickets I get within 24 hours at the latest. , yet here I am 7 days later and still in limbo. It's enough to make one sad.

Nearing the End of the Tether

I still can't play EVE Online for more than a couple of minutes and it's not by choice. Whenever I login, even a single character, I get disconnected at random with "Socket was Closed". It's a little better than it was but I can't do anything aside the basics such as checking hangers, jump cloning or just spinning ships at the moment.

 

I have been able to keep towers up to date and functioning but its been more luck than anything. I had to update towers Sunday but I disconnected mid-warp to my first tower and then again after landing at the second tower.

 

There is no pattern I can discern but I've now escalated the issue to my ISP to see if there is something they can see/fix. Unfortunately where I used to proclaim how good their support was I now find myself on hold for an hour at a time to just speak to someone...

 

Still, speak to ISP then raise ticket with CCP.

 

If I can't get any traction on this by next Friday I'm going to terminate my accounts. I can't keep paying for a game I can't play.

Sockets, Sockets, Sockets

Socket was Closed

 

I've been getting seriously pissed off with EVE Online the past fortnight and all thanks to one of the most annoying error messages I've ever seen.

 

Socket was closed.

 

That's the message. It would appear seemingly at random when I was trying to login or doing things in EVE Online. So much has this appeared that I could not do anything but the most basic operations.

 

I wasn't even able to run my normal scheduled JF'ing in and out of null sec. Courier contracts came very close to expiring because I could not login more than one or two characters at a time without:

 

Socket was closed.

 

I persevered and eventually I've been able to get done what I needed, some contracts came close to expiring though. And moving through high sec in a freighter was so painful that for three days I didn't even login to EVE.

 

Cause

 

Other games I've had zero problems with. World of Warships has worked without issue, so has Steam and about five or ten other heavy bandwidth using applications.

 

Honestly no idea what the problem has been. It's never been the same character doing the same actions so I really can't reproduce the issue, I just know it happens when the dialog box appears.

 

I've had people suggest corrupt installation or asset downloads to others saying its a ping/route/ISP issue. I've done a fair bit of diagnostics and spoken to both my ISP and several experienced network engineers friends to no avail.

 

I have reinstalled, twice, and cleared cache and temp files. I've setup network monitoring and even swapped my ADSL router out for another brand (I keep spares so it didn't cost me anything but time). My ISP reported no problems, and I even pushed for Level 2/3 assistance which they allowed me to speak to, no blocks or throttling of traffic found to any of EVE Online servers.

 

I'm at a loss to cause, things have settled down a little but I'm still expecting to get the error message appear whenever I login.

In need of a Socket Wrench

I've had a pretty poor week in EVE, I mentioned previously that I'd been having connection issues with our old friend "Socket was closed". Well after I posted it's like EVE read my blog and decided that I needed a bigger dose of errors.

It has been very odd in the pattern I've experienced. My cyno characters had issues every time I logged them in and had them idle for 5 minutes or longer. My main character experienced disconnections at very inconvenient times, once in the middle of a cyno jump but fortunately I was able to terminate the process and login again in less than a minute so I wasn't in any real danger.

But very inconvenient.

The past 24 hours have been better, I even managed to stay logged in with 3 characters for an afternoon as I did RL and EVE stuff. Not a single disconnection. So it looks like things are on the up and I've already had my launcher issues for this release and fortunately managed to fix that with a reinstall.