Solved

LinkLink service is unreliable

  • 19 November 2018
  • 20 replies
  • 263 views
M
Badge

I've had a LineLink device for a few months now, and when it works, it seems to work well.  However, like many others, I experience persistent failures in the LineLink service from time to time.

The failure mode is that inbound calls don't get delivered to the LineLink device, but instead are diverted to voicemail.  Outbound calls fail with a "fast busy" or reorder tone.  This is annoying because the device generates a dial tone when you go offhook.  

This failure happens every so often.    Once it gets into this mode, power cycling the LineLink device will resolve the issue.  I also have a Vonage VoIP adapter in my home, connected to the same Ethernet switch and on the same network.  It continues to work just fine.  So probably not a problem with my router. 

My guess is that there's a memory leak in the device.  Or perhaps the registration with the T-Mobile network fails at the far end, but the device doesn't notice some stale registration.  Not sure, there's no external visibility into the working of this box.

I used the be Chief Technology Officer at Vonage some years ago, so I've had some experience building residential VoIP products, and having them operate on random residential networks, behind all manner of crappy routers, cable modems and the like.  I know it's possible to solve this problem, because I've personally done it.  I would really like to have this service work reliably. 

I'm about to set up some automation to power cycle the box every morning at 4:00AM.  But that doesn't fix the problem, it only mostly avoids or recovers from it.  From what little investigation I've done, this seems to have been going on with lots of end-users and for some time.   I don't know why T-Mobile hasn't been able to fix this issue.  From my personal experience when I ran a platform with more than 2 million VoIP CPE devices, we took these types of failures very seriously because lives are at risk since 911 calls probably don't work either. 

Is someone from T-Mobile going to step up and commit to solving this issue?  I opened a ticket and was told it was going to be escalated to engineering, but there was no clear response when I asked about what happens next, and how do I track this issue towards resolution?  I'd be happy to work with someone from engineering to characterize the nature of problem and do some debugging. 

If that's not going to happen, I guess I start calling and asking for bill credits every month, costing you guys money for the service, as well as another customer contact and those associated costs.  And rebooting the thing every night, which is a totally crappy workaround to what should be solvable problem.

icon

Best answer by tmo_amanda 28 November 2018, 00:49

View original

20 replies

tmo_amanda
Rank
Userlevel 4

Hey, @magenta6847193​!

Welcome to our Support Community and thank you for providing such an in-depth post about what's going on with your LineLink device. Awhile back I worked with an Engineer on a similar issue. The call would be three times then disconnect. It sounds pretty similar to what you're describing but power cycling the unit did not solve the issue. Do you know if there's a specific amount of beeps you hear or is it just a continuous busy signal? While I wait to hear back from you, I'm going to email the Engineer I was previously working with to see if I can get any additional info on a solution. Last question - is the email address associated with your Support Community account the same as your T-Mobile ID? If needed, I'll pass that info along to our Engineers to begin looking into this.

M
Badge

When the device is in the usable state:

  • inbound calls don't ring, and are delivered to voice mail
  • attempted outbound calls do not ring; I get a fast-busy/re-order call progress indication after dialing, which sounds similar to a busy, but has a different cadence.  I believe the fast-busy/reorder call progress indication is originated locally by the VoIP adapter, not from a remote media stream.

I'm not sure how long that the fast-busy persists; when I hear it, I just go on-hook and abandon the attempt.  This happens at call origination time; I've not had it occur during a call that's in-progress.  But I don't use that line for very many outbound calls, so I can't really say with certainty if there are any failures during an active call.  Just for clarity; when in this failure state, I don't get a locally (or remotely) originated ring-back call progress indication; just the fast-busy/reorder when I attempt to originate an outbound call.

So far, power-cycling the TA seems to resolve the issue in every occasion.  I just had another failure that I noticed today which required a power cycle to restore.

My email address should be the same as the T-Mobile ID email address.

tmo_amanda
Rank
Userlevel 4

Thank you so much for getting back to me with more info. I'm going to be 110% honest with you -- I understand the issues you're having, but I don't understand the back end system for LineLink at all (from your past tech experience it sounds like you have a pretty good grasp of it). I forwarded all of the info you've shared with me to the engineer I previously worked with along with your email address should they need to contact you. I will get back to you as soon as I receive a reply from them or within the next 48 hours, whichever is sooner.

I'm sorry that I don't have immediate answers but I also don't want to brush this off and have you call Tech Support when we may be able to get this handled more efficiently.

M
Badge

@tmo_amanda​, thanks for  your follow-up.  I'd be pleased if you can get my issue in front of an engineer who might be able to work efficiently towards a resolution.  I look forward to hearing back with an update.   I would agree that trying to open a case via tech support might not be the most productive way forward, as I think there's a more fundamental problem at work that won't get resolved via the usual troubleshooting process.

Thanks for your help!

P

I experience precisely the same problem as does magenta6847193, and I have since I was migrated from the old @Home service and equipment to the new LineLink service and equipment.

Like him/her, while the LEDs on the LineLink equipment indicate all is well, my outgoing calls will periodically all fail with a fast busy/error tone, and my incoming calls will fail and go directly to voicemail.  I can always fix this by power-cycling the device, typically somewhere between once every two weeks to twice a week.  My modem/Internet service seem not to be the cause (its uptime is currently 82 days with no downtime logged since September.)

I never experienced this issue with the previous @Home service; that device was only ever power-cycled by a power outage.  😊

@tmo_amanda, if you’re able to get assistance from engineering to help magenta6847193, I’d greatly appreciate if you could share the solution with me.  I’m happy to help in troubleshooting any way I can.

M
Badge

I am hoping to get this resolved.  I'll probably end up inserting a Sonoff Basic Wi-Fi switch on the power supply so that I can more easily power cycle it if necessary.  Perhaps nightly at 4:00am when I'm pretty sure no one will be using it.  From the way it behaves, it feels like a memory leak or similar in the LineLInk device; alternatively, it could be some event that happens upstream that voids the device registrations.  Unfortunately, there's no way to detect the issue until you try to actually put an inbound or outbound call into service, at least that I can tell.

When I was at large OTT VoIP operator, our devices would restart themselves upon loss of SIP registration.  Some of that was defense against weirdness in the local residential IP network and CPE router screwing up NAT mappings or other craziness that you wouldn't believe.   A restart, with new DHCP requests had the effect of working around those issues.   It also helped with firmware problems on occasion as well.

We had one hard to diagnose problem that turned out to be a memory leak.  What made it difficult is that we couldn't reproduce it in our lab environment.  Eventually, somehow, our support people noticed that the fails seemed to happen for customers that had a printer!  WTF?  Turns out the mDNS service announcements (multicast UDP DNS packets on port 5353) used for service discovery of the printers ended up taking a code path in the VoIP CPE that did not free them.  After some period of time, all the buffers used for inbound network traffic got consumed and orphaned, and the device had to be rebooted.

It would be funny if that was the case here, though TI did fix that particular bug in their software stack.  Only took 3 months of intensive effort to figure out, as those devices were in the hands of hundreds of thousands of customers.  I fear that the LineLink product is just an add-on service, and it will be difficult to get sufficient attention to the problem to get to the bottom of it.

Maybe I should put the LineLink ethernet switch port on its own VLAN with no other devices to see if that helps.  The problem is trying to measure progress when the problems on show up many days or weeks later..

tmo_amanda
Rank
Userlevel 4

@pastiche​, I will most definitely keep you in the loop. I'm sorry this is happening to you as well, but I'm hoping we make some progress thanks to the vast knowledge @magenta6847193 is sharing with us. 😊

@magenta6847193​, I haven't received a reply as of yet, but I wanted to give you a heads up that I will be out of the office until Friday. I will reply first thing Friday morning with an update.

tmo_amanda
Rank
Userlevel 4

It's me again! Just wanted to send an update your way that I have engineers looking into it but no definite answers. I'll keep you posted as I hear more and will reply back no later than next Tuesday. In the meantime, I hope you enjoy your weekend.

tmo_amanda
Rank
Userlevel 4

Phew! I finally have an update for y'all -- I'm sorry it took so long. I received a reply after I already logged off for the evening yesterday so I'm passing it along first this this morning. 😊 The Engineer I've been working with stated that they've already identified the potential root cause of this issue and are testing a new patch from one of our network vendors to make sure it fully solves the issue. He'll update me on the testing and let me know when it'll be rolling out.

P

This is great news!  Thanks, tmo_amanda.  Looking forward to what comes next!  😊

tmo_amanda
Rank
Userlevel 4

You're very welcome! I have this thread bookmarked so as soon as I'm given more info, I'll head right over here. 😊 I'm sorry this took much longer than expected to get you answers but I'm happy that our teams were already working on a solution.

M
Badge

Great news.  Please follow-up when the new software has been deployed, and I'll watch closely to see how it behaves.  Since that testing work is already in progress, I'll just do a periodic power-cycle of my LineLink in the mean time as (hopefully) a workaround.

M
Badge

Any news on the status of this update, when it might be deployed and any release notes that describe the changes and/or bugs fixed?

tmo_amanda
Rank
Userlevel 4

Hey, @magenta6847193​!

I'm sorry it has taken me a few days to get back to you. No, I don't have any updates just yet. I'm gonna go out on a limb here to say that it may not be until after the new year that we hear anything back. I'm off for the next couple of days, but when I return, I'll be sure to update the thread if I receive a reply. 😊

M
Badge

And progress update on this?  I have gone to just automatically power cycling the LineLink device at 4:00am every day as a workaround, and that seems rather crude..

M
Badge

Is there any movement on the resolution of this problem?  It's been 5 months since this thread started, and surely I'm not the first to be having this issue.

I'm presently power-cycling my LineLink device every day early in the morning to workaround the issue.  As a result of this automated daily power-cycle/reboot, I seem to experience reliable service... this leads me to believe that in fact there's some sort of memory or resource leak in the software stack, that over time causes the malfunction.

I can power cycle this thing every day since I have a wi-fi controlled switch do it all automatically, but this certainly isn't a great way to treat the as-cheap-as-possible power supply and electronic components in the device.  I worry about this because I used to build and sell very similar devices to customers.  It would really be great if this was a proper fix in the software so these workaround can be discontinued.

Just to poke the bear - I sure hope no LineLink customer has to make a 911 call, only to discover their device needs to be rebooted.  It seems pretty clear there's a known issue here, and I sure hope there's adequate attention and prioritization of remediation efforts to assure fundamental, basic functioning of this product that you put into the hands of your customers.

If there is some new software that's been rolled out, and there's a belief this problem has been resolved, I'll discontinue my daily automated reboot and test it.  Thanks for your attention to this matter.

M
Badge

Have any software or service updates occurred in the last 6 months since I reported this issue?   While my strategy of rebooting the device late every night seems to result in more or less "reliable" abilities to make and receive calls, it's not optimal.  Doing that automated restart tends to lose pending voicemail notifications, I think, which isn't great.

I'd be happy to work with someone on the engineering team with this product to resolve this issue once and for all.

M
Badge

Any progress towards resolution of this issue?  In order to get usable service, I automatically power cycle this device each day at 4:15AM, which seems to be a workaround for the whatever the actual underlying issue is.  Good thing I'm not typically using the phone at 4:15 in the morning.  Bad news, if if I'm using the phone at 4:15 in the morning, it's probably something important and my automated restart is going to annoy me..

As mentioned before, this smells like a memory leak.  It has nothing to do with my ISP service.  I have a Vonage ATA plugged into the very same ethernet switch and it continues to have good service.  I have a constant probe of background traffic (at 8 packets/sec) measuring packet loss and latency and my ISP connectivity is fine.

So, hard to piss-and-moan about a $10/month add-on service that I mostly just use to keep a local  (previously CenturyLink POTS) number alive for my in-laws to call me on.  But I have to believe there are other customers that are using this service and rely on it.

M
Badge

I called 611 and opened an issue that's supposed to be escalated to engineering, who I'm supposed to hear back from in 2 days or so.  I referenced this support thread.

The automated power-cycle process is still in effect and takes place at 4:15:42 in the morning each day.  Support/engineering people - if you'd like me to discontinue this, please let me know.  In the mean time, this workaround keeps the service working well enough to prevent my wife from complaining to me.  Note that the call volume on this line is pretty low; mostly inbound junk calls. 

If there was some new software load deployed since I opened this issue that's supposed to address my issue, I can disable the automated restart and let it run for days or a week to see if the issue comes up again.  Despite periodic inquires about the status, I've not heard anything and have no reason to experiment unless something has changed.

S

After 3 weeks of no service on my line link and countless calls and texts, including tr a new SIM card, I figured out what was wrong:  bad AC adapte.  Haven’t tested it, but my guess is that the output voltage or current was off.  Tried another one, and it was off to the races!

Reply


New badge winners

Show all badges
Terms & ConditionsCookie settings