“If It Ain’t Broke…”

Or How I Upgraded my DSL Service in 12 Not-So-Easy Steps

[This is a rather long posting, and a bit geek-techie.  It’s an narrative of my experience with a DSL service upgrade, so I hope it has value for other folks, especially for those looking to upgrade their own Internet service.  And I’m asking for advice at the end, as I’ve got to purchase a new router: What kind of network router is best for VDSL2 service?]

“If it ain’t broke, don’t fix it.”  Ageless advice from folks who have been there and done that.  I’ve been attempting to apply this wisdom to our home-office Internet connection for over a year now.  Problem was, our DSL/ADSL service from Qwest was acting up, throwing temporary intermittent, but fairly regular, outages (which I’m calling “drop-outs” in what follows) which would last for several minutes before connectivity to the ‘Net would be restored. 

That the Internet works at all is a wonderment to me.  Like biology (no engineer in his or her right mind would ever “design” a cell, or an intestine, or an octopus!… well, maybe an octopus; but definitely not a platypus!), the Internet is messy and organic — it’s just largely “happened,” evolved from rather simple (if ambitious) beginnings over the past couple of decades.

And that we can connect to the Internet and use it from home, not just business and industry, is itself a marvel.  We take it for granted, especially with routine use.  “The network is the computer.”  My wife needs her daily doses of Farmville, my daughters Facebook, email and newsletters arrive unbidden, something always needs to be searched for, we’re into Internet radio up to our ears… And now I’m blogging.  Sporadically, but I’m committed (or should be).

So, given the random, intermittent nature of our ADSL service, when Qwest started sending us promotional pieces (electronic and traditional) announcing that our Franktown neighborhood was finally blessed with “Heavy Duty™ Internet”, I took it as a clue from on-high that this was the time to address the intermittency problem while getting a connection speed upgrade.  Heck, the offer might even save us some money.  Turns out that I was partly, mostly correct.

I don’t share the general antipathy towards Qwest.  In spite of their past bad-rep for customer service (etc.), they deliver dial-tone and bits, and we’re mostly happy with that.  So, this won’t be an anti-Qwest rant — rather, I’m happy to report that the HD-Internet upgrade was a largely pleasant and mostly successful experience; devil’s in the details.  Truth is, when you dink around with something as organically complex as your Internet connection, you’re liable to uncover problems that you didn’t even know were there.  Sharing this may help some other homeowner out.  I’ve broken the following narrative into numbered steps for cross-referencing… In actuality, it wasn’t all that clear-cut.

Helpful hint:  If you’re planning to take Qwest up on an upgrade to HD-Internet (technically known as VDSL2), be sure to actually accept one of their “promotional offers,” rather than just initiating an upgrade inquiry by yourself.  The reason for this in a moment (or skip down to 9 below)…

1. I called the 1-800 phone# on one of the promo-pieces they’d sent, and a nice lady took me through the options, confirmed that my neighborhood, and specifically my phone line, qualified for the upgrade, talked me through the pricing levels and tradeoffs, and — especially because she showed me that my monthly bill for Internet would actually be lower with the upgrade — she took my order.  That’s right, higher speed DSL for less money: what was formerly costing us about $59/month drops to ~$35/month.  Good deal, and I’d be foolish to turn up my nose at it… (Strangely, I had to accept the deal with “MSN Services” to get this price; we’re a Linux household, and have no interest in using MSN as “our home page,” but without it, the VDSL service costs more!)

2. This resulted, about 5 business days later, with a box delivered containing my new VDSL modem, roughly coinciding with Qwest’s service order to “upgrade the service line” — swap some gear, and the old ADSL modem is now consigned to the useless e-junk pile.  Using the “idiot-proof” instruction pamphlet, it really was a snap to get the new modem physically connected to my existing DSL phone line.  Should’a been, right?… I’m just substituting new technology & speed for old.  Simple assumption, almost accurate: Hooking up the wires correctly is one thing… getting everything “configured” is certainly another.

That’s where the plot thickens — on the assumptions.  In order to deliver a technology service to the maximum number of folks, Qwest itself makes a number of assumptions, some of which are valid, others not so much.

3. Qwest assumes that you (me… er, the customer) are “technologically naive” and a reader of “XYZ for Dummies” for anything we’d need to know about technology.  Thus, the “idiot-proof” instructions and color-coded phone and Ethernet cables packaged with the new modem.  Everything is broken down into a few nice, simple, imperative statements:  “1. Which phone outlets need filters?” (with a simplified line drawing of a filter)… “2. Plug in the power cord.”  Et cetera.  Check — modem installed (hooked up).

That’s all fine, as far as it goes — After all, all I’m doing is plug-for-plug replacing the old modem with the new.  So far, so good.  But Qwest makes a few more assumptions, ones that go beyond an “idiot-proof” upgrade…

They assume that we’re connecting one or more Microsoft Windows PCs.  Wrong guess.  We’re a 100% Linux (Ubuntu) shop here at RickerNet Enterprises.  Thus, Qwest’s “High-Speed Internet Online Installation v7.4” installation CD-ROM, with several megabytes worth of Windows software, does me no good at all.  Besides which: We’re installing a DSL modem here… There’s no software needed on any PC (other than perhaps some “help text,” an online manual maybe?) to activate or operate this modem — it’s independent and autonomous part of my SOHO-LAN, for pete’s sake!  (N.B. to large corporations: Contrary to your market studies, the world does not revolve around Microsoft, and those of us who use Linux, or even Macs, are not social pariahs.)

Which begs the bigger question: Qwest assumes, correctly in my case, that the modem will be hooked up to an “internal” WAN-to-LAN router (one of a few standard configuration scenarios that they support) — in my case, a Linksys BEFSR81: check.  Furthermore, they know that said router will be doing Network Address Translation (NAT), allowing the home-based (internal) PCs (and other devices) to be IP-addressed on a “private LAN network”, a reserved Internet Protocol (IP) address group which allows for a bunch of non-public IP-addresses to be assigned to all my home-based gear (and thus helping defer the day when the world expends all available public IPv4 addresses).  Also, check.  (Static vs. DHCP addressing is irrelevant to the discussion here.)

Private network IP-addresses look like 172.16.x.x, 192.168.x.x or 10.0.x.x.  The fun comes from then next assumption: Qwest, like most suppliers of network devices, presumes that everyone uses a 192.168-net private network numbering scheme.  Not-check.  For a variety of business reasons, some present, some past, my SOHO network uses the 10.0-net scheme.  The new DSL modem, which now needs to be configured, needs simply to be connected directly by Ethernet wire to one of my PCs.  But wait!  The modem is configured, by default, to respond to IP 192.168.0.1 — my PC is IP’d on the 10.0-net.  By convention (it’s intended to work this way), a 10.0-net device cannot talk to a device on a 192.168-net (or any other IP network)… that’s part of the essence of NAT.  My PCs talk to each other, and to the router, and it talks to the outside Internet world for us.

4. What’s required next is for me to “re-IP” my PC — that is, temporarily assign it with an address from the 192.168-net group, effectively removing it from the rest of my SOHO network.  Swapping Ethernet wires around for the exercise is trivial — actually convincing my PC to “change IP-address gears” takes some digging around with some network admin tools that don’t see much use once things are properly set up… But I double-down, dig out some old notes in my computer log-book, and get it figured out (remember, once done with this configuring stuff, I’ll have to “put my PC back the way it was,” re-re-IP it back to the 10.0-net).

Once this preliminary stuff is all done, I can now (hopefully) get to work on configuring the DSL modem.  Remember, this is s’posed to be a “drop-in replacement” exercise, right?

5. So, correctly (temporarily) wired and re-IP’d, I next bring up a web-browser (Firefox! …not Microsoft’s IE!) and type in the modem’s IP-address in the browser’s address bar: http://192.168.0.1 — hit return, and viola!  There’s the modem’s “Setup Screen”.  Now I can start exploring and configuring… In particular, how can I tell the DSL modem to become part of my SOHO 10.0-net?…  Lots of screens and displays to explore, much of it familiar, because I’ve traveled some of these roads before.  But what to configure for my exact situation?

6. Cut to the chase — The “idiot-proof” install pamphlet helpfully suggests: “For questions about installation (or anything else), you can reach us 24hours a day at 1-800…” [their emphasis].  So I call for some help just to speed things along, and get a guy who actually seems like he knows what he’s doing.

Unfortunately, our conversation starts with the tedious re-capping of “did I follow all the instructions correctly” (N.B. to technology vendors: Please don’t assume that all of us who call your help-desk are inexperienced!  Give us a magic-code-phrase so that your support agent can “take off the training wheels” and get right to the technical problem!) before I can get him to listen to my one and only question and reason for calling:  “How do I set up the modem to work with my internal LAN 10.0-net?”  Oh.  Well, he does know what he’s doing, because he’s very forthcoming with the answer:  Set the DSL modem for “IPoE transparent bridging”, and then “transfer your login credentials to your router so it logs you into our Qwest DSL network.”

Okay!  With this one call, in pretty short order, we’re now pinging Internet domain addresses and we’re browsing websites.  Seems like we’re done — Thank you, telephone support guy!  All that’s left is to reconnect the Ethernet wires back to their “normal” wiring configuration (modem to router, router to PCs, etc.), and then to restore my own PC’s IP-address to it’s “normal” place in the 10.0-net — and finally re-check that everything’s still working.  And… it does.  Mission complete.  Maybe.

Note that this still leaves me with one minor problem — the DSL modem, now in transparent bridging mode, still thinks that its own IP-address is part of the 192.168-net.  But I’ve restored my own PC back to its 10.0-net address, so I can’t check on the modem’s internal state (counters, etc.) unless I re-re-configure it back to the 192.168-net (and move an Ethernet wire each time).  Or, I can figure out how to re-IP the modem itself to become part of my 10.0-net — looking through a few more setup screens — Oh, there it is: just set its “LAN IP Address” to 10.0.1.1, and now maybe I can see it either through the router, or with a quick Ethernet cable change.

7. So here’s where things sat, just after hanging up with the tech-support guy and restoring everything to its “new normal” situation:  I’ve upgraded an ADSL service, one which had intermittent drop-out issues (and which was comparatively slow), to a spiffy new VDSL2 service, all by swapping out a modem (well, there was some work at Qwest’s end, too — but that’s what they’re in business to do).  Indeed, I could immediately tell that there’s a world of difference between the ADSL and the new VDSL — we’ve gone from ~1.5Mb/sec service to one that’s rated for up to 12.0Mb/sec, roughly 10 times faster.  Yup, web pages “jump” onto the screen instead of crawling, email downloads sprightly, and… but wait!  It still has drop-outs!

With any technology upgrade, it pays to observe and contemplate things before leaping to conclusions.  With so many “variables” having just changed, it takes time to come to understand the new state of affairs.  But still… after about a day of watching and experiencing the new service, it’s definitely still dropping-out randomly and intermittently.  This means that: a) The ADSL drop-outs were not a figment of our imagination; b) Swapping out the old ADSL modem for the new one didn’t fix the problem (it wasn’t a faulty modem); c) Even though we’re getting faster line service (when it works), there’s still a lingering problem, and that it could be anything from Qwest’s DSLAM, through the copper line to our house, to our hookup box, to our inside wiring.  Problem is: what and where?

8. Time for another service call to technical support.  But I was less enthusiastic about making this one; it’ll likely turn into a “who’s problem is it?” kind of conversation.  Made the call: Got a nice lady on the help-desk who immediately wanted to walk me through her standard “trouble-shooting script,” mostly the same stuff as I’d covered previously with the smart tech-support guy.  When we get to  “Please power-cycle the modem…” “Please reset the modem to factory defaults…” (effectively undoing the entire setup from the previous evening… go back to step 4 above to unwind from this “diagnostic mistep”), I’m starting to get frustrated.  And so on… she was polite, but adamant about sticking to her “script,” none of which addressed the issue of why my service was dropping out.  (N.B. reprise to Qwest:  Please, a code-phrase which gets me to advanced help and bypasses the Windows help-script!)

Then she came to the script sections which said, effectively, to “Reboot your Windows PC…” and then to “Click on the Start button…”; I had to gently call a halt to the whole charade.  “No,” I informed her, “I’m not going to reboot or click on the Start button — mine’s a Linux PC; we don’t reboot.”  That brought her to a full stop (script pages are shuffled… to the emergency stuff at the back).  “Sir, we have to ask you now to please call your PC manufacturer for further help.”  Classic help-desk bailout:  When faced with non-Microsoft Windows stuff, refer customer to a non-existent authority who cannot possibly help or know anything about the problem.  Point the finger somewhere else.

9. I don’t take the bailout; instead, I asked for her supervisor.  When Steve came on the line, I gave him a summary of the situation, and he quickly grasped the fact that help-desk support was out-of-its-depth on this one.  He agreed that on-site technical support would be required.

Then he asked me a couple of times, in different ways: “Did you initiate the HD-Internet upgrade yourself, or did you respond to a promotional offer?”  At first, I didn’t get why he was asking this, but it turned out that answering “Yes, I responded to a promo offer” was the correct response, and here’s why:  If Qwest’s marketing and promotional activities, like a TV ad or direct mail piece, are what convinces you to upgrade to Heavy Duty Internet, then any on-site service calls that Qwest determines it must make to get things working are done for free.  Otherwise, if you “just decide on your own” to do the upgrade, Qwest can (and probably will) charge you for the service call(s).  So here’s your helpful hint:  Always answer “Yes!  I took you up on your promotional offer!”

Once we figured that out, Steve promised to have a DSL-experienced network tech at my door within a five-hour window the next day — a Saturday, it turned out!  When Bob arrived, he told me that he’d already been working on my line, at the neighborhood Qwest substation box, and had found and fixed “a couple of problems.”  Great!  “Let’s look at your outside service connection box next…” — “Yup, I see what the original installer guy did.  Mostly it’s right, but look.  He got this part wrong…”

Through our conversation, it became apparent that Bob really new his stuff at the network wiring layer, and he was quite aware of and experienced with the kind of “intermittent drop-out” problem we were having.  As he examined and tinkered with various wiring connections, he explained a lot more to me about the electronics and signaling end of DSL than I’d been able to scrounge from the ‘Net by myself.

He corrected some wiring, validated my phone filtering, approved my self-installed modem, and we hooked up his own (Windows) laptop to the modem.  He knows his stuff, at least the physical network layer, and showed me the modem setup screen where the DSL signal strength, attenuation, retrain, framing and error counts can be monitored.  I traded him that for some informal training on IP/NAT and “higher level” networking stuff — fair exchange.

After watching the modem monitoring screen for quite a bit, we were both convinced that the existing line quality and signal strengths were really good (better than expected), and over a period of many minutes, we observed a zero-error rate for the line.  Looking good — that should fix the intermittent drop-outs, right?  When he left (the first time), we were both convinced that “we’d nailed it” — that the intermittents were a thing of the past.

10. Fortunately, Bob didn’t leave the neighborhood right away — and he left me his cellphone number.  Within 20 minutes of Internet use, I again observed several intermittent drop-outs, so I called him and he rolled back in to check the modem again.  This time, the modem’s monitoring display continued to show “zero errors,” “no retrains” and “great signal strength” — apparently (and convincingly), nothing wrong with the new modem.  At this point, I was as sure as Bob was that the Qwest VDSL signal to and through the modem was rock solid.  So what’s the problem?  We discussed some additional troubleshooting techniques and ideas, and it was again time to observe and contemplate (see Step 7 above).  Clearly, Bob had found and resolved more than a couple of long-standing DSL problems, yet the intermittents continued.

Often in debugging complex systems (yes, a residential DSL service qualifies as “a nontrivial system”), you’ll find that fixing one (or more) apparent causes of a problem can contribute to an overall fix — but at the same time, those fixes can uncover additional problems which either are contributing to the symptoms, or are merely “laying in wait” (latent) for their turn to contribute to the problem.  Rather than a single, simple cause of symptoms, it’s a chain of causes, some of which may mask others.  Fix the ones that mask, and you uncover the latents.  Gotta identify, understand and fix ’em all.

11. Bob promised to talk to his “other experts in the shop,” and would get back to me.  He did: Consensus is that my old Linksys router (its firmware date tells me that it’s really at least 10 years old, “ancient” technology) is suspect, and may be unable to keep up with the higher data transmission rates.  To me, this is plausible, not just “buck passing.”  We have: a) eliminated the old ADSL modem (now discarded); b) verified that the new VDSL modem is reporting “no errors,” and it’s unlikely to be “broken out of the box”; c) fixed previous wiring and circuit problems, all the way back to the DSLAM, seem to be resolved; leading me to conclude that d) Qwest is delivering “clean” DSL all the way to/through the modem.  This leads to the router as the next and most likely culprit.

So, this leaves me at 12. Looks like I need to replace that router.  Question is:  With what?  Googling for “best VDSL router” has led me to a bunch of “product comparison” pages, none of which provide me with enough technical information or practical-use advice to make a quick decision.  Any solid advice, based on experience, out there?  Got Qwest’s “HD-Internet” (or some other vendor’s form of VDSL2)?  What router are you using?  What setup and/or performance issues have you encountered?  I’d appreciate your feedback.

Summary:  In the final analysis, even with the service call and help-desk phone calls, and with all the Windows-centric assumptions, Qwest really delivered on this one — actually exceeded my expectations.  Each Qwest representative whom I talked to was pleasant, and helpful within the scope of what they’re tasked to do at their job.  Kudos especially to Steve, the supervisor, and to Bob, the network tech who came to the house, and followed up with call-backs as he’d promised.  And, officially:  Thank you, Qwest!

But!  Don’t assume (there’s that word again) that a “simple DSL speed upgrade” will actually be… simple.  If you’re planning to do an upgrade of your own, don’t be discouraged by this story — but do be prepared for something more than just “swapping in a new modem.”  In particular, it’s likely that your existing phone circuit will need some on-site attention from an experienced network technician to support the higher DSL data transmission speed.

This upgrade actually helped uncover the systemic issues that have been lurking in our DSL service, probably from the date of installation (several years ago).  Why weren’t these show-stopper problems before?  Simple answer: The lower ADSL data rates of less than 1.5Mb/sec simply were not “stressing the system” enough to really matter.  All we’d see was an occasional drop-out, enough to be noticed and annoying, but not bad enough to muster a service call for.  With the upgrade to 12Mb/sec service, and with latent line problems resolved, we’re now stressing the weak link (the old router) enough to really matter.

I research a lot of geek-type stuff when I’m browsing the ‘Net — seems like all the stuff dealing with networking (especially for the home) is pretty superficial; every now and then, a true nugget of wisdom shows up.  I wrote this long article to document my own experience to-date, with the hope that a comprehensive narrative might actually help someone else.  Somebody with more, or different, experience than mine just might contribute a bit of wisdom or a fix — that’s the nature of open-systems collaboration.  Certainly, I’ll welcome any feedback, even if it’s not about a recommendation for a router.

Obviously, this personal saga continues.  I’m confident that I’ll resolve the router issue with a suitable replacement unit.  Hopefully, that’ll be the end of “intermittent drop-outs.”  I will update this story (with brevity) when I learn and fix more…

—