Finally, I decided it wasn't something I could debug/fix over the phone, so I packed up a laptop, cable tester, a bag of 8P8C5 plugs, a crimping tool for same, a couple of patch cables, and a pocket toolkit and headed out.
A quick look didn't reveal any obvious problems, and it didn't look like a DSL feed or anything, it appeared that the wire from downstairs was plain Ethernet. I decided to try the downstairs router next. It turned out not to actually be a router, but the power supply for the demarc. It consisted of two boxes, one plugged into the AC mains (I'm assuming it's a 48VDC supply) with a power LED6, and a bigger box containing a 7Ah SLA7 battery and some more indicators, and a heavy 2-conductor cord running off to the demarc. All looked well there.
Out at the demarc, there were actually two boxes. The cable came out of the ground and went to the smaller box first. I opened it, which revealed a sloppy coil of fibre and a splice (my mom has FiOS9 internet). Apparently it's just a fibre junction box, which would be where the connection is made from the underground feed to the demarc itself. was provision for a battery as well, but that one was not installed. There were POTS10 jacks, an F connector for TV (unused), and an 8P8C jack for ethernet, with a wire plugged into it. The link light was out again. I had brought my laptop with me, so I unplugged the ethernet wire, grabbed a patch cable, and plugged it into the jack in the demarc and into my laptop. Hey presto, I had internet connectivity! However, my mom immediately noticed a problem. The plastic block that holds the plug in place had come loose. Sure enough, it wasn't attached to the board at all. That could cause problems, as the cable would tend to fall off the connector. There was no obvious way to repair it (and it's Verizon property anyway), but we wanted to get back online, so I decided to stuff something springy in the box to hold the wire in the jack when the lid was closed. There was no foam around, so we ended up finding a little disc of coarse spongy material rather like Scotch Brite. It was a little thick for this use (we needed to get the pressure right so the lid would close and not warp but keep the wire in the jack). I tore the disc in two, aiming to produce two thinner discs of different thickness. After a little tweaking and adjusting, the door shut with an acceptable amount of pressure.
Then we went back upstairs to see if things were better off. They weren't. So we decided to check out the wire itself. I wasn't sure it was, in fact, the other end of the wire at the demarc, there could still be a mysterious additional box or splice somewhere, and my mom had noticed the wires didn't look properly seated. So we went back to the demarc, put the foam aside, unplugged the ethernet wire, and plugged it into one of the terminators that came with the cable analyzer. Then we went back upstairs and plugged the (presumed) other end of the wire into the cable analyzer. It went through its procedure and announced that the cable was faulty, the outer two pins (1 and 8) were not connected at all.
So I whacked off the end, grabbed a fresh 8P8C plug and crimper, prepared the end of the cable, put on the plug, and crimped it good. The cable analyzer confirmed that all 8 connections were in order. So I had my mom go back downstairs, remove the terminator, put the wire and the pads back, and close up the demarc again. Then we plugged the wire into the router and checked for internet. No go. Well, it had worked with my laptop, so perhaps the router was both broken and unnecessary. I tried plugging it directly into the computer. No go. I tried it again with my laptop and it worked instantly.
Maybe the router did perform some sort of magic that my fancy new laptop just happened to be capable of, but it was broken. With the computer plugged into the router, I was able to use the router's admin interface to determine that it was basically alive, but didn't see the internet connection. Maybe it needed a firmware update?
Using the internet connection to my laptop, I surfed to the router's manufacturer's web site and downloaded the firmware for that model and version, then hooked the laptop to the router and tried to install the firmware. No joy, it said it wasn't compatible.
Since the router's version was C3, which was between the C1 and D1 listed on the web site, I figured I'd try the D firmware. No, it didn't work either.
So I did some web research and found out that it's a special Verizon version of the router, running special Verizon firmware, which can only be upgraded with Verizon firmware. Then I went to the Verizon site, which suggested that installing new firmware can fix a variety of router problems, and gave a link to download it. But following the link resulted in a message that our web browser wasn't supported, it could only be downloaded using Internet Explorer, a giant steaming security hole of a browser that we didn't have on either of our computers.11
After all these machinations, the computer was unable to talk to the router any more. I remembered that my mom had pointed out that the monitor connector and the network connector leaned on each other, and adjusting one had been known to cause the other to fail. It occurred to me that there might be a third (or fourth) problem, the ethernet jack on the computer itself could be flaky (the laptop could still connect if I plugged the ethernet cable directly into it).
So I downloaded a movie on how to disassemble a Mac Mini (with a putty knife, it turns out), and my mom fetched a handy spreading knife. I pried it open, figured out how to remove the assembly containing the CD-ROM drive, disk drive, and fan, pulled out the memory, and removed the main board. Some examination under good light with a good magnifier (my mom had found a really nice German one in Europe a while back) didn't reveal any cracked solder joints or the like.
We discussed ordering a new computer or perhaps adding a USB hub and a USB ethernet jack, but neither of those were a Sunday night option (barring a drive to the all-night computer store in NYC).
I tried playing with the connector and jack with the bare board and noticed that the cable didn't want to seat properly. Were the 8P8C connectors I brought the right type? I tried plugging a fresh one in, and it didn't fit either, but I realized it wasn't crimped yet, so its contacts were somewhat proud of the slots they lived in. I tried a patch cable, and it popped right in. So I crimped the connector with no wire just to check, and it fit just fine. Perhaps it was a bad crimp, even though it worked with the laptop and the cable analyzer.
So I cut off that one and crimped another one on. It didn't fit either, nor did it work with the laptop. Eyeballing it carefully revealed the problem — the connectors I had brought were for stranded wire, and this cable was solid wire. The little contacts simply couldn't be mashed down correctly. And there weren't any sources available late Sunday evening in that part of the world.
Figuring that I had solved the mystery, I reassembled the computer (which was quite confused for a bit, as the clock battery had been out and it thought it was 1994, and couldn't connect to the internet for a time check). I manually set the time and tried the router again. No joy.
Then I realized I had been juggling several patch cables for several hours, and seen some things come and go. So I ran all the patch cables through the cable analyzer. Sure enough, the pretty one that was a handy length and had sporty boots on the ends was a dud. I cut off the ends of that one, so it wouldn't be mistaken for a working cable any more.
Swapping a good patch cable restored communication with the router. The flaky patch cable was stranded wire, and compatible with the connectors I'd brought, but this didn't solve the problem, it was less than a meter long, and would not reach to the demarc.
By this time, it was getting quite late, but we called around and found the local Wal*Mart was open 'til midnight, a scant 20 minutes hence. They said they didn't have any 8P8C wall jacks, but we figured we'd have a look and see if they were either mistaken or had something else useful, as we were rather running out of options.
The computer aisle yielded nothing of use, so I went scouting around for phone type stuff. I didn't find it, and the employees were avoiding me. Trying to chase one down that my mom had actually gotten to talk to her, I happened upon the phone stuff. No 8P8C stuff at all. Not even a peg for it. But I rifled through the carded wall plates and found one with a different label. Sure enough, it was a dual, with an RJ-11 up top and an 8P8C below! So we bought it, took it home, and I hand-wired it. No internet joy. So I had my mom go put the terminator back on the wire at the demarc and checked it with the cable analyzer. Completely backwards! Oops, I had guessed wrong on the wiring. So I reversed everything, tried it again, and that time the cable analyzer confirmed all 8 wires connected correctly. So again I had my mom go plug the cable back in at the demarc. And lo and behold, running a patch cable from that jack to the computer resulted in working internet service!
It was fairly ugly, with wires flapping around in the breeze, so I followed mom's suggestion to use the packaging the wall plate came in to wrap it all up and keep the cats off the wires.
I'll come back when she gets back from Orkney and replace the cable end with the right connector.
1demarc: Demarcation point. This is the box on your house where the phone wire comes in. It is officially where the phone company wiring ends and yours begins. They're generally constructed so you can open your part with an ordinary screwdriver, but the phone company's part opens with a separate fastener (often a big hex nut).
2DSL: Digital Subscriber Line, a method for delivering internet connectivity and telephone service over the same pair of wires at the same time.
3UTP: Unshielded Twisted Pair — the most common way of routing ethernet these days, using 8P8C5 connectors. It used to be done with coax, known as "thicknet" (which used big stiff coax and clunky bolt-on "vampire" transceivers that could only be attached to the cable at certain locations, and required special tools to core drill through the insulation and shield to get at the center conductor). There was also a version known as "thinnet", which used thinner coax and BNC4 connectors.
4BNC: a type of coaxial connector that attaches with a quarter-twist. It's easy to use, has good performance, and is popular with pro video gear, scientific instruments, oscilloscopes, and a variety of other purposes.
58P8C: 8 pin 8 circuit connector, this is the plastic locking plug that looks like a modular phone connector (RJ-11), and is the standard connection for UTP ethernet. It is often miscalled by the very similar looking (but incompatible and obsolete) RJ-45.
6LED: Light Emitting Diode. Basically a solid-state lamp, these are the little colored (or white, nowadays) lights seen all over electronic equipment built in the last 25 years.
7SLA: Sealed Lead-Acid, a generic term for lead-acid storage batteries that don't have caps that can be removed to add water or check electrolyte. Also known as maintenance free, non-spillable, or VRLA (valve regulated lead acid) batteries. These are very common in UPS8 and emergency lighting service.
8UPS: Uninterruptible Power Supply, a battery backup unit to provide power for equipment (often computers and medical gear) when the power fails.
9FiOS: Fiber Optic Service, phone, internet, and optionally television service delivered via a fiber optic cable instead of a pair of copper wires.
10POTS: Plain Old Telephone Service, a fun old acronym for ordinary phone lines. Occasionally known as PSTN for Public Switched Telephone Network. We were amused to see this silly old acronym on the shiny new FiOS demarc.
11I tried the download the next day, at work, where they did have Internet Explorer. Then Verizon attempted to install an "ActiveX" control, which is like taking IE's tissue paper security, soaking it in gasoline, and setting it on fire. My employer's security policy prohibits Active X, and for good reason. I called Verizon, who determined that the firmware on the router was in fact the latest available (which they could have mentioned on their web site, saving us much time), and explained that browser dependency like that is in violation of the current ADA12 laws, and companies have been known to suffer large fines for such violations.
12ADA: Americans with Disabilities Act, which includes provisions to ensure that people can use specific web browsers that support speech or other enabling features.