A Rubble of Bits

Update: 12/30, 10 AM – problem appears fixed. Will call to find out what it was.
The backstory:
So on 12/22, Win noticed that email to Cathy wasn’t being delivered.  She’s using an IMAP server here at Serissa Galactic HQ, and our mail gateway, hosted on a virtual machine at Rackspace, normally delivers her mail to the IMAP server.
By two days ago, we figured out that in fact we can’t establish TCP connections between the mail gateway and systems at Serissa that happen to use a particular one of our 5 static IP addresses.  The others work fine.
This is just weird, but the VZ supplied Actiontec MI424 router is, well, just weird . . . but the problem isn’t the router.  After several hours of trying to configure various port forwarding and static NAT setups in the router, I called Verizon tech support.  After about 2 hours of phone hell, I got through to a fellow who was, well, clueful. It turns out you can set up screen sharing with them, and jointly click around in the router configuration screens.  The support rep eventually agreed with me that the problem existed, but at midnight December 23, there was not much help to be had from Actiontec.  He suggested connecting our system upstream of the router with a a switch, or using a different router if we had one.
I did not know that is now Verizon FIOS static IP works, but it makes a lot of sense.  There is an ethernet between the optical network terminal (ONT) and the Verizon supplied router, but you don’t have to use their router.  I unplugged the router and plugged in my macbook.  I said to Win “OK, I’m on the Internet… wait.  I am ON the internet!” I have actually never been before directly connected, not through a firewall, since Arpanet days.  Cool.
We have five IP addresses, and with the macbook running tcpdump, it was easy to see what wasn’t happening. With the macbook configured with our .10 address, we would attempt to open a TCP connection to our cloud system, but never got any replies.  Attempts to open connections from the cloud end never showed up.  By running netstat on the cloud end, we could see connections in “SYN_RCVD” state, but not getting ESTABLISHED.  Packets were going out, but not coming in.  Incidently, and strangely, ping, traceroute, and other ICMP stuff worked fine.
By changing the macbook IP address from .10 to .11 (another of our static IPs), it worked fine.
This was enough evidence to open a trouble ticket at Verizon.  We were told that they would get back to us in 24 hours…NOT.
In the meantime.  We changed our IMAP server to static NAT on a working IP address, and changed the port forwarding for inbound SMTP to match.  Now future email could be delivered, but 400 odd messages were stuck in the queue.  Win figured out how to add new Postfix rules to rerun the queue and translate the address, and we cleared the backlog.
Win also noticed that we can’t talk to www.dropbox.com, which may be hosted by Rackspace as well. The IP address is on a different class B, but it isn’t much different.  Same symptom.  We can’t talk to dropbox via our .10 address, but we can via .11 or other.
Christmas evening, after about 48 hours of silence from Verizon, I tried to get the trouble ticket status.  This is quite difficult. There is evidently no online way to do it, you have to go through phone hell.  After a few tries, I again got a skillful and helpful tech.    He told me that the ticket was assigned to the network techs, but there were no comments indicating anyone was working on it.  However, he searched around and found an outage report saying, roughly, that Massachusetts business fios static IP customers can’t talk to certain websites, and this outage report now had 75 trouble tickets linked to it.  He said he couldn’t tell me about other customers, but did mention trouble contacting www.experian.com, so I tried it.  We can’t talk to www.experian.com from .10 but we can from .11.
Our trouble ticket is now number 76, but there is no clue about who or when anyone might work on the problem.  Evidently other folks are much worse off than us, with their credit card processing machines unable to talk to the processors.
I will call back tomorrow or Monday to see what is going on.
I find this fascinating, but now fairly stress-free since Cathy’s email has been delivered.  What could cause reliable lossage of TCP connection setup, between stable, but seemingly random addresses?  Works fine for ICMP, but fails every time for SYN packets.  fios-10.serissa.com fails, but fios-11.serissa.com works. www.experian.com fails, but www.google.com works.  Maybe a corrupted hash table somewhere?  It seems like a very subtle and mysterious kind of thing.
Oh.  This blog is hosted by our cloud system, so I can’t talk to it via FIOS.  I’ve changed my laptop’s default route to use Win’s Comcast DSL instead, which works fine.  More proof that having a gigabit fiber between our houses is just a good idea.
One of the many problems with the Internet is that most people are at the mercy of their ISP.  The ISP controls the last mile and you have no real alternative.  Serissa happens to have both FIOS and Comcast links, but that isn’t as useful as you might think.  Inbound traffic knows about one or the other, and failover is manual and tedious.  I think we need an ASN so we can just let BGP deal with this, but that solution doesn’t scale well.
Update 12/26/2010 9 PM
We’ve found that our other IP addresses also don’t work … to different sets of sites.  For example, .11 can’t reach www.patternreview.com.
I called Verizon at 888 244 8880 to report this and to find out ticket status.  I was on hold for 35 minutes and reached a fairly clueless agent this time.  He couldn’t get any information out of the network technician group, which probably means that no one is working on the problem.  He was able to pull up the group outage report RIEH032H87.
I asked why I couldn’t get online status, and he says because my trouble ticket is linked to a group ticket, I can’t see status anymore.  That seems unlikely.
I’ve created a #fios hashtag on Twitter, just for fun.
Update Monday 12/27/2010 11 PM
I called Verizon again to find out if there is any progress.  Evidently the problem has been passed up from the network technicians to IP Engineering, and the NOC.  This seems good.  However, according to the rep I talked to, they are looking into a theory that traceroutes along affected paths are showing the trouble outside the Verizon network.
That doesn’t match what I see.  As an example, from our .10 IP address, we cannot reach www.stewart.org (this blog).  However, traceroute works.  From our .11 IP address, we cannot reach www.patternreview.com (never mind), but traceroute works.  From .10, patternreview works fine, and from .11, stewart.org works fine.
Here’s (part of) the trace for .11 to patternreview.com

Here’s part of the trace for .10 to www.stewart.org
4  so-7-2-0-0.BOS-BB-RTR2.verizon-gni.net (130.81.29.174)  9.101 ms  9.121 ms  9.028 ms
5  0.so-0-2-0.XL4.BOS4.ALTER.NET (152.63.16.141)  18.682 ms  18.757 ms  21.011 ms
6  0.xe-4-1-0.XL4.NYC4.ALTER.NET (152.63.3.102)  21.096 ms  19.589 ms  19.308
The only common elements there are verizon (and the fact that the paths both go into Alternet.
Both traceroutes work all the way to the destinations, it is just TCP SYN/ACK packets that don’t come back.
I’ve heard a theory that someone is blacklisting fios addresses.  Until yesterday, we never used .11 for outbound connections, so I am skeptical.
In other news, we got about 14 inches of snow here. The kids are happy.
Update Tuesday 12/28/2010
Today’s wait on 888-244-8880 was 28 minutes.  Verizon needs better music on hold.
The representative today said the problem affects 71.x.x.x addresses (true) because when the 71 addresses were assigned to Verizon, website admins are notified to unblock them, but sometimes they don’t.
This is a fairly pathetic claim.  We’ve had the addresses for 5 years, they worked fine until a week ago, I control a machine I can’t talk to from one of my addresses, and ICMP traffic works fine, just not TCP.
It sounds like Verizon still has a theory about websites blacklisting Verizon addresses.  I think it is much more likely that some fancy router in the broken paths has a bad memory module,  My guess about which one it is based on the rather small differences between traceroutes of working paths and non-working paths. All of the non-working paths I know about pass from Verizon to Alternet in New York, for example, before branching off into other networks.    Try rebooting
6  0.ae1.BR2.NYC4.ALTER.NET (152.63.18.37)  26.290 ms  24.964 ms  24.691 ms
and see if that helps…
Update 12/29 at 11 PM
I called Verizon again.  As expected, there was a 35 minute wait on hold, and the representative said “they are still working on it”.  I asked for a supervisor and got very little more.  There are now 120 tickets linked to the group outage (up from 57), but there have been no comments added to the log since 12/27.   I suggested that certainly gave me the impression Verizon didn’t take the problem very seriously.

Here’s a thought experiment.
Consider the problem of online storage of music libraries.  There are various free sites that do this, and Apple is rumored to be planning cloud storage for users’ iTunes libraries, making it possible for a user to stream his or her music from anywhere.
It is fairly obvious that there is no need to store every copy of a song separately.  In the enterprise market, the idea of storing only one copy of each file, and then keeping track of who has a copy is called “deduplication”.
As an aside, this can work at the block level rather than the file level, and it can work even when common blocks of data in files are misaligned, due to some fairly cool technology called rolling hashes.
Now the upload phase of storing music into the cloud cannot work by just artist and title, because there may be many performances of a work that were separately recorded and are distinct.  Classical music fans are especially devoted to particular recordings of their favorites.  Consequently, the upload is likely to be accomplished by sending the hash of your local file, and the cloud server immediately says “yup, got that one!”  The user says, “Man, that upload was fast!”
Now it seems entirely possible to game the system, by sharing, not music, but <hashes of music>.  I go to the music hash warez site, and grab a set of hashes, and then I use my slightly hacked music uploader to say “Here are the hashes of the music I want to upload” and the cloud server says “yup, got those!”
Then, later, I can stream all the music, or reload my local copy after my local library is “accidently lost”.
So is it illegal to share hashes of music? Is the hash copyrighted?  Is this a bug in cloud deduplication? It would be a shame to require users to upload all the bits, just so the hashes can be computed in a trusted environment.
One possible solution is to compute keyed hashes. For example, the cloud server says “compute your music hashes using this personally customized algorithm”.  Of course, then the user can merely forward the instructions to a friend who <does> have the music.  Is it a crime to compute a hash function for someone?
-L

Last Friday our FIOS Router died.
Actually it started to die two weeks ago, but I didn’t recognize the problem.  At first, my Macbook started failing to connect with the WiFi.  I was puzzled, but just switched to another access point.  Friday, however, the router started dropping packets on the LAN.  My daughter, for whom latency is a matter of WOW life and death, got on Bonjour to complain about the service to my neighbor Win.  (We have fiber in a conduit between the houses for Serissa Research).  Win in turn IMed me over Bonjour.  We found we couldn’t even connect to the FIOS router web server.
Down to the engine room…er.. basement.  I turned off the router and turned it on again, and it didn’t even light up.  Then I discovered the wall wart power adapter was super hot.  Never one to ignore a clue, I said “Aha!” to myself.  5 Volts DC at 3 Amps.  I’m an engineer, how hard could it be?  After rummaging through various boxes, I found a 5 Volt at 2 Amps supply from some DLink thing with the right connector, and we’re up and running again, for the moment.
Verizon says they will send a new power supply.
The moral?  Peer to peer IM like Bonjour lets the kids complain about the internet service without having to walk downstairs.  Those old-technology centralized things like AIM and Jabber only work when the Internet works.
This issue is also why I am leery of custom home NAS boxes like Drobo, for all their good properties.  I want something I can fix using junk PC parts late at night on a weekend, not something that requires a week turnaround time.  I am not religious about this, and you will pry the Time Capsule from my cold fingers.

We got a promotion code for a Redbox movie rental somewhere, and I went to use it.
Evidently, the machine was not on its home screen when I got there, but on the search screen instead.  Consequently, the “rent with a promotion code” button was not visible.  I followed the only path available, which was swiping my credit card.  I expected a “promotion code” box on the checkout screen, but there wasn’t one.  Instead I got the dvd out of the slot.  Alice in Wonderland, if you want to know. <After this> the home screen came up, showing the “rent with promo code” option, but it was too late.
Well this is bad, I thought.  My first instinct was to just put the disk back in, but I decided to call customer service first.  They said you have to be on the home screen and I would be charged, but on the bright side, my promo code was still good.
This sort of thing enrages me, and I didn’t want to waste any more time on them, but Cathy said to call back and ask for a supervisor.
The new agent, (Hi Jessica!), gave me the same story, but kindly went off to fetch a supervisor.  I never got to talk to him or her, but they say they are fixing it for me, and will refund the dollar when I return the disk, and have cancelled the code.
Good – Redbox seems to be doing the right thing here, but it cost me a lot of time calling them, and it is costing them two expensive customer service calls.
Bad – The UI design seems wrong.  It shouldn’t be modal, with the customer having to choose between pay or promo before starting!  In my case, I never even saw the choice because when I got to the machine it was already on a search screen down the “pay” path.  It is possible some previous person left it that way.  I don’t know if there is a timeout back to the home screen, but adding one is the wrong fix!  A better design is to have the promo code screen as part of the checkout flow, where the “what is your email” already is.  Doesn’t even add a screen that way.
I’ll let you know if the credit doesn’t appear.  Netflix is starting to look better already, although we don’t watch many movies.

There is a gap in function between mobile phones and landlines.  Mobiles can send and receive text messages and landlines cannot.  This should be fixed.
Idea 1 – text messages for landlines
The network already has technology to send caller ID. The same signalling could be used to transmit text messages.  Caller ID works by sending an ascii character string to the phone between the first two rings. The signalling is at 1200 bps, and could last for three seconds, giving room for up to 450 characters.  Since texts are shorter than that, the central office could repeat the message for reliability, or we could add technology to let the phone acknowlege the signal.  The upstream ack could be modem tones or touch tones, which the phone can already generate.
Since current phones generally do not have good displays, and don’t expect text messages, this capability could be added to answering machines instead of phones.
* The answering machine could record the text for later display
* The answering machine could send a tone or modem acknowlegement
* In principle, the answering machine could have cell-phone like software to generate texts using T9 predictive keyboarding, or use a full keyboard.
* An answering machine with this capability could answer calls with a distinctive beep or tone sequence that inform the CO and equipment at the caller end that the capability to receive texts was present.
I know this is starting to sound like minitel!
Idea 2 – Texting <during> calls
I frequently call someone to get a phone number.  On other occasions, folks call me to get a number. We do this in speech.  The recipient has to write down the number or try to remember it.  When the caller is mobile, this adds to the danger of using the phone while driving.
Why not make it easy to send numbers during the call?
* If you press the keypad during the call, the phone will send touch tones.
* The receiving phone should detect this, and greatly attenuate the earphone path, so the tones don’t blast the ear, and should remember the digit sequence.  After the call, the phone could make those numbers available to dial.
* On the iPhone, and probably others, in the contact list there is a “share contact” button. This can send a contact via text.  If you try to do this during a call, the phone could send the number in-band as above.
* For landline users, an answering machine (as above) could listen in parallel to an arriving call and record such in-band messages.
Rich phone companies, send your licensing inquiries to me!

There is a lot of grumbling out there about how the iPad and the iPhone are closed environments, and you can only run programs that are blessed by Apple, and available through the App Store.
It occurs to me that one is perfectly free to write software for your own use, simply by joining the Apple developer program.  You can download the SDK and write code anytime.  If you want to test the code on a real device, then you need to join the paid developer program, which currently costs $99/year.
As long as you don’t use pre-release features of the iPhone OS, you may even be able to share your source code with others.  There are lots of sites out there with code examples.
So perhaps the iPhone and iPad are not closed at all, you can run Apple approved software for no extra charge, but you can run anything you write for $99/year.