Verizon FIOS Static IP routing outage

Update: 12/30, 10 AM – problem appears fixed. Will call to find out what it was.

The backstory:

So on 12/22, Win noticed that email to Cathy wasn’t being delivered.  She’s using an IMAP server here at Serissa Galactic HQ, and our mail gateway, hosted on a virtual machine at Rackspace, normally delivers her mail to the IMAP server.

By two days ago, we figured out that in fact we can’t establish TCP connections between the mail gateway and systems at Serissa that happen to use a particular one of our 5 static IP addresses.  The others work fine.

This is just weird, but the VZ supplied Actiontec MI424 router is, well, just weird . . . but the problem isn’t the router.  After several hours of trying to configure various port forwarding and static NAT setups in the router, I called Verizon tech support.  After about 2 hours of phone hell, I got through to a fellow who was, well, clueful. It turns out you can set up screen sharing with them, and jointly click around in the router configuration screens.  The support rep eventually agreed with me that the problem existed, but at midnight December 23, there was not much help to be had from Actiontec.  He suggested connecting our system upstream of the router with a a switch, or using a different router if we had one.

I did not know that is now Verizon FIOS static IP works, but it makes a lot of sense.  There is an ethernet between the optical network terminal (ONT) and the Verizon supplied router, but you don’t have to use their router.  I unplugged the router and plugged in my macbook.  I said to Win “OK, I’m on the Internet… wait.  I am ON the internet!” I have actually never been before directly connected, not through a firewall, since Arpanet days.  Cool.

We have five IP addresses, and with the macbook running tcpdump, it was easy to see what wasn’t happening. With the macbook configured with our .10 address, we would attempt to open a TCP connection to our cloud system, but never got any replies.  Attempts to open connections from the cloud end never showed up.  By running netstat on the cloud end, we could see connections in “SYN_RCVD” state, but not getting ESTABLISHED.  Packets were going out, but not coming in.  Incidently, and strangely, ping, traceroute, and other ICMP stuff worked fine.

By changing the macbook IP address from .10 to .11 (another of our static IPs), it worked fine.

This was enough evidence to open a trouble ticket at Verizon.  We were told that they would get back to us in 24 hours…NOT.

In the meantime.  We changed our IMAP server to static NAT on a working IP address, and changed the port forwarding for inbound SMTP to match.  Now future email could be delivered, but 400 odd messages were stuck in the queue.  Win figured out how to add new Postfix rules to rerun the queue and translate the address, and we cleared the backlog.

Win also noticed that we can’t talk to, which may be hosted by Rackspace as well. The IP address is on a different class B, but it isn’t much different.  Same symptom.  We can’t talk to dropbox via our .10 address, but we can via .11 or other.

Christmas evening, after about 48 hours of silence from Verizon, I tried to get the trouble ticket status.  This is quite difficult. There is evidently no online way to do it, you have to go through phone hell.  After a few tries, I again got a skillful and helpful tech.    He told me that the ticket was assigned to the network techs, but there were no comments indicating anyone was working on it.  However, he searched around and found an outage report saying, roughly, that Massachusetts business fios static IP customers can’t talk to certain websites, and this outage report now had 75 trouble tickets linked to it.  He said he couldn’t tell me about other customers, but did mention trouble contacting, so I tried it.  We can’t talk to from .10 but we can from .11.

Our trouble ticket is now number 76, but there is no clue about who or when anyone might work on the problem.  Evidently other folks are much worse off than us, with their credit card processing machines unable to talk to the processors.

I will call back tomorrow or Monday to see what is going on.

I find this fascinating, but now fairly stress-free since Cathy’s email has been delivered.  What could cause reliable lossage of TCP connection setup, between stable, but seemingly random addresses?  Works fine for ICMP, but fails every time for SYN packets. fails, but works. fails, but works.  Maybe a corrupted hash table somewhere?  It seems like a very subtle and mysterious kind of thing.

Oh.  This blog is hosted by our cloud system, so I can’t talk to it via FIOS.  I’ve changed my laptop’s default route to use Win’s Comcast DSL instead, which works fine.  More proof that having a gigabit fiber between our houses is just a good idea.

One of the many problems with the Internet is that most people are at the mercy of their ISP.  The ISP controls the last mile and you have no real alternative.  Serissa happens to have both FIOS and Comcast links, but that isn’t as useful as you might think.  Inbound traffic knows about one or the other, and failover is manual and tedious.  I think we need an ASN so we can just let BGP deal with this, but that solution doesn’t scale well.

Update 12/26/2010 9 PM

We’ve found that our other IP addresses also don’t work … to different sets of sites.  For example, .11 can’t reach

I called Verizon at 888 244 8880 to report this and to find out ticket status.  I was on hold for 35 minutes and reached a fairly clueless agent this time.  He couldn’t get any information out of the network technician group, which probably means that no one is working on the problem.  He was able to pull up the group outage report RIEH032H87.

I asked why I couldn’t get online status, and he says because my trouble ticket is linked to a group ticket, I can’t see status anymore.  That seems unlikely.

I’ve created a #fios hashtag on Twitter, just for fun.

Update Monday 12/27/2010 11 PM

I called Verizon again to find out if there is any progress.  Evidently the problem has been passed up from the network technicians to IP Engineering, and the NOC.  This seems good.  However, according to the rep I talked to, they are looking into a theory that traceroutes along affected paths are showing the trouble outside the Verizon network.

That doesn’t match what I see.  As an example, from our .10 IP address, we cannot reach (this blog).  However, traceroute works.  From our .11 IP address, we cannot reach (never mind), but traceroute works.  From .10, patternreview works fine, and from .11, works fine.

Here’s (part of) the trace for .11 to

4 (  12.707 ms  4.781 ms  4.873 ms
5 (  14.932 ms  13.907 ms  13.301 ms
6 (  23.043 ms  12.119 ms  12.602 ms

Here’s part of the trace for .10 to

4 (  9.101 ms  9.121 ms  9.028 ms
5 (  18.682 ms  18.757 ms  21.011 ms
6  0.xe-4-1-0.XL4.NYC4.ALTER.NET (  21.096 ms  19.589 ms  19.308

The only common elements there are verizon (and the fact that the paths both go into Alternet.

Both traceroutes work all the way to the destinations, it is just TCP SYN/ACK packets that don’t come back.

I’ve heard a theory that someone is blacklisting fios addresses.  Until yesterday, we never used .11 for outbound connections, so I am skeptical.

In other news, we got about 14 inches of snow here. The kids are happy.

Update Tuesday 12/28/2010

Today’s wait on 888-244-8880 was 28 minutes.  Verizon needs better music on hold.

The representative today said the problem affects 71.x.x.x addresses (true) because when the 71 addresses were assigned to Verizon, website admins are notified to unblock them, but sometimes they don’t.

This is a fairly pathetic claim.  We’ve had the addresses for 5 years, they worked fine until a week ago, I control a machine I can’t talk to from one of my addresses, and ICMP traffic works fine, just not TCP.

It sounds like Verizon still has a theory about websites blacklisting Verizon addresses.  I think it is much more likely that some fancy router in the broken paths has a bad memory module,  My guess about which one it is based on the rather small differences between traceroutes of working paths and non-working paths. All of the non-working paths I know about pass from Verizon to Alternet in New York, for example, before branching off into other networks.    Try rebooting

6  0.ae1.BR2.NYC4.ALTER.NET (  26.290 ms  24.964 ms  24.691 ms

and see if that helps…

Update 12/29 at 11 PM

I called Verizon again.  As expected, there was a 35 minute wait on hold, and the representative said “they are still working on it”.  I asked for a supervisor and got very little more.  There are now 120 tickets linked to the group outage (up from 57), but there have been no comments added to the log since 12/27.   I suggested that certainly gave me the impression Verizon didn’t take the problem very seriously.

Cloud Deduplicaton

Here’s a thought experiment.

Consider the problem of online storage of music libraries.  There are various free sites that do this, and Apple is rumored to be planning cloud storage for users’ iTunes libraries, making it possible for a user to stream his or her music from anywhere.

It is fairly obvious that there is no need to store every copy of a song separately.  In the enterprise market, the idea of storing only one copy of each file, and then keeping track of who has a copy is called “deduplication”.

As an aside, this can work at the block level rather than the file level, and it can work even when common blocks of data in files are misaligned, due to some fairly cool technology called rolling hashes.

Now the upload phase of storing music into the cloud cannot work by just artist and title, because there may be many performances of a work that were separately recorded and are distinct.  Classical music fans are especially devoted to particular recordings of their favorites.  Consequently, the upload is likely to be accomplished by sending the hash of your local file, and the cloud server immediately says “yup, got that one!”  The user says, “Man, that upload was fast!”

Now it seems entirely possible to game the system, by sharing, not music, but <hashes of music>.  I go to the music hash warez site, and grab a set of hashes, and then I use my slightly hacked music uploader to say “Here are the hashes of the music I want to upload” and the cloud server says “yup, got those!”

Then, later, I can stream all the music, or reload my local copy after my local library is “accidently lost”.

So is it illegal to share hashes of music? Is the hash copyrighted?  Is this a bug in cloud deduplication? It would be a shame to require users to upload all the bits, just so the hashes can be computed in a trusted environment.

One possible solution is to compute keyed hashes. For example, the cloud server says “compute your music hashes using this personally customized algorithm”.  Of course, then the user can merely forward the instructions to a friend who <does> have the music.  Is it a crime to compute a hash function for someone?


Order of Operations – Evil and Pernicious

Back in November, my son came home with a 6th grade math test in which he lost a point because he put in parenthesis that were not strictly necessary, according to the order of operations.

Here’s the note I sent to the math teacher:

I’ve been meaning to write about this, but not getting around to it.  I am moved to write because on Alex’ recent math test, he lost a point because he put in parenthesis that were not necessary due to the order of operations.

I’m not going to argue about the grading, which is fine given the syllabus, but rather I want to express my view that teaching order of operations at all is evil and pernicious.

The only correct way to handle math is to always put in all the parenthesis.  Here’s why.

In 6th grade math, the order of operations is pretty simple, multiply and divide are “stronger” than addition and subtraction.  Once you get to the rest of mathematics, and then to programming languages, the situation becomes impossible.

I hate to cite wikipedia, but this article is relevant.

Just look at the page and its examples and just the visual impression of vast complexity is there.

It is beyond dangerous to teach these things <and expect folks to remember them>. They won’t remember the details, but <will think they know>.  Smart, capable engineers will write expressions, thinking they understand what they mean, and <they will be wrong>.

A few years ago I was working at SiCortex, and we built a custom chip with about 150 million transistors, as part of a supercomputer.  The logic is expressed in the VHDL programming language, which like many, has a defined order of operations.  An engineer did something quite innocuous, confusing the order of operations of logical OR and bitwise AND, and in consequence the mathematics expression meant something quite different than intended.  This was caught quite by accident, but had it gone through, the cost would have been a half million dollar replacement chip mask and about 3 months of schedule.

I very strongly feel that order of operations is a quaint dated idea that we really need to stop teaching and stop depending on.  If you always specify exactly what you mean by grouping operations with parenthesis, you and the computer will always agree about what the math means.

This also means that putting in the parenthesis, even if not needed, is a good idea, it makes the meaning of the expression clear without any risk.  This sort of care should be applauded, not penalized!

Some programming languages, like LISP, get this right – they don’t allow chained operations at all, and have no need for order of operations.  Of course they don’t even use infix operators.  In LISP, one says (+ 3 4) or (+ (* 2 4) (* 5 6)) and there is never any confusion about it.


PS  Don’t get me started about mean, median, and mode.  After 4th grade, has anyone actually used Mode?

Type conversion run wild

Many languages have the idea that if you assign a value of one type to a variable of another type, then the value will be converted to the same type as the variable.  So in C, for example

float x = 2;

converts the integer “2” to a floating point “2.0” before assignment.

So far so good.

Today I received an email with the following header field:


This is just outstanding! My best idea of how this happened is that a function intended to return a value of type email-address instead threw an exception, which was faithfully type-converted to an email address.

I will send a reply, just to see what happens.