A Rubble of Bits

Hello from within our modest tropical storm Irene.  Here it is just windy and rainy.  The power went off about 4 hours ago, right in the middle of the coffee maker cycle.  I dumped the rest of the water in the reservoir into a pan and brought it a boil on the gas stove, then poured it into the basket. Worked fine.  Without power you have to start the gas stove with a match, and the exhaust fan doesn’t work, but that is OK for minor cooking.
After about 15 minutes, the little UPS on the ethernet switches and FIOS router stopped working.  The FIOS optical network terminal kept running on its own battery.
I suspect this little neighborhood in Wayland is pretty low on NStar’s list of power problems, so I wheeled out the generator to the garage entrance. This is a 6KW electric start machine.  We haven’t needed it for several years, since a round of tree limb triming in town dramatically improved power reliability.  Unfortunately, the generator battery is ten years old,  and hasn’t worked for the last five.  I’ve never been successful in pull starting it unless it was already working, so I gave it a jump start from the DR field mower.
The generator plugs into the house via a 30′ pice of 10-4 cable with 30 Amp connectors.  The house connector in turn is wired to a manual transfer switch that moves 10 circuits from line to generator.  When the house was built, we thought pretty carefully about what to power:
* boiler controls, to permit hot water to work
* refrigerator
* freezer
* kitchen outlets
* outlet near the TV in the family room
* outlets in master bedroom
* outlets near the computer equipment in the basement
* outlet in the study (for my computer!)
* … and I don’t remember where the other two circuits are.  Note to self: find out.
Plus there is 300 feet of 12 gauge extension cord running across the lawn to the neighbor’s house to power their freezer.
This all made sense, but things change, and the house wiring hasn’t.  The FIOS ONT is in the utility room, and there is no generator outlet in there.  So now there is a 25 foot extension cord connecting it to the server outlets.  Similarly, we moved the freezer so now there is another extension cord connecting it to a powered outlet.
The little UPS is a problem. When the power came back on, the UPS hasn’t switched back. It just beeps fitfully. Note to self: a cheap UPS from Best Buy is probably worth every penny!
My son Alex was so offended by the lack of power for the family iMac that he’s moved it to the floor of the MBR and figured out which outlet is live. He also moved the Time Capsule that supports upstairs WiFi, and then I had to show him how to interpret the patch panel diagram to get it plugged into a live network port.  Cathy doesn’t approve of kids using the internet during a power outage,  but I figure I should reward initiative.
The home server had been up for 242 days, but it hasn’t restarted.  I will have to go troubleshoot.  The only difficulty with this is that we don’t have DNS service for the inside machines.  For talking to the world, we can just switch to Google’s DNS at 8.8.8.8, which is easy to remember.
The roof is leaking, but it is the place that just happens to drip into the kitchen sink.  Is that good planning or just luck?
I don’t know whether to expect FIOS to stay up long term or not.  The fiber goes to the local CO, which has lots of batteries, but I don’t know if there are active components between here and there, and I don’t know what is upstream from the local CO.
Updates:
The home server came up fine, and if you wait long enough, ssh to it works.  The problem is that its upstream DNS is the server in Win’s basement, which is down right now.
One of the smoke detectors is unhappy about the lack of AC power.  It probably needs a new battery, but it is the one about 14 feet off the ground in the loft.  I can reach it with the extension ladder, but that is out in the rain behind the house.  Ah well.
So far the chicken coop hasn’t blown over, and the run is still standing.  The chickens, sensibly, are staying inside.

I write this on a Google Chromebook while flying to San Francisco on Virgin America.
I am happy that the Google is trying out this concept, but it is on the wrong side of technology and its not what I want.

• Storage is cheap, communications are not
• Storage is low power, communications are not
• Local storage always works, communications does not
• My use of local storage is private, in the cloud there are watchers
• Local operations have predictable performance, remote does not

The key issue is that storage is really inexpensive and getting more so.  My three year old phone has 16GB os space. My iPad has 64GB.  The Macbook Air I covet has, well, who knows?  Removing the storage from the device solves a non-problem by introducing serious new problems.  I don’t get it.
My laptop (yes, a Macbook Pro) has a 500 GB drive. When I am disconnected, I can write, I can read, I can watch the movie backlog, I can program. I can learn. I can tag photos. I can do quite a  lot. I have pretty much my entire working set with me.  There are a couple of terabytes of other stuff laying around at home, but I don’t need that very often.
The pressing problem with mobile devices is power, not storage.  Why replace a low power storage device, that has predictable and good performance, with a slow, unreliable, communications channel that has a variable cost structure?
There are important roles for cloud storage:  backup, search, bulk processing, but it doesn’t make sense to move active storage to the other side of a high latency low bandwidth channel.  Let’s imagine that the communications is actually reliable and has zero variable costs for a moment.  But it still has, say 40 millisecond latency and a megabit or so bandwidth.  This is going to work file for email, chat, and so forth. But it cannot be a good video editor, or image browser.  I’ve had the experience of using Aperture to browse a few thousand photos on a local SSD. It is a surreal experience – the closest we’ve yet come to Minority Report.
The chromebook is a decent effort. I like the keyboard. The screen is nice, the weight is nice, the battery life is nice, but the lack of storage and a real local filesystem is just silly.

I have started a new job a couple of months ago, working part time at Quanta Research Cambridge.  I’ll say more about that later, but this post is about bicycles.
My new boss, Steve Heller, mentioned that one could park in downtown Lexington MA for two dollars a day, and take the bike path to Cambridge. From Lexington to the Alewife T station in Cambridge is about 6.5 miles, along a very nice bike trail, then it another 3.5 miles to Kendall Square, part path to Davis Square, and then down Hampshire Street.
This is a very fun ride, inbound is slightly downhill, 200 feet over 10 miles, with no particular hills.  Outbound is a little uphill, and mostly upwind in the afternoon, but fun.
Now there is another fellow at the lab, Willie Walker, who sometimes bicycle commutes in from New Hampshire, and that is a different matter altogether.  For some reason I thought he lived in the Western suburbs somewhere, so I thought I would try biking in from Wayland to Cambridge, which is about 18 miles each way.
I am not certain of the best route for this, but so far I take Route 20 to the old Boston Post Road to Weston center (4.4 miles) then Church Street up to 117, and 117 back to Route 20 in Waltham. Just past Prospect Park there is the Blue Heron trail that runs along the Charles River, from Waltham to Newton Corner.  From there you can go on the south side of the river along Nonantum Road to the Soldiers Field area, or you can go on Charles River Road and Greenough Drive along the North side of the river.  Both have bike lanes, although Nonantum is under construction.  At JFK Drive, I head in towards Harvard Square, but turn right on Mt Auburn Street and follow it to Central Square, then take Bishop Allen, Ellis, Harvard, and whatever else seems handy over to the office at 1 Kendall Square.
Inbound is easier than outbound, the Waltham hills are steeper on the East side, it is hotter in the afternoon, and still upwind. I now look forward to this and try to do it twice a week. When I can also do the Lexington route once a week I am a happy boy with another 100 miles.
It should be straightforward to beat my old SiCortex bicycling goals of 1000 miles a year.  But remember Will?  As of mid July, he’s already at 4000 miles for the year.
Oh yes, along the Blue Heron trail, about a mile and a bit from the Western end, is this beautiful bicycle and pedestrian bridge.

I’m becoming increasingly outraged by stories of American citizens having arbitrary searches of their laptops and other electronic gadgets at the border.    Here’s a Salon story on this, and there are lots of others:
http://www.salon.com/news/opinion/glenn_greenwald/2011/01/15/laptops/index.html
Generally I don’t think there should be a distinction between data transmitted into the country electronically and data carried by hand.  Electronic data can be searched, subject to FISA court oversight.  I think the same standards ought to apply to data carried by hand.  In addition, I don’t think the border agents should be able to seize and keep any electronic gadger that is legal to have.
I’ve written my congresswoman and senators, with approximately the following letter:

DHS claims the right to search and seize electronic devices at the border. Passwords are demanded from citizens on pain of refusal to admit them to their own country.  Devices are seized and not returned for months if ever.  There are no standards or accountability.
I support the ability of DHS and NSA to search electronic communications across the border, and it makes sense to permit searches of electronic devices as well, but demanding passwords and seizing devices is not OK.  i’d like you to work to stop these practices.
I would recommend that standards be put in place, requiring probable cause for such searches, that judicial oversight at least to the FISA level be applied, and that agents not be permitted to seize and keep devices or demand passwords.  These protections would put data transmitted electronically on the same footing as data carried by hand.
When data is searched, either it should not be retained, or the retained data should be kept under extreme protections, as it may include personal healthcare information or attorney client privileged information, or corporate trade secrets, and I think the government has a duty not to be careless with such things.

Since then, I’ve gone off to research FISA a bit more. FISA oversight applies to intercepts of communications of foreign nationals – no eavesdropping on Americans is allowed without probable cause and a warrant. I think that level of protection should apply to citizens whereever they are: inside or outside the country or at the border

Update: 12/30, 10 AM – problem appears fixed. Will call to find out what it was.
The backstory:
So on 12/22, Win noticed that email to Cathy wasn’t being delivered.  She’s using an IMAP server here at Serissa Galactic HQ, and our mail gateway, hosted on a virtual machine at Rackspace, normally delivers her mail to the IMAP server.
By two days ago, we figured out that in fact we can’t establish TCP connections between the mail gateway and systems at Serissa that happen to use a particular one of our 5 static IP addresses.  The others work fine.
This is just weird, but the VZ supplied Actiontec MI424 router is, well, just weird . . . but the problem isn’t the router.  After several hours of trying to configure various port forwarding and static NAT setups in the router, I called Verizon tech support.  After about 2 hours of phone hell, I got through to a fellow who was, well, clueful. It turns out you can set up screen sharing with them, and jointly click around in the router configuration screens.  The support rep eventually agreed with me that the problem existed, but at midnight December 23, there was not much help to be had from Actiontec.  He suggested connecting our system upstream of the router with a a switch, or using a different router if we had one.
I did not know that is now Verizon FIOS static IP works, but it makes a lot of sense.  There is an ethernet between the optical network terminal (ONT) and the Verizon supplied router, but you don’t have to use their router.  I unplugged the router and plugged in my macbook.  I said to Win “OK, I’m on the Internet… wait.  I am ON the internet!” I have actually never been before directly connected, not through a firewall, since Arpanet days.  Cool.
We have five IP addresses, and with the macbook running tcpdump, it was easy to see what wasn’t happening. With the macbook configured with our .10 address, we would attempt to open a TCP connection to our cloud system, but never got any replies.  Attempts to open connections from the cloud end never showed up.  By running netstat on the cloud end, we could see connections in “SYN_RCVD” state, but not getting ESTABLISHED.  Packets were going out, but not coming in.  Incidently, and strangely, ping, traceroute, and other ICMP stuff worked fine.
By changing the macbook IP address from .10 to .11 (another of our static IPs), it worked fine.
This was enough evidence to open a trouble ticket at Verizon.  We were told that they would get back to us in 24 hours…NOT.
In the meantime.  We changed our IMAP server to static NAT on a working IP address, and changed the port forwarding for inbound SMTP to match.  Now future email could be delivered, but 400 odd messages were stuck in the queue.  Win figured out how to add new Postfix rules to rerun the queue and translate the address, and we cleared the backlog.
Win also noticed that we can’t talk to www.dropbox.com, which may be hosted by Rackspace as well. The IP address is on a different class B, but it isn’t much different.  Same symptom.  We can’t talk to dropbox via our .10 address, but we can via .11 or other.
Christmas evening, after about 48 hours of silence from Verizon, I tried to get the trouble ticket status.  This is quite difficult. There is evidently no online way to do it, you have to go through phone hell.  After a few tries, I again got a skillful and helpful tech.    He told me that the ticket was assigned to the network techs, but there were no comments indicating anyone was working on it.  However, he searched around and found an outage report saying, roughly, that Massachusetts business fios static IP customers can’t talk to certain websites, and this outage report now had 75 trouble tickets linked to it.  He said he couldn’t tell me about other customers, but did mention trouble contacting www.experian.com, so I tried it.  We can’t talk to www.experian.com from .10 but we can from .11.
Our trouble ticket is now number 76, but there is no clue about who or when anyone might work on the problem.  Evidently other folks are much worse off than us, with their credit card processing machines unable to talk to the processors.
I will call back tomorrow or Monday to see what is going on.
I find this fascinating, but now fairly stress-free since Cathy’s email has been delivered.  What could cause reliable lossage of TCP connection setup, between stable, but seemingly random addresses?  Works fine for ICMP, but fails every time for SYN packets.  fios-10.serissa.com fails, but fios-11.serissa.com works. www.experian.com fails, but www.google.com works.  Maybe a corrupted hash table somewhere?  It seems like a very subtle and mysterious kind of thing.
Oh.  This blog is hosted by our cloud system, so I can’t talk to it via FIOS.  I’ve changed my laptop’s default route to use Win’s Comcast DSL instead, which works fine.  More proof that having a gigabit fiber between our houses is just a good idea.
One of the many problems with the Internet is that most people are at the mercy of their ISP.  The ISP controls the last mile and you have no real alternative.  Serissa happens to have both FIOS and Comcast links, but that isn’t as useful as you might think.  Inbound traffic knows about one or the other, and failover is manual and tedious.  I think we need an ASN so we can just let BGP deal with this, but that solution doesn’t scale well.
Update 12/26/2010 9 PM
We’ve found that our other IP addresses also don’t work … to different sets of sites.  For example, .11 can’t reach www.patternreview.com.
I called Verizon at 888 244 8880 to report this and to find out ticket status.  I was on hold for 35 minutes and reached a fairly clueless agent this time.  He couldn’t get any information out of the network technician group, which probably means that no one is working on the problem.  He was able to pull up the group outage report RIEH032H87.
I asked why I couldn’t get online status, and he says because my trouble ticket is linked to a group ticket, I can’t see status anymore.  That seems unlikely.
I’ve created a #fios hashtag on Twitter, just for fun.
Update Monday 12/27/2010 11 PM
I called Verizon again to find out if there is any progress.  Evidently the problem has been passed up from the network technicians to IP Engineering, and the NOC.  This seems good.  However, according to the rep I talked to, they are looking into a theory that traceroutes along affected paths are showing the trouble outside the Verizon network.
That doesn’t match what I see.  As an example, from our .10 IP address, we cannot reach www.stewart.org (this blog).  However, traceroute works.  From our .11 IP address, we cannot reach www.patternreview.com (never mind), but traceroute works.  From .10, patternreview works fine, and from .11, stewart.org works fine.
Here’s (part of) the trace for .11 to patternreview.com

Here’s part of the trace for .10 to www.stewart.org
4  so-7-2-0-0.BOS-BB-RTR2.verizon-gni.net (130.81.29.174)  9.101 ms  9.121 ms  9.028 ms
5  0.so-0-2-0.XL4.BOS4.ALTER.NET (152.63.16.141)  18.682 ms  18.757 ms  21.011 ms
6  0.xe-4-1-0.XL4.NYC4.ALTER.NET (152.63.3.102)  21.096 ms  19.589 ms  19.308
The only common elements there are verizon (and the fact that the paths both go into Alternet.
Both traceroutes work all the way to the destinations, it is just TCP SYN/ACK packets that don’t come back.
I’ve heard a theory that someone is blacklisting fios addresses.  Until yesterday, we never used .11 for outbound connections, so I am skeptical.
In other news, we got about 14 inches of snow here. The kids are happy.
Update Tuesday 12/28/2010
Today’s wait on 888-244-8880 was 28 minutes.  Verizon needs better music on hold.
The representative today said the problem affects 71.x.x.x addresses (true) because when the 71 addresses were assigned to Verizon, website admins are notified to unblock them, but sometimes they don’t.
This is a fairly pathetic claim.  We’ve had the addresses for 5 years, they worked fine until a week ago, I control a machine I can’t talk to from one of my addresses, and ICMP traffic works fine, just not TCP.
It sounds like Verizon still has a theory about websites blacklisting Verizon addresses.  I think it is much more likely that some fancy router in the broken paths has a bad memory module,  My guess about which one it is based on the rather small differences between traceroutes of working paths and non-working paths. All of the non-working paths I know about pass from Verizon to Alternet in New York, for example, before branching off into other networks.    Try rebooting
6  0.ae1.BR2.NYC4.ALTER.NET (152.63.18.37)  26.290 ms  24.964 ms  24.691 ms
and see if that helps…
Update 12/29 at 11 PM
I called Verizon again.  As expected, there was a 35 minute wait on hold, and the representative said “they are still working on it”.  I asked for a supervisor and got very little more.  There are now 120 tickets linked to the group outage (up from 57), but there have been no comments added to the log since 12/27.   I suggested that certainly gave me the impression Verizon didn’t take the problem very seriously.

Here’s a thought experiment.
Consider the problem of online storage of music libraries.  There are various free sites that do this, and Apple is rumored to be planning cloud storage for users’ iTunes libraries, making it possible for a user to stream his or her music from anywhere.
It is fairly obvious that there is no need to store every copy of a song separately.  In the enterprise market, the idea of storing only one copy of each file, and then keeping track of who has a copy is called “deduplication”.
As an aside, this can work at the block level rather than the file level, and it can work even when common blocks of data in files are misaligned, due to some fairly cool technology called rolling hashes.
Now the upload phase of storing music into the cloud cannot work by just artist and title, because there may be many performances of a work that were separately recorded and are distinct.  Classical music fans are especially devoted to particular recordings of their favorites.  Consequently, the upload is likely to be accomplished by sending the hash of your local file, and the cloud server immediately says “yup, got that one!”  The user says, “Man, that upload was fast!”
Now it seems entirely possible to game the system, by sharing, not music, but <hashes of music>.  I go to the music hash warez site, and grab a set of hashes, and then I use my slightly hacked music uploader to say “Here are the hashes of the music I want to upload” and the cloud server says “yup, got those!”
Then, later, I can stream all the music, or reload my local copy after my local library is “accidently lost”.
So is it illegal to share hashes of music? Is the hash copyrighted?  Is this a bug in cloud deduplication? It would be a shame to require users to upload all the bits, just so the hashes can be computed in a trusted environment.
One possible solution is to compute keyed hashes. For example, the cloud server says “compute your music hashes using this personally customized algorithm”.  Of course, then the user can merely forward the instructions to a friend who <does> have the music.  Is it a crime to compute a hash function for someone?
-L

Last Friday our FIOS Router died.
Actually it started to die two weeks ago, but I didn’t recognize the problem.  At first, my Macbook started failing to connect with the WiFi.  I was puzzled, but just switched to another access point.  Friday, however, the router started dropping packets on the LAN.  My daughter, for whom latency is a matter of WOW life and death, got on Bonjour to complain about the service to my neighbor Win.  (We have fiber in a conduit between the houses for Serissa Research).  Win in turn IMed me over Bonjour.  We found we couldn’t even connect to the FIOS router web server.
Down to the engine room…er.. basement.  I turned off the router and turned it on again, and it didn’t even light up.  Then I discovered the wall wart power adapter was super hot.  Never one to ignore a clue, I said “Aha!” to myself.  5 Volts DC at 3 Amps.  I’m an engineer, how hard could it be?  After rummaging through various boxes, I found a 5 Volt at 2 Amps supply from some DLink thing with the right connector, and we’re up and running again, for the moment.
Verizon says they will send a new power supply.
The moral?  Peer to peer IM like Bonjour lets the kids complain about the internet service without having to walk downstairs.  Those old-technology centralized things like AIM and Jabber only work when the Internet works.
This issue is also why I am leery of custom home NAS boxes like Drobo, for all their good properties.  I want something I can fix using junk PC parts late at night on a weekend, not something that requires a week turnaround time.  I am not religious about this, and you will pry the Time Capsule from my cold fingers.