Town Meeting

Here in New England, many towns are run by Town Meetings.  It is very democratic. Things that need voting, such as the budget, accepting gifts of land, and putting up no-left-turn signs at popular intersections are written up as articles for Town Meeting.  Any resident who wants a say in how things are run has to show up.
Votes used to be taken by the moderator’s estimate of ayes and nays, and if that seemed close, by standing votes, and if that seemed close, by counts taken by pairs of volunteers.
These days we using Electronic Voting.  Each voter gets a little wireless keypad.  When the voiting lamp is on “the window is open” you press 1 to record a Yes or 2 to record a No.
This got me thinking.  It sometimes happens that a small subset of residents is Very Upset about something, like those no-left-turn signs.  Eventually, the rest of us get sleepy, and someone makes a non-debatable motion to close debate.  This requires an immediate vote and a 2/3 majority.  How about making the process continuous?  The voting gizmos have 10 buttons, I think you should be able to press “terminate debate” at any time, and whenever the total goes over 2/3 that’s it.
I would also recommend buttons for “the current speaker is annoying” and so forth, the results of which are put up on the jumbotron, but tiny steps for tiny feet.
And why do sponsors of popular articles, like the new solar panels, go on for 10 minutes presenting all the details of something that is going to win in a landslide anyway?
 

Cryptographic Modules

Steve Bellovin has a post The Uses and Abuses of Cryptography in which he comments on the recent Anthem data breach.  At Anthem, supposedly, the database of important stuff like customer addresses and social security numbers was not encrypted, because it was in use all the time.Steve says, “If your OS is secure, you don’t need the crypto; if it’s not, the crypto won’t protect your data.”
His point is that the decryption keys have to be in RAM somewhere for the system to work, so if the OS is insecure, the keys can be stolen, and the encrypted database decrypted anyway.  This is not necessarily true.  IBM (see, for example IBM PCIe Cryptographic Coprocessor) http://www-03.ibm.com/security/cryptocards/pciecc/overview.shtml) and others make hardware units that provide encryption and decryption, and store the master keys.  With appropriate hardware, the keys are NOT in RAM and can’t be stolen.  This still isn’t enough, because a compromised host system can still command the crypto box to decrypt the data.  To go further, you have to have velocity checks inside the trusted part of the system, to alarm on and halt unexpected volumes of traffic.
The data itself also has to be carefully organized. Each record has to have its own key.  Record keys are stored in the database encyrpted by a master key, which is only stored in the cryptographic hardware module.
It is probably going to be impossible to prevent theft of individual records.  An insider can always photograph data off the screen in a call center.  I think we can do much better about technical means to prevent bulk data breaches.
There is a whole new area of research on how to make cloud computing trustworthy.  How can you get anything done when your code is running on potentially compromised hardware or on a virtual machine pwned by the bad guys?  It might be possible!  Homomorphic encryption makes it possible to perform computations on encrypted data, and perhaps cloud servers will at least come with cryptographic modules that at least can limit the rate at which your data can be stolen.
Update: February 24, 2015
Steve points out via email that many kinds of tasks, such as a batch job generating annual statements, have to touch all records, so rate-limiting (velocity checks) might not be effective.

Net Neutrality

I wrote a letter to the editor of the Wall Street Journal today.  In my opinion, Internet service providers and backbone providers should be “common carriers”.  They should not be allowed to charge different rates for different bits, and they shouldn’t be allowed to even look at the traffic other than for routing.  Today I was so offended by the disingenuousness and misrepresentation of L. Gordon Crovitz’ op-ed that I felt compelled to respond:

Timothy Lemmer
Letters Editor
Wall Street Journal

Regarding “The Great Internet Power Grab” by L. Gordon Crovitz, Feb. 8, 2015.  Mr Crovitz is misinformed or disingenuous.

The FCC proposes to reclassify broadband Internet access services – consumer access to the net – as a telecommunications service rather than as an information service.  The FCC does not propose to regulate content providers or startups providing innovative services, or end users of any sort.

Mr. Crovitz proposes we should be so afraid of unlikely future abuses by regulators that we should not move to stem current and actual abuses by the cable and telephone industries that provide the majority of internet access.

  • Verizon spies on customer communications to install tracking cookies (1)
    Comcast demands payments from content provider Netflix merely to get access to customers (2)
  • ATT blocks customers who attempt to encrypt their own email (3)
  • These are actual abuses by companies exploiting their near monopoly positions to damage competition, harm innovation, and endanger customer privacy.

It would be great if Congress would get its act together to promote innovation and forbid discrimination.  Until then, the FCC appears to be doing its best to protect the public from the telecom companies who are the current unaccountable gatekeepers of the net.

Lawrence Stewart
Wayland, MA

References

(1) https://www.eff.org/deeplinks/2014/11/verizon-x-uidh
(2) http://www.wsj.com/news/articles/SB10001424052702304899704579391223249896550
(3) http://arstechnica.com/tech-policy/2014/11/condemnation-mounts-against-isp-that-sabotaged-users-e-mail-encryption/

Windows 7 Disk Upgrade

It is a mystery to me why laptop makers charge such a premium for SSDs.  Well, no, it’s not a mystery, they do it because they can.  Part of the reason is that it is such a pain, in the Windows world, to upgrade.
Cathy recently got a new HP ProBook 640 G1, replacing her ancient Vista machine.  The new laptop came with a 128 GB SSD, which served its purpose of demonstrating how dramatically faster the SSD is than a regular hard drive, but it is too small.  Her old machine, after deleting about 50 GB of duplicate stuff, was already at 128.
It is much cheaper to buy an aftermarket 256 GB SSD than to buy the same laptop with a larger SSD. so we set about an upgrade.
HP Laptops, at least this one, do not ship with install disks, instead, they come with a 12 GB “recovery partition” that soaks up even more of the precious space.  You can reinstall the OS from the recovery partition as often as you like, or you can, exactly once, make a set of recovery DVDs or a recovery USB drive.
There are two main paths to doing a disk upgrade:

  • Replace the disk, and reinstall from the recovery media
  • Replace the disk, make the old disk an external drive, and clone the old disk to the new one.

The first path is less risky, so we tried that first.  I had purchased a nice, large USB3 thumb drive for the purpose, and … the HP recovery disk creator would not create a USB!  What is this, 2004?  HP support is actually quite good, and I suppose that is part of what you pay for when you buy a “business” notebook.  They were surprised by this lack of functionality, since it is supposed to work, and eventually decided to send us recovery media.  They sent DVDs, which is not what we want, but fine.
The HP media worked fine to install onto the new 256 GB SSD, but did not restore much of the HP add on software.  Most manufacturer add-on software is crapware, but HPs isn’t bad.  We got most of the missing bits from the support.hp.com website except for the HP documentation!  You can get the PDF files for the user and service manuals, but not the online HP Documentation app.
Our plan was eventually to trickle down the 128 GB SSD to one of the kids, so we didn’t mind using up its ability to create recovery media, so we tried that next.  Rather than screw up the almost-working 256 GB drive, we installed an old 160 GB drive from Samantha’s old Macbook (replaced earlier by an SSD).
The home-created recovery media did better, installing all the HP add-ons…except the documentation!
Now with three working drives, and two sets of recovery disks, I felt confident enough to try the alternative: cloning the original drive.  I had a copy of Acronis True Image 2010, but couldn’t find the disk for it.  The new SSD came with a copy of True Image 2014, but first I read up on the accumulated wisdom of the Internet.  There’s a guy, GroverH, on the Acronis forums (see  https://forum.acronis.com/forum/3426 ) who has an astonishing set of howtos.
Manufacturers who use recovery partitions really don’t want you to clone drives, perhaps this is pressure from Microsoft.  It works fine if the new drive is exactly the same as the old one, but if not, unless the partition sizes are exactly the same, the result is not likely to work.  The cloning software will scale the partitions if you restore to a bigger drive, but they won’t work.  You have to manually tweak the partition arrangement.  Typically the recovery partition is at the end, the boot partition is at the beginning, and the “C:” drive uses the space inbetween.
Now earlier when I couldn’t find the True Image install disk on another project, I tried the Open Source CloneZilla and was quite happy with it.  It is not for the faint-hearted, but it seems reliable. I used CloneZilla to make a backup of the original drive, and then, because the recovery media had already created a working partition structure, merely restored C: to the C: of the experimental 160 GB drive.  Windows felt like it had to do a chkdsk, but after that it worked, and lo, the HP documentation was back!  (And Cathy’s new screen background.)
As the last step, we put the 256 GB SSD back in, and used CloneZilla to restore C: and the HP_TOOLS partition contents that weren’t quite the same in the original and recovered versions.
Whew!
So, contrast to a disk upgrade on a Mac:  Put in new drive, restore from Time Capsule, done.  And this restores all user files and applications!
Next challenge: migrating Cathy’s data files and reinstalling applications.  Memo to Microsoft:  it is just unreasonable that in this new century we still have to reinstall applications one by one.
 

Hotel Internet – Hyatt French Quarter

I write from my room at the Hyatt French Quarter.
Your hotel internet service stinks.
I would rather stay in a Hampton Inn or like that than a Hyatt.  You know why?  The internet service in cheap hotels just works.  Yours does not.
You advertise “free internet”, but it costs rather a lot in the inconvenience and irritation of your customers, who are paying you quite a lot of money for a nice experience.
I have three devices with me.  A laptop, a tablet, and a phone.  On each one, every day of my stay, at (apparently) a random time, each one stops working and I have to connect again.
Here is what that takes:

  • Try to use my email.  Doesn’t work
  • Remember that I have to FIRST use a web browser.
  • Connect to hotel WiFi (ok, this step is expected, once)
  • Get browser intercept screen
  • Type in my name and room number
  • Wait
  • Read offer to pay $5 extra for “good” internet service, rather than crappy. The text says this offer “lasts as long as your current package”  is that per day? Per stay? What?
  • Click “continue with current package”
  • Wait
  • Get connected to FACEBOOK.

Why?  I can’t explain it.  People my age think Facebook is something kids use
to share selfies.  The kids think Facebook is for, I don’t know, old people, they
are all on Twitter.
Then I have to remember what I wanted to do.
Are you serious?  Do you think this process, repeated for my three devices, EVERY DAY, is going to make me recommend your hotel?
Now let us talk about privacy.
It irritates me that you want my name and room number. I do not agree that you can track my activities online.  It is none of your business.  I run an encrypted proxy server back home.  So all your logs will show is that I set up one encrypted connection to the cloud for my web access.  My email connections are all encrypted.  My remote logins to the office are all encrypted.  My IMs are encrypted.
I read the terms and conditions, by the way.  They are linked off the sign on page.   They are poorly written legalese, and there are a number of ways to read them.  One way says that you track all my connections to websites but only link them to my personally identifiable information if you need to “to enforce our terms and conditions”.  They also say that you have no obligation to keep my activities confidential.  And who or what is Roomlynx?
Even if your terms said otherwise, I wouldn’t believe you.  I don’t trust you OR your service providers.
Here’s my suggestion:
I think all this effort you’ve gone to is a waste of time, effort, and money. You do not have the technical means to monitor or control how I use the net anyway, so why make your customers jump through hoops?
If your lawyers tell you these steps are necessary, get different lawyers who have a clue.  If you still think it is necessary, have the terms and conditions be attached to the room contract!
If you seriously have a problem with non-guests soaking up your bandwidth, then by all means add a WiFi password, and hand it out at checkin.
If you seriously have a problem with bandwidth hogs, then slow down the connections of actual offenders.
Basically, try your best to make the Internet work as well as the electricity you supply to my room.  I turn on the switch, the lights go on. Done.
By the way, modern OS’s like Apples MacOS Yosemite, frequently change the MAC address they use. This will likely break your login system, raising the frustration of your guests even more.  They will not blame Apple for trying to protect their privacy.  They will blame you.  I already do.
PS  I don’t like to help you debug a system that is fundamentally broken, but:

  • The hotel website still says Internet costs $9.95 per day.  Update that maybe?
  • There is no way to go back and pay the extra $5 for better service one you’ve found out how crappy the regular stuff is.
  • After you connect, you can no longer find the terms and conditions page
  • I accidently tried to play a video, and your freaking login screen showed up in the video pane.  That just makes you look even sillier.

Random Walks

One blog I follow is GÖDEL’S LOST LETTER
In the post Do Random Walks Help Avoid Fireworks, Pip references George Polya’s proof that on regular lattices in one and two dimensions, a random walk returns to the origin infinitely many times, but in three dimensions, the probability of ever returning to the origin is strictly less than one.
He references a rather approachable paper explaining this by Shrirang Mare: Polya’s Recurrence Theorem which explains a proof of this matter using lattices of reisistors in an electrical circuit analogy.  The key is that there is infinite resistance to infinity in one or two dimensions, but strictly less than infinite resistance to infinity in three dimensions.
This is all fine, but there is another connection in science fiction. In 1959, E.E. “Doc” Smith’s The Galaxy Primes was published in Amazing Stories.
Our Heros have built a teleporting starship, but they can’t control where it goes.  The jumps appear long and random.  Garlock says to Belle:

“You can call that a fact. But I want you and Jim to do some math. We know that we’re making mighty long jumps. Assuming that they’re at perfect random, and of approximately the same length, the probability is greater than one-half that we’re getting farther and farther away from Tellus. Is there a jump number, N, at which the probability is one-half that we land nearer Tellus instead of farther away? My jump-at-conclusions guess is that there isn’t. That the first jump set up a bias.”
“Ouch. That isn’t in any of the books,” James said. “In other words, do we or do we not attain a maximum? You’re making some bum assumptions; among others that space isn’t curved and that the dimensions of the universe are very large compared to the length of our jumps. I’ll see if I can put it into shape to feed to Compy. You’ve always held that these generators work at random—the rest of those assumptions are based on your theory?”

Garlock is right – this is a three dimensional random walk and tends not to return to its starting place, but James is wrong when he says this isn’t in any of the books.  Polya proved it in 1921.
 

Duress passwords and other side effects

Fifteen years ago when we built our house, we had a home security system installed.  It has the usual alarm panel with a keypad inside the door.  When you come in the house, you have 30 seconds to key in your password to stop the alarm from going off.
If the alarm does go off, the monitoring company will call you to find out if it was a mistake or a real alarm.  Each authorized user has a passcode to authenticate themselves to the monitoring company.  You can’t have the burglar answering the phone “No problem here! False alarm…”
In fact, there are two passcodes, one authenticates you, and the other is a duress password.  If the burglar is there with you, you use the duress password, and the monitoring company behaves exactly the same way, but they also call the local police for you.  It is important that the burglar cannot tell the difference.
It seems to me that ATM cards should have duress PINs as well as real ones.  If a criminal says “type in your ATM pin or else” then fine, you enter the duress PIN.  The ATM behaves exactly the same way, but the bank alerts the police and sends them the surveillance video.
Duress passwords have a lot of other potential uses.  If your school principal demands your facebook password, you give up your duress password.  What happens next could depend on which password you give.  At the extreme, your whole account could be deleted.  It could be archived on servers out of legal jurisdiction, your stuff visible only to friends could seem not to exist for a week.  Whatever.  Options that appear not to do anything are best, because then the school admins can’t tell you have disobeyed them and suspend you.
While I am riffing, there should be a  phrase you can say, like “I do not consent to this search” or a similar account setting, that makes the administrator’s access an automatic CFAA violation. (I think the CFAA should be junked, but if not, it should be used to user’s benefit, not just the man’s.)
Finally, regarding authentication, there should also be two-factor authentication for everything, and single-use passwords for everything.  Why not?  Everyone has a nice computing device with them at all times.  Of course your phone and the authentication app should have a duress unlock code.
So next time you are building an authentication structure, build in support for one-time passwords, two factor authentication, and a flexible set of duress passwords.
 

Home HVAC

It has finally gotten cold here.  Right now is it about 17F outside.  Previously we had been getting by with just the heating zones for the kitchen/family room and the master bedroom turned on.
A few days ago, the boys had trouble getting to sleep while we were watching TV, because the noise from the set was keeping them up.  Alex closed the door.  The next morning, I noticed it was 55F in their room.  Well, I reasoned, the heating zone up there is not turned on, and with the door shut, warm air from the rest of the house can’t get in so easily.  I turned on the heat.  The next night Alex happened to close the door again, and in the morning it was 52F.  That isn’t so good.
Friday we had the neighbors over for dinner so I turned on the dining room heat.  A couple hours later I went to check on it and it wasn’t any warmer.

Heating System
Heating System Central in the basement

This is our heating system.  This is a gas fired hot water system.  The “boiler” is the green box on the lower left.  It heats water to 160F or so.  From there, there are 9 heating zones. The horizontal pipe manifold in the front is the the return path to the boiler. The vertical pipes with yellow shutoffs representing the returns for each zone.  The supply manifold is behind, along with the pumps and so forth. One zone heats water in the blue tank for domestic hot water faucets and showers.  The other zones have circulating pumps that feed tubing that zigzags under the floors .  This is called radiant heating.
Radiant Zone Manifold
Hot water is routed through many plastic tubes that warm the floor from below.

Each zone typically has a manifold like this one that routes hot water through synthetic rubber tubes that are stapled to the undersize of the floors, and insulated below that to direct their heat upwards.  This lets you walk around on warm floors and actually get by with colder air temperatures.  Our oldest daughter was in the habit of leaving the next day’s clothes on the floor covered with a blanket, so they would be prewarmed in the morning. Notice that one tube is turned off. That one runs underneath the kitchen pantry, which we try to keep colder.
In the main system photo, on the left, you can see electronics boxes on the wall. Here’s a closeup.
Zone control box
Zone control box

Each zone has a thermostat, which comes into one of these boxes.  This is a three channel box, with three 24 volt thermostats coming in on brown wires at the top, and red wiring for three 120 volt zone circulator pumps at the bottom.  The box also signals the main boiler that heat is being called by at least one zone. Each zone has a plug in relay, one of which I have unplugged.
The circulator pumps look like this
Circulator Pump
Taco 007-F5 Circulator Pump

So there is a central gas water heater, which feeds a number of zones. Each zone has a water circulation pump, controlled by a thermostat. The pump feeds hot water through rubber tubes on the underside of the floors.
Individual zones have failed before.  I have fixed them by replacing the circulator pump.  You can get these anywhere.
New circulator pump
New circulator pump

The hardest part about replacing these is the electrical wiring, which is hardwired by wirenuts in the green box attached to the pump.  First, turn off the power.  I did this by physically pulling the relay for the appropriate zone.  Then I measure the pump current using a clamp on ammeter. Then I measure the voltage.  Only then do I unscrew the wirenuts protecting the wires, and without touching the bare wires, touch the end to ground. Then brush the wire with the back of your hand only.  If the wire is live, the electricity will contract your arm muscles, pulling your hand away.  If you can’t think of at least four ways to make sure the wires are not live, hire someone to do this for you.  Really. There are old electricians, and there are bold electricians. There are no old, bold electricians.  I am an old electrician.
Our system has shutoff valves immediately on both sides of the pump.  By turning those off, you can swap out the pump without draining all the water out of the system.  As you can see in the picture, the pump is held in place by flanges at the inlet (bottom) and outlet (top). Each flange has two stainless steel bolts, so they won’t rust.  In a burst of cleverness or good design, the nuts on these bolts are 11/16 and the bolts themselves are 5/8, so you can take them apart with only one set of wrenches.  Here’s the pump I removed.
Old pump
Old Pump

 
Note the corrosion inside the pump.  I put the new pump in place and turned this zone back on, and now the dining room was getting heat.  While I was down there, I took a look at this thing.
Air removal valve
Hy-Vent air removal valve

 
This is an air removal valve. It is installed on top of the boiler, along with a pressure relief valve.  On some intuition, I lifted the pressure relief valve toggle, and air came out, followed by water. That is not good.  The water for a heating system like this comes from town water, which has dissolved gas in it. Typically this will be air, although in the Marcellus Shale areas it can be natuural gas (in those areas, you can set your sink on fire).  Air is bad for forced hot water systems.  it corrodes the inside of the pipes, and water pumps won’t pump air, usually.  If the radiant tubes get full of air, they will not be heating. By the way, these pipes are so rusty because some years ago the boiler was overheating to the point that the relief valve was opening, getting water everywhere. This was because the temperature sensor had come unstuck from the pipe it was measuring. Fixed by a clever plumber with a stainless pipe clamp.  As collateral damage from rapid cycling, I had to get a new gas valve too.  Separate story.
After waiting a few few minutes, I tried the relief valve again and got more air.  This meant that the air removal valve wasn’t working, and probably some of my zones weren’t working because of air-bound pumps or bubbles in the pipes.  You might be wondering how the valve knows to let out air, but not water.  Inside the cylinder is a float.  When there is water inside, the float rises and closes the output port.  When there is air inside, the float falls, opening the outlet port and letting out the air.  It is pretty simple.  I called a plumber friend to see if he could fix this and he said “if you can replace a zone pump, you can replace this valve too.” Basically, you turn off the system, close all the valves, to minimize the amount of water that will come out, depressurize the system, and work fast.  A new valve was $13 at Home Depot. The fact they had 10 in stock suggests they do go bad. Unfortunatly I failed to depressurize the system as well as I thought, and I got a 3 foot high gusher of 130F water.  Be careful!  Heating systems run at around 10 psi. The pressure comes partly from town water pressure through a pressure regulator, and partly from the expansion of hot water.  There is an expansion tank to reduce that effect.
The next day, I tried the pressure relief valve again and got water immediately. Probably this means the new valve is working.
Temperature Gauges
Temperature Gauges

Each zone has a temperature gauge.  You can see that the two on the right are low, and the two on the left in this picture are not.  The right hand zone had the pump I replaced. The next one was not turned on.  The temperature gauges are there because you don’t want to run 160F water through these radiant tubes. The floors will get too hot and the tubes won’t last very long. Instead, each zone has a check valve and a mixing valve.
Check Valve
Check Valve

The check valve keeps the loop from flowing backwards, or generally keeps it from circulating by gravity.  Cold water is slightly denser than hot water, so the water on the colder side of the loop will fall, pulling hot water around the loop even without the pump running. The spring in the check valve is enough to stop gravity circulation.
Mixing Valve
Mixing Valve

The mixing valve has a green adjusting knob. This valve mixes hot water from the boiler with cooler water from the return leg of the zone, and serves to adjust the temperature of the water in each zone.  Some water recirculates, with some hot water added.
When I turned on the zone second from the right, it did not work.  The temperature gauge stayed put at 80F, (conducted heat through the copper pipes).  I used my ammeter to confirm the pump was drawing power.  I turned off the valves for all the other zones, so that this one would have more water.  Didn’t work.
Measuring pump current
Measuring pump current

There are three reasons why a hot water zone might not work:  the pump is not spinning, the pump is trying to pump air, or the pipes are clogged.  I had just replaced a pump to fix a zone, but was there a second bad pump? Or something else?
I have an intra-red non-contact thermometer, and I used it to measure the pump housing temperatures.  The working pumps were all around 125F, the non working pump was around 175F. That might mean that it was stalled, and not spinning, or that it was pumping air, and not being cooled by the water.  I had one more spare pump, but I was getting suspicious.
I got to wondering if the pump I removed was really broken. I knew that these Taco 007-F5 pumps have a replaceable cartridge inside, but since the cartridge costs almost as much as a new pump I had never bothered with it.  I decided to take apart the pump I removed to see what it looks like.
Disassembled pump
Disassembled Taco 007-F5 Pump

The pump housing is on the left. The impeller attached to the replaceable cartridge is in the center, and the motor proper is on the right.  The impeller wasn’t jammed, but I wanted to  know if it was working at all.  I cut the cord off a broken lamp and used it to wire up the pump.
Hotwired pump
Hotwired pump

I was careful not to touch the pump when plugged in, because you will notice there is no ground. The impeller worked fine.  Probably there was never anything wrong with the pump.  While I had it set up like this, I measured 0.7 Amps current when running, which is what it should be.  I then held on to the (plastic) impeller and turned it on.  When stalled, the motor draw rose to 1.25 Amps. I now had a way to tell if a motor was stalled or spinning!  The suspect zone was drawing .79 Amps, which probably means it was spinning, and the high temperature meant there was no water inside.
Around this time, Win called to ask me to go pick up firewood.  While waiting I explained all this to Cathy.  She has a PhD in Chemical Engineering, and has forgotten more about pipes and fluid flow that I will ever learn.  She says “are the pumps self-priming?”  Priming is the process of getting water into the pump so that they have something to pump.  A self priming pump will pump air well enough to pull water up the pipe from a lower reservoir.  A non-self-priming pump will not.  These pumps are not self-priming. They depend on something else to get started.  Cathy says “are the pumps below the reservoir level?”  No.  they are above the boiler.  Cathy says “I would design such a thing with the pumps below the reservoir level, so they prime automatically”.  Um, OK, but how does that help me?  Cathy says “Turn off the top valve, take off the top flange and pour water in the top.” Doh.
I didn’t quite do that, because I remembered the geyser I got taking off the air vent.  If I could let air out the top, water might flow in from below.  All I did was loosen the bolts on the top flange a little. After about 10 seconds, I started getting water drops out of the joint, so I tightened the bolts and turned on the pump. Success! After a few minutes, the temperature gauge started to rise.
So probably my problems were too much air in the system all along.
On the way to buy a new air vent, I stopped at Win’s house to check his air vent, but we couldn’t find it!  Either it is hidden away pretty well, which seems like a bad idea, or there isn’t one, which also seems like a bad plan.  We’re puzzled, but he has heat. And now, so do I!
UPDATE 12/15/2013
One heating zone still doesn’t work.  The temperature gauge near the pump rises to 100, and the nearby pipes are warm, but the pipes upstairs (this is a second floor zone) are cold.  I replaced the cartridge of the pump with the one I took apart the other day, and it spins, but there is no change.  The pump is drawing current consistent with spinning.  I loosen the top flange above the pump and water comes out.  These symptoms are consistent with the pump spinning, and having water, but there is no flow all the way around the loop.
I took a detour to the Taco website and looked at the pump performance curves for the 007-F5, which are at http://www.taco-hvac.com/uploads/FileLibrary/00_pumps_family_curves.pdf.  A pump has a certain ability to push water uphill. The weight of water above the pump more or less pushes back on the pressure generated by the pump.  This height of water is called the “head”.  A pump will pump more water against a lower head, and as the head is larger, the pump will deliver less and less water.  Above a certain head it won’t work at all.  According to the performance curves for my circulating pumps, their flow rate will drop to 0 at 10 feet of head.  From the pump location to the distribution manifold in the wall behind the closet in the upstairs bedroom is about 18 feet.  This pump cannot work if the pipe is not completely full of water.  If both the supply pipe to the upstairs and the return pipe coming back are full of water, then because water is incompressible, the suction of the water falling down the return pipe will balance the weight of water in the supply pipe.  If the pipe is full of air, as it likely is, then this pump is not powerful enough to lift water to the top.
The solution to this is to “purge” the air out of the pipes, by using some external source of pressure to push water into the supply end until all the air is pushed out of the return end.  For this to work, the return end must be opened up to atmosphere, otherwise there’s no place for the air to go.  (It will likely just get squeezed by the pressure, but there is no route for it to get to, for example, the air vent.  I think you need a pretty high flow rate to do this, because the return pipe is 3/4 inch, and without a high flow rate, the air bubbles will float up against the downwards flow of water.
Some systems have air vents at the high points.  Mine do not. This would help, because water would flow up both the supply and return pipes, lifted by the 10PSI system pressure.  Since it only takes 7.8 psi pressure to lift water 18 feet, this would completely fill the pipes.  Of course there would be a potentially leaky air vent inside the walls upstairs, to cause trouble in some future year. I don’t know if the lack of vents is sloppy installation or if one is supposed to use some other method of purging.
My system installation has no obvious (to me anyway) purge arrangements.  To purge, you shut off valves on the boiler, put a hose from a valve on the return side into a bucket of water, and turn on external water on the supply side.  When the host stops bubbling air, you are good to go.
In my system, makeup water comes from the house cold water pipes, through a backflow preventer and a pressure reduction valve to the hot water manifold.  The return pipes from the zones flow to the boiler return manifold and then to the boiler.  There is no master return shutoff, and no purge tap on the return maniforld. There is a drain tap on the boiler itself, and there is a tap between the boiler and a valve that can isolate the boiler from the hot water supply manifold.  The pressure regulator has a little lever on the top that according to its user manual will open the regulator and let more water through for purging.
I could close the valve to isolate the boiler from the supply manifold, but then the purge water has to run all the way through the boiler to get to the outlet hose.  I would lose all the hot water in the boiler.
But I have a missing pump!  Years ago, I borrowed the pump from the zone that heats the study, and never put it back.  I closed all the supply zone valves except the bad zone, and closed all the return valves except the bad zone and the study zone.  I closed the main boiler output valve.  At this point, the only path through the system was from the makeup water regulator, through the broken zone, to the return manifold, backwards into the study zone return pipe, through the cold side of the study zone mixing valve, and out the bottom flange of the not-present study pump.
I put a bucket under it and opened the bottom study zone pump valve.  Water came out, but after a few gallons, I only get a trickle.  I can hear hissing when I open the regulator toggle, but I suspect there is not enough flow to do effective purging.  The setup is complicated, so I am not completely sure.  In any case, this didn’t fix the not-working zone.
Next step: test the pressure regulator flow by closing all valves except makup water and the tap that is connected to the boiler outlet manifold.  That will let me see the flow supplied by the regulator.  I found an old backflow valve and regulator set on the floor. Evidently it was replaced at some point.  The old one had a pretty clogged looking input screen, so perhaps that is the trouble with the current one as well.  That wouldn’t affect normal operations because you don’t need makeup water unless there is a leak.
 

Bike Safey

I wrote this for the Wayland E News.  I’m putting a copy here as well.
I’ve been biking to work. In Cambridge. Not that often, because I am not one of these spandex bikers, but a middle aged, somewhat overweight, t-shirt biker.
I just wanted to mention a few things that would help me survive the week.
I am eagerly awaiting a paved Wayland Rail Trail from the town Library through to Weston, but in the meantime, I bike along route 20. The problem is that few roads in Wayland are bike friendly, but you can help!
(About that rail trail, please see Wayland Rail Trail and check out the Minuteman Bike Trail from Lexington to Alewife or the Charles River Bike Path )
For my fellow residents:

  • Take a look at the street in front of your house or other property.
  • Keep the shoulders clear of debris, sand, leaves, sticks, broken glass, etc.
  • Try and deal with the poison ivy that loves the edges of roads. I am so allergic to that stuff that I don’t dare ride right at the edge.
  • If you have a sidewalk, please keep it clear. In addition to the debris, it is hard to navigate around those mailboxes and trash cans.

For our public works folks:

  • When we do have sidewalks, they tend to be pretty awful, and unusable for bicycles. The paving isn’t up to street standards, and is broken by roots, holes. etc.
  • The sidewalks tend to fill up with leaves, fallen branches, and so forth, which make them unusable.
  • Guy wires cross from utility poles at just the right hight to clothesline a tall guy like me. Of course they are invisible at dusk!
  • Many road corners lack curb cuts, so you can’t actually get on or off the sidewalk anyway.

Without sidewalks, I have to ride in the street. That is fine, but…

  • The shoulders are, um, badly paved: potholes, jagged gaps in the top paving, bumpy drains
  • The shoulders collect sand, which is like ice for bicycles, you can’t steer on sand.
  • On Route 20, there is an unfortunate amount of broken glass.

Maybe we could street sweep more than once a year?
And that paving on Pelham Island Road is nasty, but that is a topic for a different letter.
For Drivers:
Most drivers are actually pretty awesome with bicyclists, Thank you! However:

  • Look at that right side mirror once in a while. When you are caught in traffic, I will be passing you at my astounding 12 miles an hour or whatever. I’ll be coming up on your right.
  • Don’t keep so far to the right that there isn’t room for me! The lanes are actually fairly wide and the shoulder is often very narrow.

For my part, I signal, I don’t run red lights, and I really try to watch where I am going and to be aware of my surroundings, but not every cyclist (especially the kids) will follow the rules. Treat them with suspicion and when possible, give extra space when passing a cyclist, just in case they have no idea you are there and swerve to miss a stick or pothole.
-Larry

BIOS vs GPT

This might be the 1000th blog posting on this general topic, but for some reason, the complexity of booting grows year over year, sort of like the tax code.
Back in 2009, Win and I built three low power servers, using Intel D945GCLF2 mini-ITX motherboards with Atom 330 processors.  We put mirrored 1.5 Terabyte drives in them, and 2 GB of ram, and they have performed very well as pretty low power home servers.  We ran the then-current Ubuntu, and only sporadically ran apt-get update and apt-get upgrade.
Fast forward to this summer.  We wanted to upgrade the OS’s, but they had gotten so far behind that apt-get update wouldn’t work.  It was clearly necessary to reinstall.  Now one of these machines is our compound mail server, and another runs mythtv and various other services.  The third one was pretty idle, just hosting about a terabyte of SiCortex archives.  In a previous blog post I wrote about the month elapsed time it took me to back up that machine.
This post is about the adventure of installing Ubuntu 12.04 LTS on it.  (LTS is long term support, so that in principle, we will not have to do this again until 2017.  I hope so!)
Previously, SMART tools were telling us that the 2009 era desktop 1.5T drives were going bad, so I bought a couple of 3T WD Red NAS drives, like the ones in our Drobo 5N.  Alex (my 14 year old) and I took apart the machine and replaced the drives, with no problem.
I followed directions from the web on how to download an ISO and burn it to a USB drive using MacOS tools.   This is pretty straightforward, but not obvious.  First you have to convert the iso to a dmg, then use dd to copy it to the raw device:

hdiutil convert -format UDRW -o ubuntu-12.04.3-server-amd64.img ubuntu-12.04.3-server-amd64.iso
# Use diskutil list, then plug in a blank USB key >the image size, run diskutil list again to find the drive device.  (In my case /dev/disk2)
sudo dd if=ubuntu-12.04.3-server-amd64.img.dmg of=/dev/disk2 bs=1m
# notice the .dmg extension that MacOS insists on adding
diskutil eject /dev/disk2 (or whatever)

Now in my basement, the two servers I have are plugged into a USB/VGA monitor and keyboard switch, and it is fairly slow to react when the video signal comes and goes.  In fact it is so slow that you miss the opportunity to type “F2” to enter the BIOS to set the boot order.  So I had to plug in the monitor and keyboard directly, in order to enable USB booting.  At least it HAS USB booting, because these machines do not have optical drives, since they have only two SATA ports.
Anyway, I was able to boot the Ubuntu installer.  Now even at this late date, it is not really well supported to install onto a software RAID environment.  It works, but you have to read web pages full of advice, and run the partitioner in manual mode.
May I take a moment to rant?  PLEASE DATE YOUR WEB PAGES.  It is astonishing how many sources of potentially valuable information fail to mention the date or versions of software they apply to.
I found various pieces of advice, plus my recollection of how I did this in 2009, and configured root, swap, and /data as software RAID 1 (mirrored disks).  Ubuntu ran the installer, and… would not reboot.  “No bootable drives found”.
During the install, there was an anomaly, in that attempts to set the “bootable” flag on the root filesystem partitions failed, and when I tried it using parted running in rescue mode, it would set the bootable flag, but clear the “physical volume for RAID” flag.
I tried 12.04.  I tried 13.04.  I tried 13.04 in single drive (no RAID).  These did not work. The single drive attempt taught me that the problem wasn’t the RAID configuration at all.
During this process, I began to learn about GPT, or guid partition tables.
Disks larger than 2T can’t work with MBR (master boot record) style partition tables, because their integers are too small.  Instead, there is a new GPT (guid partition table) scheme, that uses 64 bit numbers.
Modern computers also have something called UEFI instead of BIOS, and UEFI knows about GPT partition tables.
The Ubuntu installer knows that large disks must use GPT, and does so
Grub2 knows this is a problem, and requires the existence of a small partition flagged bios_grub, as a place to stash its code, since GPT does not have the blank space after the sector 0 boot code that exists in the MBR world (which grub uses to stash code).
So Ubuntu creates the GPT, the automatic partitioning step creates the correct mini-partition for grub to use, and it seems to realize that grub should be installed on both drives when using an MD filesystem for root. (it used the command line grub-install /dev/sda /dev/sdb) Evidently the grub install puts a first stage loader in sector 0, and the second stage loader in the bios_grub partition.
Many web pages say you have to set the “bootable” flag on the MD root, but parted will not let you do this,because in GPT, setting a “bootable” flag is forbidden by the spec.  Not clear it would work anyway because when you set it, the “physical volume for raid” flag is turned off.
The 2009 Atom motherboards do not have a UEFI compatible BIOS, and are expecting an MBR. When they don’t find one, they give up.  If they would just load the code in sector 0 and jump to it it would work. I considered doing a bios update, but it wasn’t clear the 2010 release is different in this respect.
So the trick is to use FDISK to <create an MBR> with a null partition.  This is just enough to get past the Atom BIOS’ squeamishness and have it execute the grub loader, which then works fine using the GPT.  I got this final trick from http://mjg59.dreamwidth.org/8035.html whose final text is

boot off a live CD and run fdisk against the boot disk. It’ll give a bunch of scary warnings. Ignore them. Hit “a”, then “1”, then “w” to write it to disk. Things ought to work then.

The sequence of steps that worked is:

Run the installer
Choose manual disk partitioning
Choose "automatically partition" /dev/sda
This will create a 1 MB bios_grub partition and a 2GB swap, and make the rest rootDelete the root partition
Create a 100 GB partition from the beginning of the free space
Mark it "physical volume for RAID" with a comment that it is for root 
Use the rest of the free space (2.9T) to make a partition, mark it physical volume for raid.  Comment that it is for /data
Change the type of the swap partition to "physical volume for raid"
Repeat the above steps for /dev/sdb
Run "configure software RAID"
Create MD volume, using RAID 1 (mirrored)
Select 2 drives, with 0 spares
Choose the two swap partitions
Mark the resulting MD partition as swap 
Create MD volume, RAID 1, 2, and 0
Select the two 100 GB partitions
Mark them for use as EXT4, to be mounted on /
Create MD volume, RAID 1, 2, and 0
Select the two 2.9T partitions
Mark them for use as EXT4, to be mounted on /data 
(I considered BTRFS, but the most recent comments I could find still seem to regard it as flakey)
Save and finish installing Ubuntu
Pretend to be surprised when it won't boot.  "No bootable disks found"
Reboot from the installer USB, choose Rescue Mode
Step through it. Do not mount any file systems, ask for a shell in the installer environment.
When you get a prompt,
fdisk /dev/sda
a
1
w
Then
fdisk /dev/sdb
a
1
w
^d and reboot. Done
Now I have a working Ubuntu 12.04 server with mirrored 3T drives.