It has finally gotten cold here.  Right now is it about 17F outside.  Previously we had been getting by with just the heating zones for the kitchen/family room and the master bedroom turned on.
A few days ago, the boys had trouble getting to sleep while we were watching TV, because the noise from the set was keeping them up.  Alex closed the door.  The next morning, I noticed it was 55F in their room.  Well, I reasoned, the heating zone up there is not turned on, and with the door shut, warm air from the rest of the house can’t get in so easily.  I turned on the heat.  The next night Alex happened to close the door again, and in the morning it was 52F.  That isn’t so good.
Friday we had the neighbors over for dinner so I turned on the dining room heat.  A couple hours later I went to check on it and it wasn’t any warmer.

Heating System
Heating System Central in the basement

This is our heating system.  This is a gas fired hot water system.  The “boiler” is the green box on the lower left.  It heats water to 160F or so.  From there, there are 9 heating zones. The horizontal pipe manifold in the front is the the return path to the boiler. The vertical pipes with yellow shutoffs representing the returns for each zone.  The supply manifold is behind, along with the pumps and so forth. One zone heats water in the blue tank for domestic hot water faucets and showers.  The other zones have circulating pumps that feed tubing that zigzags under the floors .  This is called radiant heating.
Radiant Zone Manifold
Hot water is routed through many plastic tubes that warm the floor from below.

Each zone typically has a manifold like this one that routes hot water through synthetic rubber tubes that are stapled to the undersize of the floors, and insulated below that to direct their heat upwards.  This lets you walk around on warm floors and actually get by with colder air temperatures.  Our oldest daughter was in the habit of leaving the next day’s clothes on the floor covered with a blanket, so they would be prewarmed in the morning. Notice that one tube is turned off. That one runs underneath the kitchen pantry, which we try to keep colder.
In the main system photo, on the left, you can see electronics boxes on the wall. Here’s a closeup.
Zone control box
Zone control box

Each zone has a thermostat, which comes into one of these boxes.  This is a three channel box, with three 24 volt thermostats coming in on brown wires at the top, and red wiring for three 120 volt zone circulator pumps at the bottom.  The box also signals the main boiler that heat is being called by at least one zone. Each zone has a plug in relay, one of which I have unplugged.
The circulator pumps look like this
Circulator Pump
Taco 007-F5 Circulator Pump

So there is a central gas water heater, which feeds a number of zones. Each zone has a water circulation pump, controlled by a thermostat. The pump feeds hot water through rubber tubes on the underside of the floors.
Individual zones have failed before.  I have fixed them by replacing the circulator pump.  You can get these anywhere.
New circulator pump
New circulator pump

The hardest part about replacing these is the electrical wiring, which is hardwired by wirenuts in the green box attached to the pump.  First, turn off the power.  I did this by physically pulling the relay for the appropriate zone.  Then I measure the pump current using a clamp on ammeter. Then I measure the voltage.  Only then do I unscrew the wirenuts protecting the wires, and without touching the bare wires, touch the end to ground. Then brush the wire with the back of your hand only.  If the wire is live, the electricity will contract your arm muscles, pulling your hand away.  If you can’t think of at least four ways to make sure the wires are not live, hire someone to do this for you.  Really. There are old electricians, and there are bold electricians. There are no old, bold electricians.  I am an old electrician.
Our system has shutoff valves immediately on both sides of the pump.  By turning those off, you can swap out the pump without draining all the water out of the system.  As you can see in the picture, the pump is held in place by flanges at the inlet (bottom) and outlet (top). Each flange has two stainless steel bolts, so they won’t rust.  In a burst of cleverness or good design, the nuts on these bolts are 11/16 and the bolts themselves are 5/8, so you can take them apart with only one set of wrenches.  Here’s the pump I removed.
Old pump
Old Pump

Note the corrosion inside the pump.  I put the new pump in place and turned this zone back on, and now the dining room was getting heat.  While I was down there, I took a look at this thing.
Air removal valve
Hy-Vent air removal valve

This is an air removal valve. It is installed on top of the boiler, along with a pressure relief valve.  On some intuition, I lifted the pressure relief valve toggle, and air came out, followed by water. That is not good.  The water for a heating system like this comes from town water, which has dissolved gas in it. Typically this will be air, although in the Marcellus Shale areas it can be natuural gas (in those areas, you can set your sink on fire).  Air is bad for forced hot water systems.  it corrodes the inside of the pipes, and water pumps won’t pump air, usually.  If the radiant tubes get full of air, they will not be heating. By the way, these pipes are so rusty because some years ago the boiler was overheating to the point that the relief valve was opening, getting water everywhere. This was because the temperature sensor had come unstuck from the pipe it was measuring. Fixed by a clever plumber with a stainless pipe clamp.  As collateral damage from rapid cycling, I had to get a new gas valve too.  Separate story.
After waiting a few few minutes, I tried the relief valve again and got more air.  This meant that the air removal valve wasn’t working, and probably some of my zones weren’t working because of air-bound pumps or bubbles in the pipes.  You might be wondering how the valve knows to let out air, but not water.  Inside the cylinder is a float.  When there is water inside, the float rises and closes the output port.  When there is air inside, the float falls, opening the outlet port and letting out the air.  It is pretty simple.  I called a plumber friend to see if he could fix this and he said “if you can replace a zone pump, you can replace this valve too.” Basically, you turn off the system, close all the valves, to minimize the amount of water that will come out, depressurize the system, and work fast.  A new valve was $13 at Home Depot. The fact they had 10 in stock suggests they do go bad. Unfortunatly I failed to depressurize the system as well as I thought, and I got a 3 foot high gusher of 130F water.  Be careful!  Heating systems run at around 10 psi. The pressure comes partly from town water pressure through a pressure regulator, and partly from the expansion of hot water.  There is an expansion tank to reduce that effect.
The next day, I tried the pressure relief valve again and got water immediately. Probably this means the new valve is working.
Temperature Gauges
Temperature Gauges

Each zone has a temperature gauge.  You can see that the two on the right are low, and the two on the left in this picture are not.  The right hand zone had the pump I replaced. The next one was not turned on.  The temperature gauges are there because you don’t want to run 160F water through these radiant tubes. The floors will get too hot and the tubes won’t last very long. Instead, each zone has a check valve and a mixing valve.
Check Valve
Check Valve

The check valve keeps the loop from flowing backwards, or generally keeps it from circulating by gravity.  Cold water is slightly denser than hot water, so the water on the colder side of the loop will fall, pulling hot water around the loop even without the pump running. The spring in the check valve is enough to stop gravity circulation.
Mixing Valve
Mixing Valve

The mixing valve has a green adjusting knob. This valve mixes hot water from the boiler with cooler water from the return leg of the zone, and serves to adjust the temperature of the water in each zone.  Some water recirculates, with some hot water added.
When I turned on the zone second from the right, it did not work.  The temperature gauge stayed put at 80F, (conducted heat through the copper pipes).  I used my ammeter to confirm the pump was drawing power.  I turned off the valves for all the other zones, so that this one would have more water.  Didn’t work.
Measuring pump current
Measuring pump current

There are three reasons why a hot water zone might not work:  the pump is not spinning, the pump is trying to pump air, or the pipes are clogged.  I had just replaced a pump to fix a zone, but was there a second bad pump? Or something else?
I have an intra-red non-contact thermometer, and I used it to measure the pump housing temperatures.  The working pumps were all around 125F, the non working pump was around 175F. That might mean that it was stalled, and not spinning, or that it was pumping air, and not being cooled by the water.  I had one more spare pump, but I was getting suspicious.
I got to wondering if the pump I removed was really broken. I knew that these Taco 007-F5 pumps have a replaceable cartridge inside, but since the cartridge costs almost as much as a new pump I had never bothered with it.  I decided to take apart the pump I removed to see what it looks like.
Disassembled pump
Disassembled Taco 007-F5 Pump

The pump housing is on the left. The impeller attached to the replaceable cartridge is in the center, and the motor proper is on the right.  The impeller wasn’t jammed, but I wanted to  know if it was working at all.  I cut the cord off a broken lamp and used it to wire up the pump.
Hotwired pump
Hotwired pump

I was careful not to touch the pump when plugged in, because you will notice there is no ground. The impeller worked fine.  Probably there was never anything wrong with the pump.  While I had it set up like this, I measured 0.7 Amps current when running, which is what it should be.  I then held on to the (plastic) impeller and turned it on.  When stalled, the motor draw rose to 1.25 Amps. I now had a way to tell if a motor was stalled or spinning!  The suspect zone was drawing .79 Amps, which probably means it was spinning, and the high temperature meant there was no water inside.
Around this time, Win called to ask me to go pick up firewood.  While waiting I explained all this to Cathy.  She has a PhD in Chemical Engineering, and has forgotten more about pipes and fluid flow that I will ever learn.  She says “are the pumps self-priming?”  Priming is the process of getting water into the pump so that they have something to pump.  A self priming pump will pump air well enough to pull water up the pipe from a lower reservoir.  A non-self-priming pump will not.  These pumps are not self-priming. They depend on something else to get started.  Cathy says “are the pumps below the reservoir level?”  No.  they are above the boiler.  Cathy says “I would design such a thing with the pumps below the reservoir level, so they prime automatically”.  Um, OK, but how does that help me?  Cathy says “Turn off the top valve, take off the top flange and pour water in the top.” Doh.
I didn’t quite do that, because I remembered the geyser I got taking off the air vent.  If I could let air out the top, water might flow in from below.  All I did was loosen the bolts on the top flange a little. After about 10 seconds, I started getting water drops out of the joint, so I tightened the bolts and turned on the pump. Success! After a few minutes, the temperature gauge started to rise.
So probably my problems were too much air in the system all along.
On the way to buy a new air vent, I stopped at Win’s house to check his air vent, but we couldn’t find it!  Either it is hidden away pretty well, which seems like a bad idea, or there isn’t one, which also seems like a bad plan.  We’re puzzled, but he has heat. And now, so do I!
UPDATE 12/15/2013
One heating zone still doesn’t work.  The temperature gauge near the pump rises to 100, and the nearby pipes are warm, but the pipes upstairs (this is a second floor zone) are cold.  I replaced the cartridge of the pump with the one I took apart the other day, and it spins, but there is no change.  The pump is drawing current consistent with spinning.  I loosen the top flange above the pump and water comes out.  These symptoms are consistent with the pump spinning, and having water, but there is no flow all the way around the loop.
I took a detour to the Taco website and looked at the pump performance curves for the 007-F5, which are at http://www.taco-hvac.com/uploads/FileLibrary/00_pumps_family_curves.pdf.  A pump has a certain ability to push water uphill. The weight of water above the pump more or less pushes back on the pressure generated by the pump.  This height of water is called the “head”.  A pump will pump more water against a lower head, and as the head is larger, the pump will deliver less and less water.  Above a certain head it won’t work at all.  According to the performance curves for my circulating pumps, their flow rate will drop to 0 at 10 feet of head.  From the pump location to the distribution manifold in the wall behind the closet in the upstairs bedroom is about 18 feet.  This pump cannot work if the pipe is not completely full of water.  If both the supply pipe to the upstairs and the return pipe coming back are full of water, then because water is incompressible, the suction of the water falling down the return pipe will balance the weight of water in the supply pipe.  If the pipe is full of air, as it likely is, then this pump is not powerful enough to lift water to the top.
The solution to this is to “purge” the air out of the pipes, by using some external source of pressure to push water into the supply end until all the air is pushed out of the return end.  For this to work, the return end must be opened up to atmosphere, otherwise there’s no place for the air to go.  (It will likely just get squeezed by the pressure, but there is no route for it to get to, for example, the air vent.  I think you need a pretty high flow rate to do this, because the return pipe is 3/4 inch, and without a high flow rate, the air bubbles will float up against the downwards flow of water.
Some systems have air vents at the high points.  Mine do not. This would help, because water would flow up both the supply and return pipes, lifted by the 10PSI system pressure.  Since it only takes 7.8 psi pressure to lift water 18 feet, this would completely fill the pipes.  Of course there would be a potentially leaky air vent inside the walls upstairs, to cause trouble in some future year. I don’t know if the lack of vents is sloppy installation or if one is supposed to use some other method of purging.
My system installation has no obvious (to me anyway) purge arrangements.  To purge, you shut off valves on the boiler, put a hose from a valve on the return side into a bucket of water, and turn on external water on the supply side.  When the host stops bubbling air, you are good to go.
In my system, makeup water comes from the house cold water pipes, through a backflow preventer and a pressure reduction valve to the hot water manifold.  The return pipes from the zones flow to the boiler return manifold and then to the boiler.  There is no master return shutoff, and no purge tap on the return maniforld. There is a drain tap on the boiler itself, and there is a tap between the boiler and a valve that can isolate the boiler from the hot water supply manifold.  The pressure regulator has a little lever on the top that according to its user manual will open the regulator and let more water through for purging.
I could close the valve to isolate the boiler from the supply manifold, but then the purge water has to run all the way through the boiler to get to the outlet hose.  I would lose all the hot water in the boiler.
But I have a missing pump!  Years ago, I borrowed the pump from the zone that heats the study, and never put it back.  I closed all the supply zone valves except the bad zone, and closed all the return valves except the bad zone and the study zone.  I closed the main boiler output valve.  At this point, the only path through the system was from the makeup water regulator, through the broken zone, to the return manifold, backwards into the study zone return pipe, through the cold side of the study zone mixing valve, and out the bottom flange of the not-present study pump.
I put a bucket under it and opened the bottom study zone pump valve.  Water came out, but after a few gallons, I only get a trickle.  I can hear hissing when I open the regulator toggle, but I suspect there is not enough flow to do effective purging.  The setup is complicated, so I am not completely sure.  In any case, this didn’t fix the not-working zone.
Next step: test the pressure regulator flow by closing all valves except makup water and the tap that is connected to the boiler outlet manifold.  That will let me see the flow supplied by the regulator.  I found an old backflow valve and regulator set on the floor. Evidently it was replaced at some point.  The old one had a pretty clogged looking input screen, so perhaps that is the trouble with the current one as well.  That wouldn’t affect normal operations because you don’t need makeup water unless there is a leak.