AT&T Hell

Summary – AT&T customer service gives you bad information, tries to fix it and can’t, then lies about how it is “impossible”.
Update summary – Twitter works!  AT&T twitter team seems to have fixed the remaining problem.
“We don’t care, we don’t have to.” – Lily Tomlin
When I worked for IBM one summer, I wore a tie every day to see if I could do it.
When I drove an RX-7 in Palo Alto, I obeyed all the speed limits, to see if I could do it.
Last month I gave up my iPhone, to see if I could do it.
My daughter wanted an iPhone, but she’s in the middle of a two year contract on T-Mobile with a Palm Pixi.  My iPhone 4S is in the middle of a two year contract with AT&T that started October 2011.  It had the grandfathered unlimited data plan, and would be up for upgrade eligibility in May 2013.
On December 26, I called AT&T to see if I could port my number out and get a new number assigned to the iPhone, so I could let my daughter use it, while I would keep the T-Mobile phone, but with my number.  My number started out life a long time ago as a Verizon landline, with the number sequential to our home phone, so I am attached to it.  It is also on all my business cards and in countless contact lists.
AT&T said “sure”, when you port the number out, we’ll assign a new number to the iPhone and the contract will remain unchanged.
Life was good!  The daughter is happy, and I have a phone that is, um, interesting.  I also have an iPad, so don’t shed any tears about that!
A week or so later, we notice that the bill is $400ish.  There is an early termination charge on there!  You can’t actually figure out what the charge is from the online presentation.  You have to hunt up the pdf and look at the image of the printed bill.  This is a phone company, they know how to print phone bills, not how to build websites.
On the phone with customer service.  “When you ported out the number, that cancels your contract, and you get an early termination fee. Then you added a new line with new contract dates.”  I explained my call on the 26th, and the agent said, oh, well I can waive the early termination fee and make the contract be as it was. The only thing I can’t do is preserve the unlimited data plan.  So now the phone is on the 3GB plan.   I thought about balking, that unlimited plan made me feel like an old-time iPhone user, more privileged than the unwashed masses, but really, my usage is about 250 MB per month.  The iPad has a bigger screen.  So I let it slide.
A few days later, a website check showed the fees gone.  I noticed that the upgrade availabiltiy wording was different for this phone than for the other iPhone line, which also started October 2011, but decided to wait to see if other changes would catch up before calling.
A few days later, no change.  Called and learned that the second agent had waived the fees, but not fixed the contract dates.  I was assured that all would be fixed, and notes put in the account.
A few days later, no change to the upgrade language.  On calling, I was told that the contract would expire October 2013, as expected, but the upgrade eligibility date was July 2014.  What does that even mean?  After the contract is over, I can just create a new line, with a new contract and phone, and port the number!  It makes no sense to have an upgrade eligibility after the contract expiration.  Anyway, this is just stupid.  I explained that I had been told “the contract would be as it was” but the agent said there was just no way to change that in his system, the upgrade eligibility is tied to the phone number, not to the contract.
[By the way, this is also a lie, because, for example, if you are being stalked, you can request a new number and get it without any such collateral damage.]
I asked for a supervisor, who said
This should never have been allowed in the first place.  You can’t port out a number and keep the contract. It is our number.  The agents who tried to “fix” it for you went way outside our policies and made it worse.  What they should have done to correct their original mistake was to port your number back in, not to try and fix the contract. It can’t be fixed, it is impossible to change an upgrade eligibility date. It is tied to the phone number.
The supervisor said there were no higher supervisors to talk to, and no physical mail address to send a complaint to.
Well.  This supervisor was certainly polite, but either was really unable to fix the problems that AT&T created, or unwilling to do so.
At the moment, I have a nice iPhone, with a pleased daughter, but I am not pleased.  I made a perfectly sensible request.  I was told “Yes, of course you can do that” and now the account is scrambled beyond belief.

  •  iPhone 4S, 14 months into a 24 month contract.
  • I ask to port out my number, and get a new number assigned to the phone,without contract changes.  I’m not paying them any less, I am not getting a new phone, just changing a few bits in a database somewhere about what is the number!
  • AT&T says “yes”
  • AT&T charges an early termination fee, an activation fee, cancels my unlimited data plan, restarts the 2 year contract, and resets the upgrade eligibility data.  I am not even angry about the activation fee, they deserve some fee for the work.
  • I complain.  AT&T waives the early termination fee, promises to fix the contract, but doesn’t
  • I complain.  AT&T promises to fix the contract, but only fixes the contract termination dates, the upgrade date is now 9 months after the contract expires.
  • I complain.  AT&T says “impossible to fix”
  • AT&T supervisor says “impossible to fix, and there is noone higher than me to ask”
The only thing that an upgrade date after contract expiration might mean is that AT&T would refuse to unlock the phone until it is 2 3/4 years old.  That would piss me off, but I don’t even want to ask them right now.

And by the way, the iPhone battery doesn’t work as well as it used to, and that 18 month upgrade was starting to look pretty attractive!  Instead, I will likely have to pay Apple $79 to fix it.  At least that is cheaper than the $99 Applecare I forgot to get, if nothing else goes wrong with the phone.
Now I am not a phone company marketing person, but I think I understand the essential economics of subsidized phones.  AT&T gives a substantial discount on the phone in trade for a contract commitment.  In fact, this is still a worse deal for the customer than buying an unlocked phone on a carrier with cheaper plans, like Virgin or T-Mobile, but AT&T doesn’t discount the monthly charges if you bring your own device.  That is just another way to screw the consumer.  So with AT&T, you may as well get the subsidy if you don’t mind sticking around for two years.  And they really make their money back so quickly that they let you upgrade (and restart the two year clock) after 18 months.
This is a simple deal – AT&T discounts the phone, I promise to keep paying their (high) monthly bills for two years.  This has nothing to do with the phone number!  Changing the number has utterly no effect on the money flows.
What about that number?  AT&T says it is their number, they can attach whatever they want to it.  But that is not true.  I had the number with Verizon. I ported the number to AT&T, I ported it out.  The FCC has “local number portability”.  The numbers are managed by CLECs (I think that is the term of art for phone companies) but they really can’t be taken away from users except for some arcane technical reasons.
What has happened here?  It cannot be “impossible” to fix these sorts of problems.  There may be software limitations, but those are fixable.  Or they could merely write a note to themselves saying “Yes, the system says this contract runs until July 2014, but when the customer asks, in May 2013, for an upgrade, just waive the fees.  And when the customer cancels the contract in October 2013, waive any cancellation fee.”
Instead, they’ve spent a lot of money on customer service phone calls, which are not cheap. They’ve enraged a long-standing customer who has alternatives. They’ve provided more information to the entire internet about just how bad their service and systems are.  There is no good result for AT&T here. They’ve not gained any income. They haven’t kept control of their precious number. They may well lose me as a customer come October.  (That Nexus 4 on T-Mobile is looking pretty good, or a nice unlocked iPhone 5S or whatever.) And they are defending positions and policies that make no sense competitively or economically.
I’m not sure of the next step for me.  Probably I will tweet the URL of this blog entry to @ATTCustomerCare.  At this point, AT&T can fix the problems, or they can provide me a source of continuing amusement.  There’s a rumor that sometimes people get results by writing the CEO.  At a minimum that will cost them even more money to deal with my letter.
UPDATE – I tweeted this URL to @ATTCustomerCare and they actually answered, got me on the telephone, and fixed this, well enough.  Which is to say they can’t fix it in the database, but they’ve added a special note telling other folks to honor an upgrade request on or after the correct date.  Works for me.  (1/16/2013)
You can sort of understand how enterprise software can become unwieldy, to the point where it seems easier to correct software problems and poor specifications by adding layer upon layer of special fixes and exceptions and end-runs, but it is not good for customers or efficiency to do it that way.

The Fault in our Stars

The Fault in our Stars is the new book by John Green.  You will likely find it in the children’s section of your local library, as it is usually filed under Youth.
I think you should go read it.  Buy a copy, or check it out of the library, or borrow one from a friend, or see if your middle school english teacher has a copy.
Actually I think you should go read all of John Green’s books.  Looking for Alaska, Paper Towns, An Abundance of Katherines, Will Grayson, Will Grayson, and The Fault in our Stars.  I like them all.
The Fault in our Stars is a love story, a search for meaning. Its about the survivors, at least temporarily, of cancer, and about those whose survival is even more temporary. Is the purpose of our existance to do things? Or is the purpose of our consciousness to pay attention to what is around us?  All of our times are limited, but there are still an infinite number of moments in each day, to be used as best we can.
PS.  If you don’t believe me, (and who would?) go read some of the reviews on Amazon.

Comcast DigitalNow

Comcast has not achieved even my low expectations.
We get “Limited Basic” cable, which means we get the local broadcast channels plus a couple of things like New England Cable News. After the conversion to broadcast Digital, Comcast was sending these channels in analog (SD) format, but also sending QAM unencrypted digital versions of them.  The local broadcast HD signals were also being delivered.  Since we have a modern TV with NTSC, QAM, and ATSC tuners, life was fairly pleasant.  Our old Replay TV works with the analog channels, and we can watch the digital or HD digital versions live.
Now, Comcast is completing their “digital conversion” and taking away the analog signals.  This is fine.  They are providing “Digital Transport Adapters” so you can convert the digital versions (SD) to analog on channel 3, for analog sets or our old ReplayTV.
However, they are (evidently) also taking away, or encrypting, the local broadcast HD channels!  The only way to continue to receive them is to get a Digital Set Top Box, for which they will charge an extra $10/month.
Hello!  The HD signals are broadcast and free.  The real effect of this will be for us to drop Comcast altogether in favor of over the air HD.  Our ReplayTV will stop working, but I have a nice HDHomeRun digital tuner, and I can assemble a (free) MythTV box to record things.  I had this working already, but converted that box for the kids to do gaming.  I can make another one.  And it will work for the internet streaming TV as well: Netflix, crackle, hulu, youtube, etc.
Comcast’s only value to me was convenience, and they are making it rather less convenient and rather more expensive.
The right thing to do is just to transcode the local broadcast signals from ATSC to QAM, and to leave them alone. No DTA, no extra box, no extra remote.

iovec to messagelet ring

I’ve been working on fos, the Factored Operating System.  fos is a project at MIT CSAIL. It uses messages to communicate between applications and services, in the same way a standard operating system uses system calls and function calls.
One thing I’ve been doing is adding iovec support to the message API.  This is equivalent to the difference between write(2), which writes a single buffer of data to a file, and writev(2), which uses an iovec data structure to gather pieces of the data into a single write to the file.  An iovec is an array of structures, each of which contains a pointer and length.
One of the types of high performance message transports in fos is a ring of cache-line sized messagelets.  Each messagelet has an 8 byte header and 56 bytes of data.  To send a message, one waits until the next messagelet in the ring is free (as shown by flags in the header), then you write an 8 byte message length field, and then copy the rest of the message into the messagelet.  If it doesn’t all fit, then you mark the first messagelet as filled, and wait for the next one to be free, and continue writing the message.
This is a slow design, because the sender must copy a longer message in 56 byte chunks, but it is also a rather fast method, because the receiver can be draining the head of the message while the sender is writing the tail.  The idea comes from a communications method in the Barrelfish research operating system.
With iovec, the sender has a bigger problem. In order to know the total length of the message, you have to add up the lengths in all the iovec entries. Then, you have to step through the iovec, and copy each one into a sequential series of messagelets.  An iovec entry may end in the middle of a messagelet.
How would you write this?  I’ve just started thinking about it, and will post my code here when I figure it out.
Here’s my version.

/* iovec_to_messagelet_ring.c
 * L. Stewart
 * 2011-12-29
#include <stddef.h>
#include <stdint.h>
#include <sys/uio.h>
#include <string.h>
/* Messagelet functions */
typedef void CHANNEL; /* placeholder */
#define ML_SIZE 56
/* Returns a pointer to the data area of a messagelet.
 * The header is -8 bytes offset
void *getfreemessagelet(CHANNEL *ch);
/* sets ready flag in messagelet header, turning it over to the receiver */
void postmessagelet(CHANNEL *ch, void *m);
void send(CHANNEL *ch, struct iovec *in_iov, int in_iovcnt)
  total_size = 0;
  int iov_index;  /* current iovec entry */
  void *m = NULL; /* current messagelet */
  void *mp; /* current pointer into messagelet */
  size_t ml_len; /* space left in current messagelet */
  size_t copy_length; /* amount to copy this time around the loop */
  struct iovec iov; /* working iovec entry */
  /* calculate total size of message by adding the lengths of the iovec entries */
  for (iov_index = 0; iov_index < in_iovcnt; iov_index += 1)
    total_size += in_iov[iov_index].iov_len;
  if (total_size == 0) return; /* nothing to do */
  m = getfreemessagelet(ch);
  *((uint64_t *) m) = total_size; /* set length of message */
  mp = (void *) ((uintptr_t) m + sizeof(uint64_t));
  ml_len = ML_SIZE - sizeof(uint64_t);
  iov_index = 0;
  iov.iov_len = 0;
  while (total_size > 0) {
    if (ml_len == 0) {
      m = mp = getfreemessagelet(ch);
      ml_len = ML_SIZE;
    if (iov.iov_len == 0) {
      iov = in_iov[iov_index];
      iov_index += 1;
    copy_length = (iov.iov_len < ml_len) ? iov.iov_len : ml_len;
    memcpy(mp, iov.iov_base, copy_length);
    ml_len -= copy_length;
    iov.iov_len -= copy_length;
    mp = (void *) ((uintptr_t) mp + copy_length);
    iov.iov_base = (void *) ((uintptr_t) iov.iov_base + copy_length);
    if (ml_len == 0) postmessagelet(ch, m);
    total_size -= copy_length;



At work in the aftermath of the Halloween snow storm, one of my colleagues brought in his son because school was closed.  I joined a math discussion between the boy and my boss Steve Heller on the subject of ways to think about products of the form (x + a) (x – a). Afterwards, Steve happened to mention that it was possible to inscribe a tetrahedron inside a cube, and a cube inside a dodecahedron.
The dodecahedron sounds difficult, but I decided to build a tetrahedron inside a cube.  The tetrahedron is cut out of a manilla folder, and the cube is made from a sheet protector.

Tetrahedron inscribed in a cube
Tetrahedron inscribed in a cube

Black Friday Report: Target

Abstract: Mixed
Wednesday evening around 9:15PM I drove my daughter to the Target in Framingham to look for boots.  They were closed.  This was surprising because their newspaper ad said “Open until 11,” and their phone message said “Open until 11.”  In fact, Cathy had spoken to the store operator earlier in the day just to make sure and was told “yes, we are open until 11.”
Thursday night, my daughter and I went with my neighbor to the same store to look at Black Friday doorbusters.  The newspaper ad said they would open at midnight.  They were closed.  The line wrapped halfway around the building.  Eventually some workers came down the line handing out maps.  They said that Massachusetts law wouldn’t let them open at midnight, so they would open at 1AM.  By this time it was around 33 degrees, and still 45 minutes to wait.  We went home.
I looked into this question of law, and found an article dated about 10 days ago which said that Massachusetts Blue Laws forbid employees from working before midnight on Thanksgiving, in order to let them have a holiday.  So evidently, staff could report at midnight, but it took them an hour to unlock the doors.
I think this is one of those situations in which Target, at least this store, doesn’t get it.  They seem honestly puzzled that the public might expect them to be open when their ads say, and expect that staff give correct information about hours, or that anyone might not be grateful for the chance to stand around in freezing weather for an hour in the middle of the night in order to come into their store.
So why do I say “mixed”?  Because I was gullible enough to go back at 6:30AM Friday to the same store that locked me out twice in two days.  And you know? They did a really good job.  All the workers were there. Everyone seemed to know where everything in the store was located. They had adequate stock. They were friendly.
I should add, however, that the store was recently remodelled, turning a once open layout with long sight-lines into the sort of place where you can’t see where you are trying to get to.  The interior is now about halfway between reasonable and Walmart.

Driving Practice

My daughter now has a learner’s permit.  For her first outing, she went with my wife to the local elementary school parking lot on a weekend.  Evidently the only casualties were two traffic cones and a portapotty.
Actually they made up the part about the portapotty, but it was a good story.
Later, my wife was talking to a friend about this, and the friend suggested that after parking lot proficiency is attained, the next level is driving in the town cemetary. Nice empty winding roads.  The friend finishes with “and the best thing is you can’t kill anyone.”


This is a bit out of order, reporting on our trip to Volcano National Park on the Big Island in Hawaii.  This was before Hurricane Irene, but I am just getting to it now.
We flew from Maui to Kona on Pacific Wings airline, which the kids now call “Best Airplane Ride Ever”. We flew on a 9 passenger Cessna 208B (a Cessna Caravan single engine turboprop).  The pilot was also the counter agent, baggage handler, and ground crew.  Thinking about it afterwards, it is no wonder that it was a little tricky getting a reservation through Travelocity, our party accounted for 6 of the 9 seats!
We rented a minivan and drove around to the Hilo area, to a rental in Hawaiian Beaches.  This is pretty much at the end of the road in nowhere.  No ATT cell coverage, and no Verizon either. We looked at local attractions for a day and then went to Volcano National Park to visit the Kilauea volcano, which has been erupting, more or less, since 1983.
The current lava flows are from the Pu’u O’o crater, which is in the east fissure zone, and more or less inaccessible without a several hour hike.  Not clear it is a good idea to go there anyway, since the sulphur dioxide concentrations can be lethal within a mile or so if you get downwind.
The main caldera of Kilauea is about 400 feet deep and 2.5 miles across.  Towards the southwest side, there is a smaller crater called Halema’uma’u, which is about 250 feet deep.  Inside Halema’uma’u there is a vent about 500 feet across, and inside that, there is a lava lake whose height fluctuates with volcanic activity. The day we were there the lake level was about 550 feet below the top of the vent.
Overlooking Halema’uma’u there is the Volcano Observatory, and the Jaggar Museum, from the patio of which you can watch events.  Here is a photo I took around 7:15 PM.

Halema'uma'u at dusk

Earlier in the day we drove down the chain of craters road until the end:
Road Closed due to Lava

Across the street there is a sign that is worth reading:
Warning sign

And a short walk to the cliff is  worthwhile as well:
Lava Bridge

Our trip to the volcano was delayed by a couple of hours because the car wouldn’t start.  The dashboard merely said, helpfully, “badkey”.  The remote controls still worked, but the car wouldn’t recognize the RFID chip or whatever is inside these newfangled Chrysler keys.  Alamo rentals was full of warnings not to get the key wet, but we hadn’t.  Alamo sent a local towing company to our out of the way house with a new minivan and took away the old one.  Probably a replacement key would have been sufficient, but we had rented in Kona which is three hours away, rather than from the Hilo office.  Thank you Alamo for taking  care of us, but I guess I am old fashioned.  I’ve never had a mechanical key break and I don’t understand the attraction of the electronic version.


We’ve recently returned from a family vacation to Hawaii.  Cathy and I went to Maui and the Big Island for our honeymoon, and we returned to those islands with the kids, 20 years later.
On August 14, we drove up to the top of Haleakala (“House of the Sun”). This is the 10,000 foot volcano on Maui, and the sunrise is reputed to be spectacular.  We got everyone up at 2:30 AM and got to the top at 5AM, in time to get a parking space in preperation for the sunrise at 6AM.
It is cold up there, even in August

Bundled up on Haleakala

Before sunrise, the sky is quite interesting:
Sky above Haleakala

Then, just as the sun rises, the domes of nearby Science City light up, but not yet the ground.
Science City on Haleaka, first rays of the sun

And here is the sunrise itself:
Sunrise on Haleakala

And for those who keep track of such things, there is no cell coverage by ATT at the top of Haleakala, but Verizon works just fine.

Networking during Hurricane Irene

Hello from within our modest tropical storm Irene.  Here it is just windy and rainy.  The power went off about 4 hours ago, right in the middle of the coffee maker cycle.  I dumped the rest of the water in the reservoir into a pan and brought it a boil on the gas stove, then poured it into the basket. Worked fine.  Without power you have to start the gas stove with a match, and the exhaust fan doesn’t work, but that is OK for minor cooking.
After about 15 minutes, the little UPS on the ethernet switches and FIOS router stopped working.  The FIOS optical network terminal kept running on its own battery.
I suspect this little neighborhood in Wayland is pretty low on NStar’s list of power problems, so I wheeled out the generator to the garage entrance. This is a 6KW electric start machine.  We haven’t needed it for several years, since a round of tree limb triming in town dramatically improved power reliability.  Unfortunately, the generator battery is ten years old,  and hasn’t worked for the last five.  I’ve never been successful in pull starting it unless it was already working, so I gave it a jump start from the DR field mower.
The generator plugs into the house via a 30′ pice of 10-4 cable with 30 Amp connectors.  The house connector in turn is wired to a manual transfer switch that moves 10 circuits from line to generator.  When the house was built, we thought pretty carefully about what to power:
* boiler controls, to permit hot water to work
* refrigerator
* freezer
* kitchen outlets
* outlet near the TV in the family room
* outlets in master bedroom
* outlets near the computer equipment in the basement
* outlet in the study (for my computer!)
* … and I don’t remember where the other two circuits are.  Note to self: find out.
Plus there is 300 feet of 12 gauge extension cord running across the lawn to the neighbor’s house to power their freezer.
This all made sense, but things change, and the house wiring hasn’t.  The FIOS ONT is in the utility room, and there is no generator outlet in there.  So now there is a 25 foot extension cord connecting it to the server outlets.  Similarly, we moved the freezer so now there is another extension cord connecting it to a powered outlet.
The little UPS is a problem. When the power came back on, the UPS hasn’t switched back. It just beeps fitfully. Note to self: a cheap UPS from Best Buy is probably worth every penny!
My son Alex was so offended by the lack of power for the family iMac that he’s moved it to the floor of the MBR and figured out which outlet is live. He also moved the Time Capsule that supports upstairs WiFi, and then I had to show him how to interpret the patch panel diagram to get it plugged into a live network port.  Cathy doesn’t approve of kids using the internet during a power outage,  but I figure I should reward initiative.
The home server had been up for 242 days, but it hasn’t restarted.  I will have to go troubleshoot.  The only difficulty with this is that we don’t have DNS service for the inside machines.  For talking to the world, we can just switch to Google’s DNS at, which is easy to remember.
The roof is leaking, but it is the place that just happens to drip into the kitchen sink.  Is that good planning or just luck?
I don’t know whether to expect FIOS to stay up long term or not.  The fiber goes to the local CO, which has lots of batteries, but I don’t know if there are active components between here and there, and I don’t know what is upstream from the local CO.
The home server came up fine, and if you wait long enough, ssh to it works.  The problem is that its upstream DNS is the server in Win’s basement, which is down right now.
One of the smoke detectors is unhappy about the lack of AC power.  It probably needs a new battery, but it is the one about 14 feet off the ground in the loft.  I can reach it with the extension ladder, but that is out in the rain behind the house.  Ah well.
So far the chicken coop hasn’t blown over, and the run is still standing.  The chickens, sensibly, are staying inside.