Email Disaster Recovery and Travel adventures

Cathy is off to China for a few weeks. She wanted email access, but not with her usual laptop.
She uses Windows Vista on a plasticy HP laptop from, well, the Vista era.  It is quite heavy, and these days quite flaky.  It has a tendency to shut down, although not for any obvious reason, other maybe than age, and being Vista running on a plasticy HP laptop.
I set up the iPad, but Cathy wanted a more familiar experience, and needed IE in order to talk to a secure webmail site, so we dusted off an Asus EEE netbook running Windows XP.
I spent a few hours trying to clear off several years off accumulated crapware such as three different search toolbars attached to Internet Explorer, then gave up and re-installed XP from the recovery partition.  123 Windows Updates later, it seemed fine, but still wouldn’t talk to the webmail site.  It turns out that Asus thoughtfully installed the open source local proxy server Privoxy, with no way to uninstall it.  If you run the Privoxy uninstall, it leaves you with no web access at all.  I finally found Interwebs advice to also uninstall the Asus parental controls software, and that fixed it.
Next, I installed Thunderbird, and set it up to work with Cathy’s account on the family compound IMAP server.  I wanted it to work offline, in case of spotty WiFi access in China, but after setting that up, so I “unsubscribed” to most of the IMAP folders and let it download.  Now Cathy’s inbox has 34,000 messages in it, and I got to thinking “what about privacy?”  After all, governments, especially the United States, claim the right to search all electronic devices at the border, and it is also commonly understood that any electronic device you bring to China can be pwned before you come back.
Then I found a setting that tells Thunderbird to download only the last so many days for offline use.  Great!  But it had already downloaded all 6 years of back traffic.  Adjacent, there is a setting for “delete mail more than 20 days (or whatever) old.”
You know what happens next!  I turned that on, and Thunderbird started deleting all Cathy’s mail, both locally and on the server.  Now there is (farther down the page), fine print that explains this will happen, but I didn’t read it.
Parenthetically, this is an awful design.  It really looks like a control associated with how much mail to keep for offline use, but it is not.  It is a dangerous, unguarded, unconfirmed command that does irreversible damage.
I thought this was taking too long, but by the time I figured it out, it was way too late.
So, how to recover?
I have been keeping parallel feeds from Cathy’s email, but only since March or so, since I’ve been experimenting with various spam supression schemes.
I had made a copy of Cathy’s .maildir on the server, but it was from 2011.
But wait! Cathy’s laptop was configured for offline use, and had been turned off.  Yes!  I opened the lid and turned off WiFi as quickly as possible, before it had a chance to sync.  (Actually, the HP has a mechanical switch to turn off WiFi, but I didn’t know that.)  I then changed the username/password on her laptop Thunderbird to stop further syncing.
Next, since the horse was well out of the barn, I made a snapshot of the server .maildir, and of the HP laptop’s Thunderbird profile directories. Now, whatever I did, right or wrong, I could do again.
Time for research!
What I wanted to do seemed clear:  restore the off-line saved copies of the mail from the HP laptop to the IMAP server.  This is not a well-travelled path, but there is some online advice:
http://www.fahim-kawsar.net/blog/2011/01/09/gmail-disaster-recovery-syncing-mail-app-to-gmail-imap/
https://support.mozillamessaging.com/en-US/kb/profiles
The general idea is:

  1. Disconnect from the network
  2. Make copies of everything
  3. While running in offline mode, copy messages from the cached IMAP folders to “Local” folders
  4. Reconnect to the network and sync with the server. This will destroy the cached IMAP folders, but not the new Local copies
  5. Copy from the Local folders back to the IMAP server folders

Seems simple, but in my case, there were any number of issues:

  • Not all server folders were “subscribed” by Thunderbird, and I didn’t know which ones were
  • The deletion was interrupted at some point
  • I didn’t want duplicated messages after recovery
  • INBOX was 10.3 GB (!)
  • The Thunderbird profile altogether was 23 GB (!)
  • The HP laptop was flakey
  • Cathy’s about to leave town, and needs last minute access to working email

One thing at a time.
Tools
I found out about  “MozBackup” and used it to create a backup copy of the HP laptop’s profile directory.
MozBackup
MozBackup creates a zip file of the contents of a Thunderbird profile directory, and can restore them to a different profile on a different computer, making configuration changes as appropriate. This is much better than hand editting the various Thunderbird configuration files.
Hardware problems
As I mentioned, the HP laptop is sort of flakey.  I succeeded in copying the Thunderbird profile directory, but 23 GB worth of copying takes a long time on a single 5400 rpm laptop disk.  I tried copying to a Mybook NAS device, but it was even slower.  What eventually worked, not well, but adequately, was copying to a 250GB USB drive.
I decided to leave the HP out of it, and to do the recovery on the netbook, the only other Windows box available.  I was able to create a second profile on the netbook, and restore the saved profile to it, slowly, but I realized Cathy would leave town before I finished all the steps, taking the netbook with her.  Back to the HP.
First I tried just copying the IMAPMail subfolder files of mbox files and msf files to LocalFolders. This seemed to work, but Thunderbird got very confused about it.  It said there were 114000 messages in Inbox, rather than 34000.  This shortcut is a dead end.
I created a new profile on the HP, and restored the backup using MozBackup (which took 2 hours), and started it in offline mode.  I then tried to “select-all” in Inbox to copy them to a local folder.  Um.  No.  I couldn’t even get control back.  Thunderbird really cannot select 34000 messages and do anything.
Because I was uncertain about the state of the data, I restored the backup again (another 2 hours).
This time, I decided to break up Inbox into year folders, each holding about 7000 messages.  The first one worked, but then the HP did an undexpected shutdown during the second, and when it came back, Inbox was empty! The Inbox mbox file had been deleted.
I did another restore, and managed to create backup files for 2012 and 2011 messages, before it crashed again. (And Inbox was gone AGAIN)
The technique seemed likely to eventually work, but it would drive me crazy.  Or crazier.
I was now accumulating saved Local Folder files representing 3 specific years of Inbox.  I still had to finish the rest, deal with Sent Mail, and audit about 50 other subfolders to see if they needed to be recovered.
I wasn’t too worried about all the archived subfolders, since they hadn’t changed in ages and were well represented by my 2011 copy of Cathy’s server .maildir
Digression
What about server backups?  Embarassing story here!  Back in 2009, Win and I built some nice mini-ATX atom based servers with dual 1.5T disks run in mirrored mode for home servers.  Win’s machine runs the IMAP, and mine mostly has data storage.  Each machine has the mirrored disks for reliabiltiy and a 1.5T USB drive for backup.  The backups are irregularly kept up to date, and in the IMAP machines case, not recently.
About 6 months ago, I got a family pack of CrashPlan for cloud backup, and I use it for my Macbook and for my (non IMAP) server, but we had never gotten around to setting up CrashPlan for either Cathy’s laptop or the IMAP server.
A few months ago, we got a Drobo 5N, and set it up with 3 3T disks, for 6T usable storage, but we haven’t gotten it working for backup either.  (I am writing another post about that.)
So, no useful server backups for Cathy’s mail.
Well now what?
I have a nice Macbook Pro, unfortunately, the 500 GB SSD has 470 GB of data, not enough for one copy of Cathy’s cached mail, let alone two.  I thought about freeing up space, and copied a 160 GB Aperture photo library to two other systems, but it made me nervous to delete it from the Macbook.
I then tried using Mac Thunderbird to set up a profile on that 250 GB external USB drive, but it didn’t work because the FAT filesystem couldn’t handle Mac Thunderbird’s need for fancy filesystem features like ACLs, but this triggered an idea!
First, I was nervous about using Mac Thunderbird to work on backup data from a PC. I know that Thunderbird profile directories are supposed to be cross-platform, but the config files like profile.ini and prefs.js are littered with PC pathnames.
Second, the USB drive is slow, even if it worked.
Up until recently, I’ve been using a 500 GB external Firewire drive for TimeMachine backups of the Macbook.  It still was full of Time Machine data, but I’ve switched to using a 1T partition on the Drobo for TimeMachine.  I also have the CrashPlan backup.  So I reformatted the Firewire Drive to HFS, and plugged it in as extra storage.
Also on the Macbook, is VMWare Fusion, and one of my VMs is a 25 GB instance running XP Pro.
I realized I should be able to move the VM to the Firewire drive, and expand its storage by another 50 GB or so to have room to work on the 23 GB Thunderbird data.
To the Bat Cave!
It turns out to be straightforward to copy a VMWare image to another place, and then run the copy.  Rather than expand the 25GB primary disk, I just added a second virtual drive and used XP Disk management to format it as drive E.  I also used VMWare sharing to share access to the underlying Mac filesystem on the Firewire drive.

  1. Copy VMWare image of XP to the Firewire drive
  2. Copy MozBackup save file of the cached IMAP data and the various Local Files folders to the drive
  3. Create second disk image for XP
  4. Run XP under VMWare Fusion on the Macbook, using the Firewire drive for backing store
  5. Install Thunderbird and MozBackup
  6. Use Mozbackup to restore Cathy’s cached local copies of her mail from the flakey HP laptop
  7. Copy the Local Files mailbox files for 2013, 2012, and 2011 into place.
  8. Use XP Thunderbird running under VMWare to copy the rest of the cached IMAP data into Local Folders.
  9. By hand, compare message counts of all 50 or so other IMAP folders in the cached copy with those still on the server, and determine they were still correct.
  10. Go online, letting Thunderbird sync with the server, deleting all the locally cached IMAP data.
  11. Create IMAP folders for 2007 through 2013, plus Sent Mail and copy the roughly 40000 emails back to the server.

Notes
During all of this, new mail continued to arrive into the IMAP server, and be accessible by the instance of Thunderbird on the netbook.
A copy of Cloudmark Desktop One was active running on the Macbook using Mac Thunderbird to do spam processing of arriving email in Cathy’s IMAP account.
My psyche is scarred, but I did manage to recover from a monstrous mistake.
Lessons

  • RAID IS NOT BACKUP

The IMAP server was reliable, but it didn’t have backups that were useful for recovery.

  • Don’t think you understand what a complex email client is going to do

Don’t experiment with the only copy of something!  I should have made a copy of the IMAP .maildir in a new account, and then futzed with the netbook thunderbird to get the offline use storage the way I wanted.

  • Quantity has a quality all its own.

This quote is usually about massive armies, but in this case, the very large email (23 GB) just made the simplest operations slow, and some things (like selecting ALL in a folder with 34000 messages, impossible.)  I had to go through a lot of extra work because various machines didn’t have enough free storage, and had other headaches because the MTBF of the HP laptop was less than the time to complete tasks.
-Larry

What I do

I used Splasho’s “Up-Goer Five Text Editor.”  to write what I do, using only the most common 1000 words in English
In my work I tell computers what to do. I write orders for computers that tell them first to do this,and then to do that, and then to do this again.
Sometimes the orders tell the computer to listen for other orders from people. Then the orders tell the computer how to do what the people want, and then the orders tell the computer to show the people what the answer is.
I used to build computers. I would take one part, and another part, and many more parts, and put them together in just the right way so the computer would work right. Computers are all the same, they listen for an order, then do what it says, then listen for another order. We use them because they do this thing very very very very fast.

Equal Protection of the Law

I’ve been casting about for a way to follow up on my outrage of the government’s treatment of Aaron Swartz.
I wonder if the government’s conduct represents a violation of the equal protection clause of the constitution.
The 14th amendment says

…nor shall any State deprive any person of life, liberty, or property, without due process of law; nor deny to any person within its jurisdiction the equal protection of the laws.

Evidently this doesn’t apply to the federal government as written, but in Bolling v. Sharpe in 1954, the Supreme court got to the same point via the Due Process clause of the 5th amendment.
I think all governments, state, federal, and local, are bound to provide equal protection.
In the Swartz case, we have the following mess

  • Congress writes vague laws
  • Congress fails to update those laws as technology and society evolve
  • Prosecutors use their discretion to decide who to charge
  • Prosecutors use pre-trial plea bargaining to avoid the scrutiny of the courts

It would be nice to have a case before the Supreme Court, leading to a clear ruling that equal protection applies to the actions of prosecutors. I suspect that would also give us proportional responses to crimes, although I am not sure about that.
In the medium term, Congress needs to act.  I’d suggest a law repealing all laws more than 20 years old.  Sunset provisions need to be in all laws. The ones that make ongoing sense can be reauthorized, but it will take a new vote every time.  (Maybe laws forbidding action by the government should be allowed to stand indefinitely, while laws forbidding action by the people will have limited terms.)
In the short term, we need action by the executive branch, to provide equal protection, control of pre-trial behavior of prosecutors, and accountability of both prosecutors and law enforcement.
 

AT&T Hell

Summary – AT&T customer service gives you bad information, tries to fix it and can’t, then lies about how it is “impossible”.
Update summary – Twitter works!  AT&T twitter team seems to have fixed the remaining problem.
“We don’t care, we don’t have to.” – Lily Tomlin
When I worked for IBM one summer, I wore a tie every day to see if I could do it.
When I drove an RX-7 in Palo Alto, I obeyed all the speed limits, to see if I could do it.
Last month I gave up my iPhone, to see if I could do it.
My daughter wanted an iPhone, but she’s in the middle of a two year contract on T-Mobile with a Palm Pixi.  My iPhone 4S is in the middle of a two year contract with AT&T that started October 2011.  It had the grandfathered unlimited data plan, and would be up for upgrade eligibility in May 2013.
On December 26, I called AT&T to see if I could port my number out and get a new number assigned to the iPhone, so I could let my daughter use it, while I would keep the T-Mobile phone, but with my number.  My number started out life a long time ago as a Verizon landline, with the number sequential to our home phone, so I am attached to it.  It is also on all my business cards and in countless contact lists.
AT&T said “sure”, when you port the number out, we’ll assign a new number to the iPhone and the contract will remain unchanged.
Life was good!  The daughter is happy, and I have a phone that is, um, interesting.  I also have an iPad, so don’t shed any tears about that!
A week or so later, we notice that the bill is $400ish.  There is an early termination charge on there!  You can’t actually figure out what the charge is from the online presentation.  You have to hunt up the pdf and look at the image of the printed bill.  This is a phone company, they know how to print phone bills, not how to build websites.
On the phone with customer service.  “When you ported out the number, that cancels your contract, and you get an early termination fee. Then you added a new line with new contract dates.”  I explained my call on the 26th, and the agent said, oh, well I can waive the early termination fee and make the contract be as it was. The only thing I can’t do is preserve the unlimited data plan.  So now the phone is on the 3GB plan.   I thought about balking, that unlimited plan made me feel like an old-time iPhone user, more privileged than the unwashed masses, but really, my usage is about 250 MB per month.  The iPad has a bigger screen.  So I let it slide.
A few days later, a website check showed the fees gone.  I noticed that the upgrade availabiltiy wording was different for this phone than for the other iPhone line, which also started October 2011, but decided to wait to see if other changes would catch up before calling.
A few days later, no change.  Called and learned that the second agent had waived the fees, but not fixed the contract dates.  I was assured that all would be fixed, and notes put in the account.
A few days later, no change to the upgrade language.  On calling, I was told that the contract would expire October 2013, as expected, but the upgrade eligibility date was July 2014.  What does that even mean?  After the contract is over, I can just create a new line, with a new contract and phone, and port the number!  It makes no sense to have an upgrade eligibility after the contract expiration.  Anyway, this is just stupid.  I explained that I had been told “the contract would be as it was” but the agent said there was just no way to change that in his system, the upgrade eligibility is tied to the phone number, not to the contract.
[By the way, this is also a lie, because, for example, if you are being stalked, you can request a new number and get it without any such collateral damage.]
I asked for a supervisor, who said
This should never have been allowed in the first place.  You can’t port out a number and keep the contract. It is our number.  The agents who tried to “fix” it for you went way outside our policies and made it worse.  What they should have done to correct their original mistake was to port your number back in, not to try and fix the contract. It can’t be fixed, it is impossible to change an upgrade eligibility date. It is tied to the phone number.
The supervisor said there were no higher supervisors to talk to, and no physical mail address to send a complaint to.
Well.  This supervisor was certainly polite, but either was really unable to fix the problems that AT&T created, or unwilling to do so.
At the moment, I have a nice iPhone, with a pleased daughter, but I am not pleased.  I made a perfectly sensible request.  I was told “Yes, of course you can do that” and now the account is scrambled beyond belief.
Recapping

  •  iPhone 4S, 14 months into a 24 month contract.
  • I ask to port out my number, and get a new number assigned to the phone,without contract changes.  I’m not paying them any less, I am not getting a new phone, just changing a few bits in a database somewhere about what is the number!
  • AT&T says “yes”
  • AT&T charges an early termination fee, an activation fee, cancels my unlimited data plan, restarts the 2 year contract, and resets the upgrade eligibility data.  I am not even angry about the activation fee, they deserve some fee for the work.
  • I complain.  AT&T waives the early termination fee, promises to fix the contract, but doesn’t
  • I complain.  AT&T promises to fix the contract, but only fixes the contract termination dates, the upgrade date is now 9 months after the contract expires.
  • I complain.  AT&T says “impossible to fix”
  • AT&T supervisor says “impossible to fix, and there is noone higher than me to ask”
The only thing that an upgrade date after contract expiration might mean is that AT&T would refuse to unlock the phone until it is 2 3/4 years old.  That would piss me off, but I don’t even want to ask them right now.

And by the way, the iPhone battery doesn’t work as well as it used to, and that 18 month upgrade was starting to look pretty attractive!  Instead, I will likely have to pay Apple $79 to fix it.  At least that is cheaper than the $99 Applecare I forgot to get, if nothing else goes wrong with the phone.
Now I am not a phone company marketing person, but I think I understand the essential economics of subsidized phones.  AT&T gives a substantial discount on the phone in trade for a contract commitment.  In fact, this is still a worse deal for the customer than buying an unlocked phone on a carrier with cheaper plans, like Virgin or T-Mobile, but AT&T doesn’t discount the monthly charges if you bring your own device.  That is just another way to screw the consumer.  So with AT&T, you may as well get the subsidy if you don’t mind sticking around for two years.  And they really make their money back so quickly that they let you upgrade (and restart the two year clock) after 18 months.
This is a simple deal – AT&T discounts the phone, I promise to keep paying their (high) monthly bills for two years.  This has nothing to do with the phone number!  Changing the number has utterly no effect on the money flows.
What about that number?  AT&T says it is their number, they can attach whatever they want to it.  But that is not true.  I had the number with Verizon. I ported the number to AT&T, I ported it out.  The FCC has “local number portability”.  The numbers are managed by CLECs (I think that is the term of art for phone companies) but they really can’t be taken away from users except for some arcane technical reasons.
What has happened here?  It cannot be “impossible” to fix these sorts of problems.  There may be software limitations, but those are fixable.  Or they could merely write a note to themselves saying “Yes, the system says this contract runs until July 2014, but when the customer asks, in May 2013, for an upgrade, just waive the fees.  And when the customer cancels the contract in October 2013, waive any cancellation fee.”
Instead, they’ve spent a lot of money on customer service phone calls, which are not cheap. They’ve enraged a long-standing customer who has alternatives. They’ve provided more information to the entire internet about just how bad their service and systems are.  There is no good result for AT&T here. They’ve not gained any income. They haven’t kept control of their precious number. They may well lose me as a customer come October.  (That Nexus 4 on T-Mobile is looking pretty good, or a nice unlocked iPhone 5S or whatever.) And they are defending positions and policies that make no sense competitively or economically.
I’m not sure of the next step for me.  Probably I will tweet the URL of this blog entry to @ATTCustomerCare.  At this point, AT&T can fix the problems, or they can provide me a source of continuing amusement.  There’s a rumor that sometimes people get results by writing the CEO.  At a minimum that will cost them even more money to deal with my letter.
UPDATE – I tweeted this URL to @ATTCustomerCare and they actually answered, got me on the telephone, and fixed this, well enough.  Which is to say they can’t fix it in the database, but they’ve added a special note telling other folks to honor an upgrade request on or after the correct date.  Works for me.  (1/16/2013)
You can sort of understand how enterprise software can become unwieldy, to the point where it seems easier to correct software problems and poor specifications by adding layer upon layer of special fixes and exceptions and end-runs, but it is not good for customers or efficiency to do it that way.
 
 
 
 

The Fault in our Stars

The Fault in our Stars is the new book by John Green.  You will likely find it in the children’s section of your local library, as it is usually filed under Youth.
I think you should go read it.  Buy a copy, or check it out of the library, or borrow one from a friend, or see if your middle school english teacher has a copy.
Actually I think you should go read all of John Green’s books.  Looking for Alaska, Paper Towns, An Abundance of Katherines, Will Grayson, Will Grayson, and The Fault in our Stars.  I like them all.
The Fault in our Stars is a love story, a search for meaning. Its about the survivors, at least temporarily, of cancer, and about those whose survival is even more temporary. Is the purpose of our existance to do things? Or is the purpose of our consciousness to pay attention to what is around us?  All of our times are limited, but there are still an infinite number of moments in each day, to be used as best we can.
PS.  If you don’t believe me, (and who would?) go read some of the reviews on Amazon.
 

Comcast DigitalNow

Comcast has not achieved even my low expectations.
We get “Limited Basic” cable, which means we get the local broadcast channels plus a couple of things like New England Cable News. After the conversion to broadcast Digital, Comcast was sending these channels in analog (SD) format, but also sending QAM unencrypted digital versions of them.  The local broadcast HD signals were also being delivered.  Since we have a modern TV with NTSC, QAM, and ATSC tuners, life was fairly pleasant.  Our old Replay TV works with the analog channels, and we can watch the digital or HD digital versions live.
Now, Comcast is completing their “digital conversion” and taking away the analog signals.  This is fine.  They are providing “Digital Transport Adapters” so you can convert the digital versions (SD) to analog on channel 3, for analog sets or our old ReplayTV.
However, they are (evidently) also taking away, or encrypting, the local broadcast HD channels!  The only way to continue to receive them is to get a Digital Set Top Box, for which they will charge an extra $10/month.
Hello!  The HD signals are broadcast and free.  The real effect of this will be for us to drop Comcast altogether in favor of over the air HD.  Our ReplayTV will stop working, but I have a nice HDHomeRun digital tuner, and I can assemble a (free) MythTV box to record things.  I had this working already, but converted that box for the kids to do gaming.  I can make another one.  And it will work for the internet streaming TV as well: Netflix, crackle, hulu, youtube, etc.
Comcast’s only value to me was convenience, and they are making it rather less convenient and rather more expensive.
The right thing to do is just to transcode the local broadcast signals from ATSC to QAM, and to leave them alone. No DTA, no extra box, no extra remote.
 
 
 

iovec to messagelet ring

I’ve been working on fos, the Factored Operating System.  fos is a project at MIT CSAIL. It uses messages to communicate between applications and services, in the same way a standard operating system uses system calls and function calls.
One thing I’ve been doing is adding iovec support to the message API.  This is equivalent to the difference between write(2), which writes a single buffer of data to a file, and writev(2), which uses an iovec data structure to gather pieces of the data into a single write to the file.  An iovec is an array of structures, each of which contains a pointer and length.
One of the types of high performance message transports in fos is a ring of cache-line sized messagelets.  Each messagelet has an 8 byte header and 56 bytes of data.  To send a message, one waits until the next messagelet in the ring is free (as shown by flags in the header), then you write an 8 byte message length field, and then copy the rest of the message into the messagelet.  If it doesn’t all fit, then you mark the first messagelet as filled, and wait for the next one to be free, and continue writing the message.
This is a slow design, because the sender must copy a longer message in 56 byte chunks, but it is also a rather fast method, because the receiver can be draining the head of the message while the sender is writing the tail.  The idea comes from a communications method in the Barrelfish research operating system.
With iovec, the sender has a bigger problem. In order to know the total length of the message, you have to add up the lengths in all the iovec entries. Then, you have to step through the iovec, and copy each one into a sequential series of messagelets.  An iovec entry may end in the middle of a messagelet.
How would you write this?  I’ve just started thinking about it, and will post my code here when I figure it out.
UPDATE
Here’s my version.
 

/* iovec_to_messagelet_ring.c
 * L. Stewart
 * 2011-12-29
 */
#include <stddef.h>
#include <stdint.h>
#include <sys/uio.h>
#include <string.h>
/* Messagelet functions */
typedef void CHANNEL; /* placeholder */
#define ML_SIZE 56
/* Returns a pointer to the data area of a messagelet.
 * The header is -8 bytes offset
 */
void *getfreemessagelet(CHANNEL *ch);
/* sets ready flag in messagelet header, turning it over to the receiver */
void postmessagelet(CHANNEL *ch, void *m);
void send(CHANNEL *ch, struct iovec *in_iov, int in_iovcnt)
{
  total_size = 0;
  int iov_index;  /* current iovec entry */
  void *m = NULL; /* current messagelet */
  void *mp; /* current pointer into messagelet */
  size_t ml_len; /* space left in current messagelet */
  size_t copy_length; /* amount to copy this time around the loop */
  struct iovec iov; /* working iovec entry */
  /* calculate total size of message by adding the lengths of the iovec entries */
  for (iov_index = 0; iov_index < in_iovcnt; iov_index += 1)
    total_size += in_iov[iov_index].iov_len;
  if (total_size == 0) return; /* nothing to do */
  m = getfreemessagelet(ch);
  *((uint64_t *) m) = total_size; /* set length of message */
  mp = (void *) ((uintptr_t) m + sizeof(uint64_t));
  ml_len = ML_SIZE - sizeof(uint64_t);
  iov_index = 0;
  iov.iov_len = 0;
  while (total_size > 0) {
    if (ml_len == 0) {
      m = mp = getfreemessagelet(ch);
      ml_len = ML_SIZE;
    }
    if (iov.iov_len == 0) {
      iov = in_iov[iov_index];
      iov_index += 1;
    }
    copy_length = (iov.iov_len < ml_len) ? iov.iov_len : ml_len;
    memcpy(mp, iov.iov_base, copy_length);
    ml_len -= copy_length;
    iov.iov_len -= copy_length;
    mp = (void *) ((uintptr_t) mp + copy_length);
    iov.iov_base = (void *) ((uintptr_t) iov.iov_base + copy_length);
    if (ml_len == 0) postmessagelet(ch, m);
    total_size -= copy_length;
  }
}

 

Tetrahedron

At work in the aftermath of the Halloween snow storm, one of my colleagues brought in his son because school was closed.  I joined a math discussion between the boy and my boss Steve Heller on the subject of ways to think about products of the form (x + a) (x – a). Afterwards, Steve happened to mention that it was possible to inscribe a tetrahedron inside a cube, and a cube inside a dodecahedron.
The dodecahedron sounds difficult, but I decided to build a tetrahedron inside a cube.  The tetrahedron is cut out of a manilla folder, and the cube is made from a sheet protector.

Tetrahedron inscribed in a cube
Tetrahedron inscribed in a cube

Black Friday Report: Target

Abstract: Mixed
Wednesday evening around 9:15PM I drove my daughter to the Target in Framingham to look for boots.  They were closed.  This was surprising because their newspaper ad said “Open until 11,” and their phone message said “Open until 11.”  In fact, Cathy had spoken to the store operator earlier in the day just to make sure and was told “yes, we are open until 11.”
Thursday night, my daughter and I went with my neighbor to the same store to look at Black Friday doorbusters.  The newspaper ad said they would open at midnight.  They were closed.  The line wrapped halfway around the building.  Eventually some workers came down the line handing out maps.  They said that Massachusetts law wouldn’t let them open at midnight, so they would open at 1AM.  By this time it was around 33 degrees, and still 45 minutes to wait.  We went home.
I looked into this question of law, and found an article dated about 10 days ago which said that Massachusetts Blue Laws forbid employees from working before midnight on Thanksgiving, in order to let them have a holiday.  So evidently, staff could report at midnight, but it took them an hour to unlock the doors.
I think this is one of those situations in which Target, at least this store, doesn’t get it.  They seem honestly puzzled that the public might expect them to be open when their ads say, and expect that staff give correct information about hours, or that anyone might not be grateful for the chance to stand around in freezing weather for an hour in the middle of the night in order to come into their store.
So why do I say “mixed”?  Because I was gullible enough to go back at 6:30AM Friday to the same store that locked me out twice in two days.  And you know? They did a really good job.  All the workers were there. Everyone seemed to know where everything in the store was located. They had adequate stock. They were friendly.
I should add, however, that the store was recently remodelled, turning a once open layout with long sight-lines into the sort of place where you can’t see where you are trying to get to.  The interior is now about halfway between reasonable and Walmart.
 

Driving Practice

My daughter now has a learner’s permit.  For her first outing, she went with my wife to the local elementary school parking lot on a weekend.  Evidently the only casualties were two traffic cones and a portapotty.
Actually they made up the part about the portapotty, but it was a good story.
Later, my wife was talking to a friend about this, and the friend suggested that after parking lot proficiency is attained, the next level is driving in the town cemetary. Nice empty winding roads.  The friend finishes with “and the best thing is you can’t kill anyone.”