About 5 years ago, as we were building GHDonline, we made a few design decisions that have not stood up well to the test of time. These mistakes were due in part to our relative inexperience with Python and in part to the newness of Django (we were using a fork of the trunk because 0.96 didn’t have some of the features we needed and 1.0 was under development).

A few of these decisions were driven solely by premature optimization, which is rarely a good idea. I’ll illustrate with a small example.

The “Problem”

    in which we thought we knew better than the Django core developers

Django’s ORM contains an EmailField type, which for the purposes of this story is used in User.email (django.contrib.auth). The EmailField is essentially a varchar, but it does some validation (e.g. is it of the form name@domain.ext ?) on save. The default length for this field is 75 characters, which we decided was far too short, since the standard put forth in RFC 2821 specifies:

4.5.3.1 Size limits and minimums

There are several objects that have required minimum/maximum sizes.
Every implementation MUST be able to receive objects of at least
these sizes. Objects larger than these sizes SHOULD be avoided when
possible. However, some Internet mail constructs such as encoded
X.400 addresses [16] will often require larger objects: clients MAY
attempt to transmit these, but MUST be prepared for a server to
reject them if they cannot be handled by it. To the maximum extent
possible, implementation techniques which impose no limits on the
length of these objects should be used.

local-part
The maximum total length of a user name or other local-part is 64
characters.

domain
The maximum total length of a domain name or number is 255
characters.

Django provides a means to increase the max-length of EmailField:

address = models.EmailField(max_length=320)

We wanted to extend the length of EmailField, and we also wanted to allow for multiple email addresses per user, so we ended up creating our own Email model and stopped populating the User.email field. Though it occasionally required some code gymnastics, this approach worked pretty well for us.

The Attempted Upgrade

    which should have been much simpler than we made it

Fast-forward 5 years to this week, and we’re looking to upgrade our old fork of django-registration, a very commonly-used package that handles user signups on the web and account activations via email. A new contender, django-allauth, has emerged; it handles the same functionality as django-registration, provides more out of the box (like basic templates), and integrates social authentication.

Email addresses in django-allauth are stored both in django.contrib.auth.models.User.email (as before, a single email address per user) and their own Email model, which allows for multiple addresses per user and looks a heck of a lot like our own.

Unlike us, they’re actually using the User.email field, so we had to find a way to extend that field to 320 characters. Turns out, there’s a hack that does this via monkey-patching.

Alternatively, we could have built our own User model, but that would have required an upgrade to Django 1.5, which introduces too much risk for this project (we have a pretty complex system; we’re 3-4 months out from upgrading).

This solution seemed to work, so I kept developing… until I realized that django-allauth’s Email model also uses the default length of 75 for its EmailField, as well. The earlier hack felt slightly wrong, but it seemed to be a pretty common approach, so we went with it. However, when we hit this second case, I realized that we might be doing something wrong.

The Realization

    that the Django core devs knew what they were doing

Then it hit me: perhaps our assumption that the standard should be followed was wrong (or was at least a premature optimization). Was 75 characters actually enough?

We have five years of user data, so I created a histogram of lengths of email addresses from our current userbase. The wonderful collections.Counter helped immensely:

>>> from collections import Counter
>>> lengths = [len(e.address) for e in Email.objects.all()]
>>> c = Counter(lengths)

In case you’re curious, here are the results:

>>> print(json.dumps(c, indent=4))
{
"10": 8,
"11": 9,
"12": 90,
"13": 109,
"14": 196,
"15": 292,
"16": 577,
"17": 695,
"18": 845,
"19": 905,
"20": 956,
"21": 882,
"22": 888,
"23": 810,
"24": 679,
"25": 579,
"26": 517,
"27": 422,
"28": 357,
"29": 324,
"30": 305,
"31": 263,
"32": 213,
"33": 158,
"34": 99,
"35": 62,
"36": 47,
"37": 25,
"38": 21,
"39": 10,
"40": 6,
"41": 2,
"42": 5,
"43": 3,
"44": 3
}

So while ideally, the length should be set based on RFC 2821, it turns out that after five years of gathering members and their email addresses, we haven’t once come close to even 50 characters, let alone 75 or 320.

The Lessons

    in which I pedantically state something you already know

1. Always know what assumptions you’re making, and be proactive about questioning them.
2. Before you build something to solve a potential problem, figure out how likely that problem is to occur.

Jack is Back

May 13th, 2013

One by one, the shows I watch are ending or have ended: 30 Rock, The Office, House, Simpsons (wait, that’s still on?!), Futurama, Star Wars: The Clone Wars.

I’m not complaining (except in a few cases), since it leaves me with more healthy non-TV time… but now one of my all-time favorites, Community, was given extra life, Arrested Development is back on May 24th, and now this??

http://insidetv.ew.com/2013/05/13/official-24-returns-may-2014/

Just when I thought I was out…

About a year and half ago, a colleague recommended the book “The Shallows” to me, but my reading queue is so backlogged that I haven’t gotten to it. Honestly, this backlog is due in part to the percentage of time I spend each day consuming information via the Internet, favoring that over books and long-form papers…

…which is ironic.

So here’s a version of the message of the book, complete with quotes from the author:

I recognize that there is a lot of bite-sized-information-hating going around these days, and the knee-jerk reaction here could be the Luddite “Down with Twitter!” / “Throw away your smartphone!” / “Move to the Northwest Territories!”, but I believe there has to be a middle ground. These new streams of information bring to us new perspectives, new ideas, and with the right tools, the ability to curate the information.

So how can we give ourselves the space and time to do higher-order thinking, allowing our minds to process the corpus of new ideas and inspiration we’ve accumulated from Internet-based information streams?

The Boston Spirit

April 17th, 2013

In the moments following the attack, as I moved my family as far away from the marathon route as I could, this exact phrase kept going through my head: “they messed with the wrong city”… “they messed with the wrong city”. It’s not about revenge. It’s about our spirit.

Dennis Lehane describes it using better prose than I can muster. If you read something, read this: http://www.nytimes.com/2013/04/17/opinion/messing-with-the-wrong-city.html

I’m writing a blog post series (one; two; more to come over the next few weeks) for work about our love / hate relationship with email. There’s an experiment I have been running that wasn’t appropriate to go into on the GHD blog, but I wanted to write about it: unsyncing email from my smartphone.

I’ve been giving this a try, and I’ve made it almost five months now. “But why?!” you ask. “What’s the point of having a smartphone at all if you’re not going to use it to check your email?” Well, there are plenty of other things I can do on the phone to consume mobile bandwidth and keep myself occupied. The issue that I was having was with my own behavior. Prior to this experiment, in having my email linked to my phone–which I’ve had enabled since I first got the Droid 1 in 2009–I noticed the following effects on my behavior:

  • Compulsively checking my email account in every spare moment, often without thinking about it.
  • Opening an email and being unable to properly respond to it, either due to a lack of time to fully think through a response or the inconvenience of typing out a long reply on a phone.
  • Being stressed out by incoming email at a time where I was unable or ill-equipped to deal with the issue within the email. Filing the email away, both physically and mentally, didn’t remove the stress but rather pushed it out to my subconscious, where it festered. Imagine I’m in the middle of doing something enjoyable — now the experience has been ruined.
  • Spending more time in a “reactive” mode, waiting for emails to come in versus doing real work. I don’t know the source, but a while ago I read and have co-opted a phrase: “No one ever got a promotion or raise based on the speed of their response to emails.” Of course, if you’re in sales or you’re a CEO, this might not apply to you, but the principle holds: be proactive and do the work that generates the most value. Emails add minimal value at best, and I’d argue they often decrease value because they distract you from what’s important.
  • Being *less* responsive to email because the phone encouraged me to push off until later (when I was at a computer) longer responses. This habit carried over to email on the computer as well, and emails went a long time without a response (which increases stress).

On the flip-side, the supposed benefits–being more connected and responsive–are minimal, and there’s really only a few hours a day that I’m not on or near a computer. Very rarely does someone need an answer *right that minute*, and if they do, they’re likely someone who knows me well enough to text or call.

Honestly, though, I didn’t notice many of these until I started to reflect on my stress level, my phone use, and my lack of responsiveness to emails. At first, I thought the solution was to schedule email syncing for certain hours (not straightforward to do on the Android Gmail client, BTW). However, I was spurred on by this fantastic, reflective post by Harj Taggar of YC, and decided to give complete disconnection a try. With a few lapses, it’s been nearly five months now, and here are my observations:

  • It took me a solid two weeks to undo the muscle memory of checking the email accounts.
  • Complete removal of the accounts from the phone doesn’t work for me, because I still want my contacts and calendars. Unfortunately, though I can un-sync Gmail on Android, I can’t remove the accounts from the list in the Gmail app, so I still have a stale view of my account at the time I un-synced. Not a huge issue, but I’d rather not even see the account.
  • I’m finally starting to reclaim my inbox and get back to being a better emailer, partially because I’m more focused when I sit down to hash through my inbox.

Un-syncing email from the devices we have next to us 24-7 is just one component of dealing with the issues of email, but it’s a big step.

(I also removed Facebook from my phone, but that’s worthy of a separate blog post.)

http://talks.golang.org/2012/concurrency.slide

Rob Pike – ‘Concurrency Is Not Parallelism’ from Waza on Vimeo.

Rob (@rob_pike) is a software pioneer. His influence is everywhere: Unix, Plan 9 OS, The Unix Programming Environment book, UTF-8, and most recently the Go programming language.

At GHD, we’ve been hosting our newer code on github (most of which is private, but I’m working this year on making it public). It’s mostly worked well, but I noticed a while ago that I stopped receiving email notifications for commits (ok, technically “pushes”, but I’m still working on my conversion from Subversion).

Now I’m not a huge fan of email (in fact, I agree with Paul Graham that email is ripe for an overhaul), but I do like my commit notifications to show up in my inbox.

Turns out it was related to the new organization we created and my role as an owner of that organization, as Alex King explains here. Unforunately, github doesn’t make this clear on either their Notifications page or Teams page.

[Why wouldn’t owners want email notifications? We don’t spend all day staring at our github dashboard, but we do have email accounts…]

I fully recognize that my last three real posts were also rants. You’ll have to pardon me as I descend into “cantankerous old man” mode. Perhaps it comes from having two kids and looking with a critical eye at the world into which I’ve brought them. Who knows.

In case you missed it, Verizon Wireless just rolled out a new pricing plan that no one asked for. [1] Wait, we all get unlimited minutes and texts?!? 2003 me is very excited.

Yes, everyone gets unlimited minutes and texts, and then you pay a per-month fee for the data you use. The data is shared across all your devices (hence the “Shared Everything” moniker), and they’re going to allow hotspots again, both of which are nice touches. [2]

Essentially, they’ve switched from their former money-makers (voice, text) to their new one: data. They’ve made it easier for you to add extra devices (tablets, laptops) to your plan and share the data across your family. I’m sure with all those extra devices, you’ll be able to keep your data usage under 2GB, right?

The catches:

  • Each extra device will cost you $10-$40/month to add to your plan, but with no increase in bandwidth (today, you have to pay $30/device/month but get get 2GB in bandwidth for that device). In the new pricing scheme, devices and data are decoupled.
  • The devices that are more likely to rip through bandwidth (tablets, laptops) cost you less per month than the lower-bandwidth devices (feature phones, smartphones).
  • Exceeding your cap will cost you $15/GB.
  • Afraid your kids will blow through your cap watching HD Youtube videos? No problem, you can rate-limit them… to the tune of $4.99. Per month. Per device.

Luckily, if you’re one of the unlimited-bandwidth customers (like me) who was promised that you’d be grandfathered along forever, you can stay where you are and don’t have to switch to one of these plans. You just can’t get subsidized phones anymore, so a new iPhone or Droid will cost $650 instead of $199.

Back to the new plans, though. This is somehow supposed to be better for customers? This line from a rather comprehensive article in the Wall Street Journal indicates that the opposite is true (my comments in brackets):

“They [the carriers] all seem to be focused on the same objective—not beating each other up [cost-lowering through competition] but extracting value from their existing base [charging customers more for the same thing].” [3]

Hm, that sounds familiar. I think there’s a word for that. [4]

Look, my issue isn’t with Verizon trying to make money. I love capitalism. But therein lies the rub — I believe that the primary purpose of capitalism is to reward those who create new value. Competition is an important part of the value-creation process, because it ensures that the best ideas (read: those with the highest value) thrive, and over time, costs are driven down for goods that no longer provide as much new value.

Let me boil my complaints down to the top 3 problems I have with this move by VZW:

  1. It doesn’t add any value. That bears repeating: It doesn’t add any value. They are charging more money for exactly the same product.
  2. It’s a thinly-veiled money grab. The only quote from a Verizon employee referencing benefits for the consumer was “I really look at it as simplifying the voice tiers and giving customers less things to have to choose from…” [3] Seriously? We all know you’re finding ways to make more money with the same customer base. Make me work harder to find that out. To paraphrase Lewis Black: “you’ve got to manipulate me better.”
  3. It’s the opening move of an anti-competitive, anti-consumer shift by the entire US wireless industry. We have a set of entrenched phone companies refusing to admit that the infrastructure they built is decades behind what the rest of the world has — and they have an innovation-stifling group monopoly on the market. This move by Verizon is actually the first indication of the telcos realizing they’re nothing more than dumb data pipes, and they’re getting the business model wrong!

My lovely wife has had to listen to me rant about this for the last hour, and stated “wow, you’re really fired up about this, aren’t you?”

Damn straight I am.

References:
[1] http://solutions.vzwshop.com/shareeverything/
[2] http://solutions.vzwshop.com/shareeverything/pdf/verizon_share_everything_plan_details.pdf
[3] http://online.wsj.com/article/SB10001424052702303901504577462241394886300.html
[4] http://en.wikipedia.org/wiki/Collusion

  • 30+ hours of TV later…
    - Community is a fantastic show, one of my favorites right now. But Justin is right about the negative correlation between passive consumption and creative endeavors.