The Happy Technologist Interesting Geekdom


Datamining, Privacy, and Ethics

[I'm trying to write shorter blog posts these days -- let's see how that goes]

There was a lot of chatter recently around about how Target (the shopping chain) has used data mining to identify pregnant shoppers in an effort to woo them as loyal customers. This is a prime example of things that are of direct interest to me: data mining, privacy, and the ethics surrounding the vast amount of knowledge we can compile about everything today, so I thought I'd share my perspective.

First off, the NYT article should not have been a surprise to anyone familiar with data. I've worked very closely with data mining teams on large retailers, insurance companies, and government agencies, and they uncover correlations all the time that lead to spooky predictability. The classic example of this is correlated sales of diapers and beer (from

A number of convenience store clerks, the story goes, noticed that men often bought beer at the same time they bought diapers. The store mined its receipts and proved the clerks' observations correct. So, the store began stocking diapers next to the beer coolers, and sales skyrocketed.

One common interpretation was that a new father was sent out in the night to get much needed diapers, which put him in the mood to buy a six-pack.  Of course, that last part is purely subjective, but that's the story.

The article goes on to call this a "myth," but even if the specific case isn't verifiable, the decades-old example is on point for what it describes: Everyone™ is trying to make money by learning about predictable patterns, then exploiting those patterns to achieve their goals.  This has been going on for thousands of years at a very human level in sales: vendors put up shops in high traffic areas, they're careful what they put in plain view to attract customers, they offer sales on one item and try to get you to buy more things once you're there, they give better prices to loyal customers.  Think of those examples in a modern shopping mall, then think of them in an ancient city square. It's not hard to imagine examples in both places.


Ranting about Economics

Note:  I ramble a lot in this post, and I'm not sure I agree with everything I said, but I'm starting back to work today so I don't have a lot of time to muck with it and I'm trying to get content out, so... you've been warned.  If you want some interesting reading on the topic, here's a few links: 


I just entered West Virginia.  This is because I'm on a trip and driving to the beach, having spent the night and dropped off a couple of dogs in Kentucky (with people [well… family], not just anywhere you know), we’re now safely ensconced in a Jeep Liberty, the four of us (two real people, two seventeen year olds) enjoying the extra space the dogs left us.  Traveling this way means that you see a lot of countryside and inevitably have random conversations with family members you never see about politics and economics.

In particular, since the world economy has taken a bit of a dive lately, I figure it’s time for my personal rant on the topic.  Let me start by saying that I’m not fond of economics, at least not formally.  This stems mostly from an unfortunate economics teacher in college and my background in not-being-stupid.  In one of our early classes, the professor drew an x-y axis on the chalkboard, placed a single data point, and after only a few moments of discussion drew a very attractive wavy line through it and called it a “supply-and-demand” curve like this:

Anyone that is not very upset by that chart should stop reading now, so that I do not offend you, and immediately go unfriend me on Facebook or put me in your “icky” circle on Google+ or something.

Here’s the math 101 short course for anyone that ignored my previous paragraph:  you can’t draw a curve through only one point of data, because you don’t know which way the curve should go.  It takes two points just to make a straight line, and at least three points to make a curve (and normally lots more unless you’re sure what shape the curve has).  Is it a one-humped camel, or a two-humped camel?  Or a sea-serpent?  You get the idea.  My relationship with formal economics went downhill from there.

Now that’s just the taste in my mouth – I admit it’s hugely important to study and understand how economies work.  I am currently undergoing a microeconomic experiment by having just given the aforementioned 17 year olds $100 apiece to buy their own stuff for this trip so they won’t bother me and will hopefully learn the value of a dollar.  Already they have passed up $7 slices of pizza to save money, so we’re learning something.