AT&T's guilt-by-association algorithm for finding "terrorists"

Ed Felten Andrew Appel has some fascinating analysis of the "guilt by association" algorithm that AT&T uses to help the FBI figure out whose life to ruin with baseless accusations of terrorist involvement. Other phone companies like Verizon refused to help out with these fishing expeditions, but AT&T jumped right in. The thing is, after three hops, your social network encompasses half the planet, including many of its terrorists (or even "terrorists").

What is the “communities of interest” technology? It’s spelled out very clearly in a 2001 research paper from AT&T itself, entitled “Communities of Interest” (by C. Cortes, D. Pregibon, and C. Volinsky). They use high-tech data-mining algorithms to scan through the huge daily logs of every call made on the AT&T network; then they use sophisticated algorithms to analyze the connections between phone numbers: who is talking to whom? The paper literally uses the term “Guilt by Association” to describe what they’re looking for: what phone numbers are in contact with other numbers that are in contact with the bad guys?

Link

Update: Wired/Threat Level's Ryan Singel sez,

Following up on Freedom to Tinker's post on AT&T's calling circle research adopted by the FBI, it seems that data mining program was made possible through AT&T's development of a mass surveillance programming language called Hancock.

A variant of C, Hancock is used to process millions of records in streams as they get dumped into various databases. Uses include creating maps of cell phone users locations and tracking IP and websites addresses.

AT&T even snagged patents on some of the data mining methods, which may seem eerily familiar to the phone record data mining the NSA used post-9/11 to find targets for their warrantless targeting of American citizens.

[#1 posted by seyo , October 29, 2007 10:50 AM]

uh, if they know the phone numbers of the "badguys" to begin with, what do they need the algorithm for???

[#2 posted by han , October 29, 2007 11:11 AM]

If the algorithm is any good, they could begin with the numbers of a couple of dozen slightly naughty individuals and work their way up from there to gradually more and more evil phone owners. As is well known, they only need about six levels of guilt by association to reach whoever is the current ultimate bad guy.

[#3 posted by Sam , October 29, 2007 12:37 PM]

I prank call Bin Laden ONE TIME and now every time I go to the airport I get the full cavity search... sheesh.

[#4 posted by flamingphonebook , October 29, 2007 2:44 PM]

I wonder if Kevin Bacon is going to be arrested.

[#5 posted by sexyrobot , October 29, 2007 3:19 PM]

i didn't think at&t could get any more vile. guilt by association? ew.

i cheer myself these days with visualizing their downfall in graphic detail...

[#6 posted by airshowfan , October 29, 2007 6:35 PM]

If six degrees gets you to anyone in the world, three degrees does not get you half the planet's population. It gets you (population^(3/6)), or sqrt(population), which is about 77,500. That's if you assume a non-overlapping tree, since an overlapping tree would have to grow faster than (population^(1/6)) with each initial degree (to make up for the slower-growing subsequent degrees), but given that you have 6 steps to get to a few billion people, the total effect of the last three steps can easily be way more than just doubling. (The first three steps surely will be!).

Sorry, I'm in a nitpicky mood. Yes, what AT&T did was wrong and they should not get away with it. I am as worried about our increasingly fascist government as anyone else (including, for example, Naomi Wolf; Surveillance is step 4 out of 10, right?)

[#7 posted by jphilby , October 31, 2007 7:00 PM]

The Man and AT&T have always been *like this*.

Like any good housewife who knows her place, Ma Bell does what she's told, with docility and meek satisfaction.

She also knows why she's far better than you, which is why she was losing customers ... until the living operators were replaced by cheerful recordings. As commemorated by Lily Tomlin's "Ernestine", more reportage than comedy.