An insurance company might say, "Tell us more about yourself so your premiums can go down." When they say that, they're addressing the winners, not the losers.

Cathy O'Neil

Categorized in : Might Loser Winner

We can't just throw something out there and assume it works just... by Cathy O'Neil

We can't just throw something out there and assume it works just because it has math in it.

There are lots of different ways that algorithms can go wrong, and what we have now is a system in which we assume because it's shiny new technology with a mathematical aura that it's perfect and it doesn't require further vetting. Of course, we never have that assumption with other kinds of technology.

auras technology perfect

By construction, the world of big data is siloed and segmented and segregated so that successful people, like myself - technologists, well-educated white people, for the most part - benefit from big data, and it's the people on the other side of the economic spectrum, especially people of color, who suffer from it. They suffer from it individually, at different times, at different moments. They never get a clear explanation of what actually happened to them because all these scores are secret and sometimes they don't even know they're being scored.

data successful white

When I think about whether I want to take a job, I don't just think about whether it's technically interesting, although I do consider that. I also consider the question of whether it's good for the world.

jobs interesting thinking

I know how models are built, because I build them myself, so I know that I'm embedding my values into every single algorithm I create and I am projecting my agenda onto those algorithms.

embedding algorithms agendas

Evidence of harm is hard to come by. by Cathy O'Neil

Evidence of harm is hard to come by.

evidence hard harm

Every system using data separates humanity into winners and loser... by Cathy O'Neil

Every system using data separates humanity into winners and losers.

data loser humanity

Because of my experience in Occupy, instead of asking the question, "Who will benefit from this system I'm implementing with the data?" I started to ask the question, "What will happen to the most vulnerable?" Or "Who is going to lose under this system? How will this affect the worst-off person?" Which is a very different question from "How does this improve certain people's lives?"

data doe people

I don't think anybody's ever notified that they were sentenced to an extra two years because their recidivism score had been high, or notified that this beat cop happened to be in their neighborhood checking people's pockets for pot because of a predictive policing algorithm. That's just not how it works.

two years thinking

Micro-targeting is the ability for a campaign to profile you, to know much more about you than you know about it, and then to choose exactly what to show you.

profile ability campaigns

There might never be that moment when everyone says, "Oh my God,... by Cathy O'Neil

There might never be that moment when everyone says, "Oh my God, big data is awful."

data awful might

The disconnect I was experiencing was that people hated Wall Stre... by Cathy O'Neil

The disconnect I was experiencing was that people hated Wall Street, but they loved tech.

wall hated people

So much of our society as a whole is gearing us to maximize our salary or bonus. Basically, we just think in terms of money. Or, if not money, then, if you're in academia, it's prestige. It's a different kind of currency. And there's this unmeasured dimension of all jobs, which is whether it's improving the world.

bonus jobs thinking

I would argue that one of the major problems with our blind trust in algorithms is that we can propagate discriminatory patterns without acknowledging any kind of intent.

algorithms patterns arguing

The Facebook algorithm designers chose to let us see what our friends are talking about. They chose to show us, in some sense, more of the same. And that is the design decision that they could have decided differently. They could have said, "We're going to show you stuff that you've probably never seen before." I think they probably optimized their algorithm to make the most amount of money, and that probably meant showing people stuff that they already sort of agreed with, or were more likely to agree with.

talking people thinking

I think what's happened is that the general public has become much more aware of the destructive power of Wall Street.

streets wall thinking

Most people don't have any association in their minds with what they do and with ethics. They think they somehow moved past the questions of morality or values or ethics, and that's something that I've never imagined to be true.

people past thinking

We don't let a car company just throw out a car and start driving it around without checking that the wheels are fastened on. We know that would result in death; but for some reason we have no hesitation at throwing out some algorithms untested and unmonitored even when they're making very important life-and-death decisions.

life-and-death car decision

When people are not given an option by some secret scoring system, it's very hard to complain, so they often don't even know that they've been victimized.

secret complaining people

There's less of a connection for a lot of people between the technical decisions we make and the ethical ramifications we are responsible for.

decisions-we-make connections people