The article is implying there's more to be worried about than there actually is - they didn't mention anything actually "disturbing" but want us all to be concerned about the vague threat of city surveillance nonetheless.
What does Palantir do?
“integrate[s] disparate data sets and conduct[s] rich, multifaceted analysis across the entire range of data.”
How does NYC use it? Tax fraud, fire code violations, fake security guards, fake IDs, fake cigarettes, fake marijuana.
So the data already existed in NYC databases and the crimes they're enforcing already existed.
And yet:
"the potential for that kind of outright abuse is less disturbing than the ways in which Palantir’s tech is already being used. The city’s embrace of Palantir, outside of law enforcement, has quietly ushered in an era of civil surveillance so ubiquitous as to be invisible." -- total hyperbole!
If anything the most telling part of this article to me, was the small sums of money being made by Palantir which is frequently lauded as one of the most elite, selective startups for software engineering positions. It seems to operate in small change relative to all the hype.
I wouldn't be surprised if Palantir was paying Gizmodo to publish this article. It's basically a giant advertisement for the image of Palantir as some awesome software company. The reality is the software, once groundbreaking, is now mediocre, and they are a services company which happens to have a software product.
Talking about how the NYC government is using it to invade privacy makes it out as if ANY data blending/visualization software couldn't do the same thing. We all know that it could, and that Palantir is bullshit.
I have wondered on more than one occasion if Palantir wasn't simply made up of a bunch of people who know how to "work the data/analytics machine" well, supported by people who know how to get government & big business contracts REALLY well.
By "work the data/analytics machine", I mean putting together the full spectrum of data analysis tools & people - something that anybody with enough money/time/staff can do - anything from the ETL/stream processing, integrating across different databases, big data processing systems, BI tools/analytical tools, graph processing, search tech, conventional data mining, recurrent/convolutional neural networks, etc. etc. etc.
Partially, I say this out of personal experiences, having observed how government & big business is often quite motivated to spend some money, when seeing what they perceive to be "cool" and advanced data/analytics capabilities.
That's not saying Palantir doesn't have advanced tech, I am sure they built some pretty cool proprietary stuff. But, just that maybe what they do isn't such a mystery to people who understand how it is done anyway.
They seem to push the notion that they are the proprietors of amazing software for data analysis.
Then when you buy them, a bunch of "forward deployed engineers" come in and carefully comb through all your data sources and figure out how to get it all ingested and linked in their software.
Which, of course, is actually the hard part of doing data analysis at large organizations! By default, in any large org, important data is fragmented and sitting in silos--and departments defend it that way. Being able to see all that data come together would seem like magic. But it's mostly because of the manual work upfront.
Maybe the right way to think about Palantir is a data-aggregating operation that uses marketing to convince large organizations to allow them to aggregate data.
That's not a secret. Ask anyone at the company and they will say Palantir is primarily an integration company. The reason the demand so much money in for this integration is simply because no one else will actually do it despite the constant refrain of it being "only" data integration.
I actually hear this a lot. Companies with data/analytics problems that aren't being served, or, may have internal politics or technology challenges preventing them from doing the obvious data integration tasks that would benefit them. Plus, for the longest time, government and "data" work was viewed as a nightmare, boring and not "hip or cool". I've always seen that as an opportunity, one Palantir clearly has done well with.
One key role of the "forward deployed engineers" is a political one. Once you've let them tie up all your data into their platform, you're already pouring some insane amounts of money into this. Now, whatever the results, you certainly don't want to "start again with another promising solution". That's one of the reasons Palantir makes so much money out of their business.
The really hard part is creating a company that gets all of the data sources like the camera readers and videos from police cars that go to Palantir. Having that data (+ more that I don't know about) pretty much locks out other vendors because Palantir can bring more to the table besides processing the cities data.
I don't know if you can say it locks out vendors. Earlier this year, the reports were that Palantir had problems retaining commercial contracts. With their "forward deployed engineers", the image that's emerged is that Palantir is a contractor house with a patina of spies.
If NYC has data that has not previously been used for surveillance and Palantir is enabling them to use it for surveillance ("you're violating the fire code!") then it is ushering in an era of civil surveillance. I don't see what the hyperbole is.
I would argue that the surveillance isn't in how the data is used, but rather in the fact that NYC has that data at all.
Limits on acceptable data collection are more straightforward, enforceable, and fair than limits on acceptable data analysis. What would you say if a human had reached the same conclusion as Palantir by analyzing the same data?
> Limits on acceptable data collection are more straightforward, enforceable, and fair than limits on acceptable data analysis.
I strongly disagree. In this electronic age, it is unreasonable to to expect companies and agencies not to store data electronically. Isolated, these data points are not illegal - the USPS knows your address, the IRS knows where you work, the DOT knows your license plate - these are all necessary for these agencies to do their jobs. Private companies know stuff too - your phone company knows about calls you make, your ISP knows about sites you visit, your bank knows about purchases you make. Some of these can be forgotten, some are necessary to do business.
The problem comes when someone cross-references innocent data to a level that results in an invasve unwarranted intrusion into your privacy.
Should we penalize Facebook for keeping our photos? No, that's what we use it for. Should we penalize shopkeeper for recording security videos of their store to help analyze thefts? No, that's within his rights. Should anyone be allowed to correlate all the images with all the security footage to have a camera-by-camera record of the motions of everyone in the city? No, that's a dystopian horror story in the making. The criminal intent is in the analysis, not the storage.
Why is it quiet when IBM does it, but horrific when Palantir/Big Bad Thiel does it?
It's either: a) no one cares about what IBM does anymore, because they're seen as "old school", or b) no one was paying attention before and only now caught wind of it?
Today, fake cigarettes and fire code violations. Tomorrow, aggregating the 1M+ or so surveillance cams around the city. Cross-referenced with facial recognition databases, cell tower logs, social media droppings, what have you.
Total hyperbole!
It's not hyperbole at all.
The most telling part of this article to me, was the small sums of money being made by Palantir
It's called "a foot in the door." Palantir knows that the potential of the "smart cities" market is deep and vast. So that's why its initial deals with NYC -- and what better marquee client to have? -- are priced at teaser rates.
I used to be an organic chemist. These fake marijuana substitutes are pretty horrific: unscrupulous vendors read the scientific literature for newly developed cannabinoid receptor activating chemicals. These are then sold, without proper testing in humans usually.
The side effect profile of these compounds is worse than natural cannabis, with high incidence of psychotic symptoms.
No it's a reference to K2 which has actually been a huge problem in NYC. A month or two ago there were like a couple dozen people hospitalized in a single morning
I thought the numbers seemed small as well, it's probably a foot-in-the-door kind of deal for Palantir. If they can sign NYC, municipal governments follow.
At first I thought this was about Palantir leasing vast amounts of office space, as they did in Palo Alto. But NYC? They're not that big.
Take a look at the top 10 US government contractors.[1] Most of the top 10 make weapons systems. But two are in information processing: Leidos (used to be SAIC), and L-3 Communications. Palantir isn't even in the top 100. Maybe they're more into state and local customers.
There's lots of potential for innovation in the state and local government space. A smartphone app for building inspectors, for example. One that involves lots of picture taking and GPS tagging. There are building inspector apps, but they're basically paper forms reworked for tablets.
An ambitious project would be a system which takes the video and audio from a cop's body cam and does most of the paperwork. Show it a driver's license or a face, and it's in the record and understood by the system. Cops hate paperwork, yet have to document much of what they do. Automate that and cops will be glad to wear a cam. Difficult and controversial, but useful.
It might be easier to sell in countries where local government is more standardized. In the US, you'd have to customize a system for every police department.
Haha, my first thought was about Palantir leasing all of PA downtown as well. But your point is very interesting. I agree that there's a lot of value a company like Palantir can deliver to the traditionally luddite government.
> Co-founded in 2004 by Peter Thiel and Alex Karp, Palantir...
It is a continuous marvel that Peter Thiel, nominally an outspoken and prominent libertarian, is partially responsible for one of the most insidious powers that the U.S. government has over its people.
Thiel is not a libertarian, he is a feudalist. He wants to be lord of the manor and governments have a habit of getting in his way when he wants to make you his serf.
Yep. After several years of pointing out that feudalism was the natural outcome of his political sympathies, I finally convinced a friend that he wasn't a libertarian a while back.
I say this as someone who is very sympathetic in principle to a lot of libertarian thought. In practice, that's just not how humans work.
There is a way to salvage the term (and preserve its distinctive meaning from "liberal" etc). Most libertarians already speak of minimal government, as opposed to no government - ancaps are a vocal but small minority. So the question simply becomes, what exactly constitutes "minimal". And that is very much a subjective assessment.
I posit that there is a broader category, which is distinct from what people usually mean when they speak of or self-identify as libertarians, that adheres to the same principle. Basically, the idea is that government is always, by its very nature ("all power comes from the barrel of a gun"), an intrusion on some liberties - and so any extension of government requires a solid justification and thorough vetting. However, some freedoms and liberties have to be intruded upon in order to maintain others. Again, most bona fide libertarians would agree - say, the freedom to violently coerce other people is clearly not the one that you want.
But once you get into this mode of thinking, and ditch ideological stereotypes, there are many other limitations that appear perfectly reasonable. More importantly, you realize that whether some limitation is justifiable or not depends on your [inherently subjective] assessment of what is good and what isn't - but that is orthogonal to the minimal government principle. In other words, there are many different kinds of libertarians, who all agree on that basic principle, but disagree on what outcome they desire (and hence on how much government is "just enough").
So you can be a libertarian, but still consider public welfare programs to be a good way to spend money, because the alternative would be worse, in terms of overall individual liberties.
> So you can be a libertarian, but still consider public welfare programs to be a good way to spend money, because the alternative would be worse, in terms of overall individual liberties.
True, but all too often, I've heard otherwise reasonable people seriously argue for the repeal of the 13th Amendment (the abolishment of slavery) in the name of Freedom(tm) because, "You can't truly be free unless you can sell yourself into slavery."
Of course, we have seen this society, and even today can easily extrapolate what would be its effect due to proliferation of legal usury in the form of payday loans. But hey, we've got a Dark Enlightenment to usher in, for FREEDOM(tm).
Hmmm, even in "good" feudalism, the people below were entitled to protection and justice, and if they can always get it from the real government, there's a lot less the aspiring proto-Lord can offer.
In best Orwellian doublespeak he is selling his police state tools as a way to preserve liberty:
“I defined the problem as needing to reduce terrorism while preserving civil liberties.”
A classic case of opportunism wrapped in the "someone had to do it" excuse.
I worry about the same things he does and I also lean towards his approach (although I'm not as involved as he is). Think of it like this, if you're worried a certain technological development will change life in ways you do not think are beneficial, isn't the best defense against that to develop that technology yourself before others do, so you not only know exactly what it effectively is capable of as well as being able to shape the way it develops from a technological, legal and narrative perspective?
It sounds a little odd to me that people are objecting to enforcement of the law in "neighborhoods of color". Yes, enforcement of drug possession is probably out of whack, but this isn't about that. We passed those laws, are we saying that the laws are bad, or that the laws are good, but "people of color" don't deserve to live under them? Do they not deserve to live in neighborhoods without broken windows and graffiti? Do they not deserve to not worry about getting robbed on the way home from work? Do they not deserve to live and work in buildings are properly maintained?
It seems like it's kind of a trap. If you agree with the frame of the article, then you have to conclude that either 1. We passed those laws to make our neighborhoods better, but minorities and the lower classes don't deserve that, so we should ignore some set of laws there, and that totally won't become apartheid or 2. Maybe all of these nanny-state laws made by wannabe social engineers aren't such a good idea after all, since they give the Government too much arbitrary power to be enforced on whoever they don't like this week, so we should repeal them and tell people to mind their own business when they come up with this stuff.
Do you honestly believe that anybody is selling loose cigarettes in suburbia? I wouldn't say that it's never happened, but I'd bet it's rare enough that even if cops made it a top priority, they'd have a tough time finding anybody doing it.
But that doesn't really matter anyways. I thought we were talking about not throwing people in jail and giving them criminal records for petty crimes. If a black guy gets in trouble for, say, selling loose cigarettes, is he supposed to feel better knowing that if a white guy in a different neighborhood did the same thing, that guy would be getting in trouble too? Is that really the best we can do here? I'd rather try and find a way to not throw him in jail in the first place.
This argument is so seductive, but it bugs me that people assume that "crime rates" are based on how often people are committing crimes. It's how often crimes are reported / how often criminals are caught.
Given the greater prevalence of things like stop and frisk in neighborhoods with large minority populations, is it any surprise more people are caught for things like drug possession? That's going to further skew the stats, leading to more enforcement in those neighborhoods (since they are "high crime").
Drug use is actually higher in young white populations than it is among young black populations, but because of where law enforcement spends their time the incarceration rates differ wildly.
Possession is a red herring. The CompuStat statistics don't even count drug crimes and gun possession crimes--the incidence of which might be affected by the intensity of policing due to stop-and-frisk: http://www.nyc.gov/html/nypd/downloads/pdf/crime_statistics/....
Here is the NYC crime map: https://maps.nyc.gov/crime. Look at the maps for felony assault or rape. Stop-and-frisk isn't going to change the incidence rate of those crimes.
You obviously have a more sophisticated take on the question. Parent comment referenced "75% of crime," not "75% of violent crime" or "75% of rapes and aggravated assaults recorded in compustat."
The general point holds - there are corrective measures that need to be taken to avoid skewing crime data as a result of increased enforcement, and in the case of some jurisdictions those measures are being taken.
What's not obvious to me is whether police are increasing the severity of the charges based on where they are, even if the charges wouldn't necessarily hold up in court. There's a case to be made that that would be an efficient tactic - public defenders will encourage plea bargains and it gives the DA more leverage to settle the case quickly and efficiently. The opposite may be true when booking a drunk banker who gets in a fistfight, or a privileged college kid who rapes his date behind a dumpster.
Also, the discussion generally is not about CompuStat, it's about "quality of life" improvements prosecuted with the use of secret databases that are not publicly available. So, you know, there's that.
It seems like you're making an argument that crime statistics should not be used as a metric to focus law enforcement efforts.
What other metrics should we use? If you simply assign patrol routes based on population you are going to under-serve areas with higher crime and over-police areas that don't need it.
I'm arguing that it's a more complicated statistical problem than "we see more criminal 'events' in this neighborhood, so we should send more cops there."
Offering criticism without offering an alternative isn't helpful.
Of course it's complicated, and of course crime stats are a simplification. Most stats are. The point is, stats are much better than going by biased "gut feelings".
Yes there are underlying issues with stats such as underreporting, but the solution isn't to get rid of the stats. It's to improve the underlying cause of the bias in the data, such as improving police/public relations.
It's important to keep in mind that in a "racist society" machine learning will also tend to be racist (but much more so). After all, the "AI" is learning from human data, so it's certainly not as objective as you would think a "machine" to be.
I assume you are being sarcastic, but there are plausible hypothesis that could be tested to answer this question. For example the genes that have caused a reduction in human pigmentation in some populations have occurred very recently (within historical times) and under very strong selection. Genes that influence the propensity for impulsiveness or violence may have been carried along with the selection for low pigmentation.
Another possible hypothesis is that since melanin and dopamine have an overlapping biochemical pathway that the genes influencing melanin production may also influence dopamine levels.
A third possible hypothesis is that since different populations have historically lived at different densities this may have influenced how well people interact peacefully at high population densities (i.e. how domesticated people are in the same way cows are much calmer than wild cattle).
All of these are testable hypothesis that no sane scientist will ever study - in the end we just don't know the answer and most likely never will.
It is a question of a crime taking place vs one which results in an arrest, convictions and statistics. I worked on Wall St for many years -- there are a lot of drugs on Wall St, more than i've seen anywhere else. Yet the rules are rarely enforced as they are in other neighborhoods. So of course, Wall St wont make it into the statistics. It is a vicious cycle.
There are numerous reasonable proxies; locations of murders, reported assaults or robberies, calls to the police department reporting a possible crime, etc.
MongoDB is In-Q-Tel backed - so by that sensationalized metric - anyone who runs a MEAN stack is thus "CIA Backed" as well. Hilarious to see the kneejerky lack of intellectual rigor around HN these days.
Almost as hilarious as your strawman argument. Palantir was started with the explicit mission of aiding the surveillance-industrial complex. Mongo Inc was started with explicit mission of selling cloud databases(PaaS) to the general public.
“Basically,” Thiel explains, “I thought that some of the approaches that PayPal had used to fight fraud”—which at one point posed an existential threat to PayPal—“could be extended into other contexts, like fighting terrorism.”
It's also an interesting business model. Core platform tech and then consultancy.
A LOT of hand-cranking integrations and massaging data, presumably that's where they make most of their money - billing hours to agencies with deep pockets.
Even the most bass-ackwards database designs can be reverse engineered by bright people in a day or two (we know Palantir can attract these people) especially if they have some scars dealing with dumb database design. And it's probably not likely that the person who built that database and the program that uses it went to great lengths to obfuscate anything. I'll even go a step further and say that most data sources are probably really simple programs. Throw in any kind of documentation as a bonus and the learning curve just goes down.
What the hell could they be charging that much for?
Mixed feelings for me, too. I'd really love to see that "all things city knows about this location" tablet app. It's just amazing to have this information easily available. But then the pesky issues of privacy and human conflict pop up, and we can't have nice things :/.
Palantir likes to promote these minimal links to the CIA quietly to increase the feeling that they give to their "forward deployed engineers" that they are some Seal Team 6 of developers. Their office in the Meatpacking district of NYC is kinda reminiscent to a cult headquarters. Rooftop bar/DJ booth unlimited* vacation time, bring your pets to work, free gym membership right nearby, every desk has a "go bag"... Why would anyone want to leave?
I've visited the NYC office for dinner a few times. There's no DJ booth, nor a rooftop bar, nor free gym memberships (according to a friend), nor go bags.
There's a rooftop patio, as many buildings in the area have, and gym subsidies as most tech companies have. I've been told they have emergency readiness kits (which are advocated by every level of government in every jurisdiction), but they're definitely not at the desks (or they are, but invisible).
How are any of these criticisms distinct from Google, which owns an entire street block (no joke street to street, Ave to Ave)? I've had more than a few meals there as well and it's pretty much the same situation as far as I can tell.
"they are not all accounted for, the lost seeing stones ..."
why anyone thought it was a great idea to name their company after the remote sensing device guaranteed to lie to you and make humans suicidally depressed has always been beyond me.
The City Hall official discussed the city’s use of the data-mining technology on background, and declined to provide the full list of data sources or describe what is contained in the datasets.
Presumably this technology is supposed to be helping the people of NYC. Shouldn't these people know what data is being collected about them so they can decide whether or not they actually want it?
There's a few other recent developments of the "Big Data" city that New York is aspiring to that also give some residents mixed feelings. In chronological order:
When the government calls it "Open Data" (ie. https://nycopendata.socrata.com, https://data.ny.gov) it's lauded as an inspirational embracing of transparency and seen as just awesome. When the government uses it to make itself more efficient and effective, there's this implication that something nefarious is going on.
If you want to be offended by government software purchases, check out the POs for Oracle Enterprise whatever or the IBM Passport Advantage agreement that NYC has issued in the last year. Chances are, the city doesn't even know wtf half of the products they license are!
The difference is that the executives buying Palantir's products don't know or care what any of those acronyms are, and they're willing to pay a lot of money so they don't have to find out.
Wow, just a few million dollars in contracts from NYC agencies. That is peanuts. Not sure how they are able to justify such a high valuation. Seems like a lot of hype at Palantr
This is just Palantir's foot in the door. Other municipalities will want to follow in NYC's steps, as Palantir essentially expands the smart city market. When the market is big enough, then Palantir will start pricing in a manner that actually makes them money. At least in theory. The similarity to a typical startup (low price to build demand, growth drives high valuation) is uncanny.
I'm sure if your number is up, they will find you. :-)
There seems to be a lot of momentum in the "canvas large data sets" space. It has always been on the wish list for the authorities (see many RFP's for the DARPA "Total Information Awareness" initiative) and the challenge has always been storage and algorithm development. Storage is becoming a non-issue when you can have a petabyte in M.2 class SSDs available across a 10Gbit network of processors. The challenge is the needle-in-haystack finding activity.
Interesting that all this data is primarily being used to catch tax violations and improving property prices. So from a city manager point of view, this may be paying out for itself.
Does any one know how Palantir started? I.e. from what I can remember it seemed to come out of nowhere. How did they manage to get such huge contracts from the start?
IIRC Palantir had difficulty getting clients early on. The tech evolved from Paypal's efforts to combat fraud via data analysis, esp. against the Russian mafia trafficking on their platform. The CIA was one of its first clients, which eventually led to other government agencies trusting them.
What does Palantir do? “integrate[s] disparate data sets and conduct[s] rich, multifaceted analysis across the entire range of data.”
How does NYC use it? Tax fraud, fire code violations, fake security guards, fake IDs, fake cigarettes, fake marijuana.
So the data already existed in NYC databases and the crimes they're enforcing already existed.
And yet: "the potential for that kind of outright abuse is less disturbing than the ways in which Palantir’s tech is already being used. The city’s embrace of Palantir, outside of law enforcement, has quietly ushered in an era of civil surveillance so ubiquitous as to be invisible." -- total hyperbole!
If anything the most telling part of this article to me, was the small sums of money being made by Palantir which is frequently lauded as one of the most elite, selective startups for software engineering positions. It seems to operate in small change relative to all the hype.