Famously, the last 'enemy' we killed in Afghanistan, alongside the 7 children and others surrounding him, was somebody who was working for a US aid organization. [1] A drone operator saw him putting bottles of water into his trunk, and somehow thought they were bombs, so they tracked him for 8 hours, then killed him and his family.
This happened after I stopped working for the DOD, but I am 99% sure the program in question during this incident was something I worked on. It was effectively an anomaly detection system for identifying any weird behaviors detectable in WAMI (wide area motion imagery, which is high resolution, low framerate drone video data that covers a large geographic area). As an example from the unclassified sample project they asked us to do as part of the bidding process, our code flagged a car doing donuts in a parking lot (the sample data was just from a random US city on a random day because this was an unclassified open bidding process). The system was not and never was intended to be a "terrorist" detection system. The idea was always to flag potentially interesting events for a human analyst to follow up on. The disconnect it took for someone to blindly trust the program as a target acquisition system boggles my mind. Someone doing donuts in a parking lot should not be automatically targeted with hellfire missiles, yet in practice that is how they were using the tool. I still lose sleep over it.
Whenever you introduce (or grow) the presence of computer automation you need to consider the human / social interactions. You are responsible for the whole system, not just your software.
It is well understood that human diligence falls dramatically in the face of automation that is perceived to be highly reliable.
There is a tendency for software people to say “it has a 5% error rate, so make sure to carefully check every time”. Even though we know that humans simply won’t check carefully the results of a 95% accurate system.
> There is a tendency for software people to say “it has a 5% error rate, so make sure to carefully check every time”. Even though we know that humans simply won’t check carefully the results of a 95% accurate system.
This should really be framed differently during training. "Assume you have 100 neighbors in your area/house. We expect you to kill 5 innocent neighbors each time you use the system, and don't actually think for yourself. (After using it once every day for two weeks, you will have killed half of your neighbors.)"
Suddenly 5% doesn't sound that small/95% not as good to the laymen.
IMO you should increase the error rate synthetically, adding tests. So you get 10 targets to review, but 4 of them are known to be bad and the system is both checking the user, and also increasing the perceived error rate to encourage diligence.
Brilliant! This is the logic behind the birthday problem where you only need 45 people present to have greater than 90% chance that 2 people share a birthday.
Even rare events become very likely when you repeat it. A 5% wrong chance means half are dead after 10 tries
> Even rare events become very likely when you repeat it.
True, but in my opinion 5 innocent dead out of hundred is not remotely small. For any serious automated decision anything larger than per million is completely unacceptable. (For reference, among Germans the amount of Jews killed by Nationalsocialists, is "only" 2.3‰ [160k vs. 69M] I don't think anyone considers that to be small.)
> A 5% wrong chance means half are dead after 10 tries
Btw., after using it for a year every day, the US population would be 2.52 people. And I think the US military plans to use their expensive systems for much more than a year.
A computer can never be held accountable, therefore a computer must never make a management decision.
We have CEOs designing AI systems to replace them (or at least their workload). The DoD has done nothing to provide accountability for this war crime. It's a scapegoat to move on from, not a problem or justice.
> Your work will be used against innocent civilians in far away lands.
For now. I think this administration will absolutely have no compunction about using this against innocent civilians in THIS land as soon as they feel the need to to retain power.
The answer is not to avoid engaging with a corrupt institution. US is a democracy: vote, be active politically, go into organizations that need change.
So you are suggesting that you can change the US military industrial complex by.. working for the NSA or Lockheed Martin ?
Also it is quite questionable how much of a democracy the US is nowadays.
Gerrymandering aside, the populace does not get to choose the presidential candidates. And the whole two party thing ? Or the fact that to have a chance you need a super PAC with hundreds of millions ?
Yes, that's what I'm suggesting. If you feel that the problem is higher up or lower down in the institutional hierarchy, then go higher up or lower down. That's the civil approach, as I understand it.
Apart from the fact the if you stay true to your morals you will be incapable of rising to a position of power in such corrupt institutions, the degree of institutionalisation necessary to do so will definitely erode your ethics, turning you into another cog.
In my opinion, you underestimate the power of an individual. I'm not saying you have to be Gandhi, but he was one person. A problem with your assessment, is that if you're right, or, if everyone believes you're right, you'll have to wait for some external force to come in and change things, like a calamity or desperate revolt. By the time USSR broke apart, bread riots were happening, and, tragically, 30 years later we still have the same authoritarian, corrupt regime (in all but name) wrecking havoc at home and abroad. That's what a people get for believing in the story of disempowerment. That's my take, at least.
The Brits were bankrupting themselves with war spending, which culminated in WW2. After its conclusion, they started liberating every territory that wasn't directly profitable, including some that did not even want to be liberated. WW2 ended in 1945, India was liberated in 1947 alongside numerous other territories. Without WW2, India would likely have remained a British territory well past 1947. Beyond that, Gandhi wasn't one man - but the leader of a movement of millions.
As for Russia, I think your perception there is also driven by 'messaging'. There's a million videos of people streaming walking tours and the like in Russia. Here [1] is one done 4 days ago. That's a country that in the 90s was on the verge of collapse and was extremely dangerous. You can find similar stuff on e.g. Twitch because there's this somewhat odd subculture of delivery drivers, taxi drivers, and others streaming their days. Perhaps in part out of bemusement from this Western perspective paired against reality.
I agree in principle, but also agree with gp that in practice the things one has to do to rise to a position of power in corrupt institutions make it virtually impossible to be a credible agent of change, especially private/secretive institutions like military, intelligence, military-industrial corps. I guess politicians are the one role that is supposed to be able to operate without those institutional constraints but of course, (in the US) given corporate capture of political speech, political hyper-polarization, etc, in most cases they are just as or more constrained. It is a hard problem.
Snowden and other whistleblowers, of whom there have been many (many of which paid arguably a dearer price than Snowden did), made it plain obvious to everyone that US intelligence services are doing illegal things abroad and at home. Before that, only tinfoil-hat-types thought so: now everyone knows it. They moved the Overton window a tremendous amount.
I don't think actual change comes from people lobbying for some bill or whatever. It comes from a far more imperceptible shift in the zeitgeist of society, that is then eventually formally reflected in some way or another. For instance the Civil Rights Movement only succeeded because society had already largely shifted, before it had even begun, in a way that made it viable. A Civil Rights Movement in Saudi Arabia isn't going to succeed, because it needs the widespread desire for such to already exist.
And in this regard I think Snowden had a bigger influence on society than anybody or anything in my living memory. Prior to him there was a belief by many that government conspiracies couldn't happen because somehow if there were enough people involved, it'd show up on News at 9 or whatever. He made it clear that such conspiracies, far beyond the bounds of what even the absurd conspiracy theorist might propose, can and do exist. The NSA has friggin spies in World of Warcraft!
That sort of skepticism played a key role in fueling an anti-establishment mindset. And that mindset is what fueled the rise of everything from Trump to the DSA. I'm certainly not claiming those are good outcomes, but merely part of an increasingly visible shift away from the old establishment. And I think this shift will be, in the longrun, good for America. And I think in hindsight, it will be clear that a lot of this change started happening with Snowden.
I'm probably too focused on the Afghanistan war but I don't think that voting differently would have extended or shortened the war.
Both republicans and democrats wanted the war in Afghanistan back when 9/11 happened and they needed to get revenge on the scapegoat. Meanwhile the end of the war was initiated by Trump, then slowly concluded by Joe Biden.
So both the start and the end of the war had bipartisan support. Nothing against democracy but war doesn't follow the election cycle. It might start in one though.
Not sure about the downvotes, this is correct even if not pleasant to read (people need to be heroes of their little bullshit life story and all that).
It may be an afterthought for US citizens living in US, but its a deadly weapon pointing to 95% of the mankind and people wielding it are often shit to be polite. We the rest do care.
I mean, this site is full of people that develop the weapons used to commit atrocities and genocides (both physical and digital ones) so I am not surprised about the downvotes..
I actually ended all govt contracts I was working on after this and ended my clearance forever. This was my "I can't keep working for them" moment. One of the purest examples of the US operating on ISIS levels of morality that exists.
Killing someone for absurd reasons then lying about it because while knowing it's an absurdity/mistake risking it happening again seems almost worse than believing/buying into an extreme/insane worldview and killing someone for it.
[1] - https://www.bbc.com/news/world-us-canada-58604655