This is relatively old "news" and the problem has since been resolved. It had to do with the coating used on one of the coils interfering with the software--as the article mentions, there is no redundancy for this system. The part was supplied by a subcontractor. General Atomics [1] is VERY vertically integrated (even making servos) and uses few COTS parts. Engines (and components), unfortunately, are pretty difficult to make--many of the engines were initially purchased second hand.
Once the problem was diagnosed, fielding the solution is / was another challenge. Some of these aircraft are in pretty remote areas.
These aircraft gained popularity and rapid adoption because they're substantially less expensive and VASTLY quicker to manufacture than any other option. Almost all customers prioritized cost and acquisition speed over long term reliability, which of course comes with a certain amount of risk. Testing and certifying an airframe to standards is expensive and time consuming [2] but has the advantage of improving reliability.
Also, this is just hogwash journalism.
> Last year, the Army reported four major drone crashes, each involving the Gray Eagle — a model identical to the Predator.
Warrior / Gray Eagle / MQ-1C [3] is a 40% payload increase over Predator / MQ-1 [4] and the airframe was almost completely redesigned. Hell, the engine is even different. It's a different aircraft.
> Improper coating causing a thermal problem, either causing a sensor to fail, lie or some electronic component to break it's solder.
This is the leading theory. Most of the observed failures seem heat related and are likely due to an increased resistance of the brush to shunt connection. This can be caused by quality issues of the brush rivet during initial manufacturing (coil coating) and/or improper torque of the brush
shunt screw during assembly (installation error).
The variation in resistance at individual brush sets can lead to uneven current density, causing high heat at the brush to commutator interface which results in excessive wear and damage to the commutator surface and/or brush and ultimately leads to a starter generator failure.
In addition, the software wasn't reporting failure of the component.
I should add a disclaimer that root cause failure investigation is extremely complicated and difficult. It's possible that not all failures were / are the same and there are many other possible contributing causes.
> The Air Force also has contracted out more drone missions to private companies to meet what one general called “a virtually insatiable appetite” from military commanders for airborne surveillance.
It sounds like for now, contractors are only running surveillance missions.
What if contractors:
* Fly the drone and release ordnance that kills someone.
* Fly the drone but military personnel pushes the button that kills someone.
Does the contractor as a civilian bear liability differently than a member of the armed forces? Could a foreign country try to extradite and prosecute them? Could a foreign country arrest them if they left the USA?
Officially civilians fly unarmed surveillance missions and uniformed personnel fly missions where ordnance is carried. They may further limit civilians, or maybe not.
That said contractors have been doing the intelligence analysis behind missions to kill people using drones. Even though the missiles are fired by uniformed soldiers, there's a big gray area right behind them.
What we've become? You think if armed drones were in the inventory during Iraq, Vietnam, Korea, or WWII they wouldn't have been used because of some lost sense of nobility?
Re-framing is incredibly powerful [1]. Operators don't kill people, they neutralize targets and complete operations.
This will always happen in a military context, and this is why it is so important to have real civilian oversight. Expedience plants the seeds of future wars.
If they had happened to be captured by the forces of the opposing governments, they could rightfully have been treated as mercenaries or even as common criminals, rather than prisoners of war. This was a moot point though, since neither side made any pretense of observing international law in terms of treatment of PoWs.
In "Ghost Fleet: A Novel of the Next World War", America's warfighting is crippled as the electronics that were manufactured in China contained backdoors that allowed them to shutdown or destroy the various computer systems.
The US military is already wary of this attack vector. Likewise, Australia was leery of letting the Chinese telco Huawei do any work on their National Broadband Network infrastructure.
As we make these more lethal and harder to detect, I can't help but wonder: when will the first drone attack on US soil take place? When will the first US president be assassinated by drone?
I like how you think it will be some foreign third party doing the killing, and not CIA on the direct order signed by the president, you know, like they do it now.
No doubt the data shows a problem exists, but the stats used by this article a worded pretty dramatically. Saying "highest ever total" doesn't alarm me because drone use is (relatively) new, and increasing at a large rate. I'd expect damage costs to rise along with increased use (although ideally at a slower rate as we gain experience and make improvements).
"Things have gotten so bad that the Air Force is offering retention bonuses of up to $125,000 to its drone pilots, who have long complained of overwork."
There is a disconnect, flying by wire you don't feel turbulence and changing conditions, rather you logically go about it, being overworked its not hard to see where one can whoopsie daisy it.
That's a hazmat suit. And given the nature of the jet fuel, the paints, and metals involved in aircraft components, you'd probably want to wear something similar when wading through the crash site until it was cleared.
> Investigators have traced the problem to a faulty starter-generator, but have been unable to pinpoint why it goes haywire or devise a permanent fix.
I find it incredible that they know where the problem is, but not how to fix it. Get 100-1000 of them and start stress testing them in a lab. This stuff isn't rocket science.
I do love when people imply a "lab" is this mythical thing that can, through no effort, reproduce any likely or unlikely condition that one could ever expect to encounter.
They should just print out a sign that says "lab" put it on an empty room, place the starter motor in the middle of it, and then suddenly they would know why it was broken. It is in a lab after all! This isn't rocket science! Or if one in an empty room fails put 100-1000 into the empty room, it has GOT to work!!! It is too easy to fail! Being in a lab is all that it takes!
I'd suggest you watch this[0], this issue was literally costing human lives, they knew there was a rudder issue, they placed it in a lab, testing it for thousands of hours and ultimately found it by a little luck. It moved our understanding of failure states forward.
There's no reason to assume that finding the issue with the starter motor is easy or trivial. No amount of lab hand waving changes that, it will take just brute effort.
Yes, clearly it's not just "dump motors in room, get results", but rather that it's usually very doable to generate failures. There are only so many reasons that they can fail and it shouldn't be too difficult to generate the circumstances that cause those failures.
Obviously figuring out which circumstances are the ones that are actually responsible for failures will be more difficult. But it's not too hard to do some sensitivity analysis to get some clues.
If they're highly sensitive to having too much current drawn then that might make you suspect that somehow, somewhere the system's power demands are overwhelming the generator and that's what is causing the failure.
The point behind "it's not rocket science" is that rockets are very, very expensive to make and so you can't just test as many of them as you'd like, for as long as you'd like, until you through trial and error reproduce the conditions that caused the failure. You also don't get the rocket back when it either succeeds or fails.
But in this case the starter/generators are cheap, and they are able to in many cases sort through the debris of the crash. So that does make it qualitatively and quantitatively different than rocket science.
It's not rocket science, but it is aeronautics, and duplicating the failure conditions (temperature, airspeed, altitude, vibration, load) for such a suite of "100-1000" generators could be challenging. It strikes me as presumptuous to assume the engineers on the project are incompetent.
You can't see software. You can't touch it. It can spontaneously change shape, form and function in dramatic ways if a single memory bit gets corrupted.
That's very, very different than physical objects.
On the other hand, I can relatively (meaning compared to physical aircraft) easily and inexpensively replicate thousands of copies of my prototypes and automate exhaustive testing. Complex systems are just complex - very few things are actually as trivial as a bystander may think.
> It can spontaneously change shape, form and function in dramatic ways if a single memory bit gets corrupted.
> That's very, very different than physical objects.
True... spontaneous changes to the structure of physical objects are quite a bit more likely. One of software's greatest features is that it's not subject to the physical wear that physical tools experience.
Please point me to the software that controls the alternator/generator/starter motor. I'm not talking about the whole thing, just the part whose failure is causing the larger failure.
When software controls hardware, yes things can get weirder. But the software can't cause magical things to happen; the atoms don't rearrange themselves.
Bit flips tend to be much more like magic, if applied to the real world. A wrench doesn't stop working because you swap out a single atom. But software can, if you flip the wrong bit.
Could we please have a 'usually' in that sentence. I've had one or two where even though reproduction was easy fixing it was not a small step by any accounting method. In one case the fix required a total overhaul of the system. Several months turnaround on that one between the reproduction and the eventual fix.
> Obviously figuring out which circumstances are the ones that are actually responsible for failures will be more difficult.
Actually if the components catch on fire or something finding out which one shit the bed is by far the easiest part of the entire ordeal, assuming it's not spread over the face of a mountain.
I find it incredible, too. As UnoriginalGuy says, lab tests don't always tell you what's going wrong in the field. But the fact that they don't know what's happening (if that's true at all, some other posts have called the whole article into question) is still the designers' fault. With modern hardware, the most important thing -- the thing you start building on Day #1 of the project -- is built-in diagnostics.
In the case of something like the 737 rudder problem, those planes were designed 40 years ago, in an era when it wasn't practical to record every bit of operating data in real time, or add cheap, lightweight sensors to everything in sight. That's not true anymore. Whether they're building a drone, a fighter plane, a spacecraft, or even a modern passenger car, there is no excuse for requiring people to stand around some wreckage scratching their heads. The onboard controllers should be able to tell them exactly what went wrong. If they can't, then that's the problem to be solved first.
Yes, we do now have far better onboard sensor suites and logging than in decades past.
And this can provide incredible amounts of information, and get us closer to the source of issues much faster. Many times, it makes the issue immediately obvious and documented.
HOWEVER! Do NOT make assumptions and fool yourself into thinking this is a cure-all. There are many problems that, while the logged sensor data narrows it down and provides clues, it can sometimes either not provide the critical clues, or can even be misleading. Yes, sometimes adding more sensors focused on the issue may help, but not always.
Sometimes it is just down to the hard work of finding the peculiar circumstances and interactions that cause previously unknown failure modes.
Been there, done that (UAV & motorsport systems), seen lots of very smart & experienced engineers stumped for way longer than you'd expect.
I'm having a difficult time imagining anything "cheap" about a complex sensor diagnostic system integrated into a defense aerospace platform with big daddy ACAT 1D program status whose acquisition strategy revolved around cost reduction.
Ultimately the issue here is budget and time. With aircraft there are TONS of variables: temperature, pressure, humidity, vibration environment, dust. Even once you THINK you know all the variables, reproducing the problem in the lab might not happen--then after hundreds of hours of testing you're still at square one.
Once the problem was diagnosed, fielding the solution is / was another challenge. Some of these aircraft are in pretty remote areas.
These aircraft gained popularity and rapid adoption because they're substantially less expensive and VASTLY quicker to manufacture than any other option. Almost all customers prioritized cost and acquisition speed over long term reliability, which of course comes with a certain amount of risk. Testing and certifying an airframe to standards is expensive and time consuming [2] but has the advantage of improving reliability.
Also, this is just hogwash journalism.
> Last year, the Army reported four major drone crashes, each involving the Gray Eagle — a model identical to the Predator.
Warrior / Gray Eagle / MQ-1C [3] is a 40% payload increase over Predator / MQ-1 [4] and the airframe was almost completely redesigned. Hell, the engine is even different. It's a different aircraft.
[1] https://en.wikipedia.org/wiki/General_Atomics_Aeronautical_S... [2] http://www.ga-asi.com/certifiable-predator-b [3] https://en.wikipedia.org/wiki/General_Atomics_MQ-1C_Gray_Eag... [4] https://en.wikipedia.org/wiki/General_Atomics_MQ-1_Predator