> Apple has begun designing its own servers partly because of suspicions that hardware is being intercepted before it gets delivered to Apple, according to a report yesterday from The Information. "Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter," the report said. "At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there. Building its own servers with motherboards it designed would be the most surefire way for Apple to prevent unauthorized snooping via extra chips."
> At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there.
I have done this before, and we actually found an unspeced part! Thankfully, it was not from a malicious state actor, but just one supplier being creative and not telling anybody. Especially if you don't have an iron grip on your supply chain, you have to be vigilant. As a manufacturer, there are more problems to watch out for than espionage.
This is more common than one would think. I've done something like this several times before, too -- not taking photographs, but reviewing the actual board against the layout and specs. I found unspecced parts twice; the first time it was pretty innocent, just a couple of 0-ohm resistors that weren't marked on the schematics (understandable from ODMs who want to reuse the same design but annoying to debug when it goes wrong). The second time it was a pair of clamping diodes that should have been there from the very beginning (I don't know how it slipped by the initial schematic review; I wasn't working there when it happened). They weren't on any schematics, and when we requested an up-to-date BoM, they were tucked away under another set of diodes, despite being a different part.
I also found parts that had been changed without notice (one of which had the potential to be tons of fun because it was a crystal oscillator with a far worse tolerance than the original).
When the supply chain itself, the management effort and the handling of the supply chain gets so large that it's done almost completely overseas, by a whole team of different people, under constant time pressure and in various degrees of partnerships with other companies (not just those who sell the supplies), these things can slip between the cracks surprisingly easy.
Not necessarily for that, but it does routinely happen that an ODM uses the same PCB design for multiple projects. This results in various chips remaining unpopulated, 0-ohm resistors used here and there to route pins to the right peripheral and so on.
As for stupid changes, while I don't remember the details now, I definitely remember drafting at least one schematic that supported accessing the same peripheral in different ways (or something of this type?) because we couldn't figure out the best one (or the right one?) from the datasheet. It's definitely the kind of thing that I'd rather not see in a final design, and which I'd iron out in a subsequent revision, but I suppose if you work under the consumer industry's tight deadlines...
Oh, and of course, some PCB traces literally don't lead anywhere in the connection sense. E.g. guard rings aren't there to connect electronics together. I suppose it wouldn't be hard to mask some malicious connections that way.
Oh yeah, since I started scraping electronics I see how often boards are designed for multiple price points :) there's indeed a lot of place to toy with.
You know, it would be possible to insert a small component between pcb layers too (during pcb manufacture/lamination) if agents were that much determined.
This doesn't really mean anything, though. For all we know the "person familiar with the matter" is the same source Bloomberg used in their report. There's still not acknowledgement from Apple in 2016 that anything like this was actually happening.
If you have all the resources of a state actor to accomplish this, it’s owuld not be a chip on the motherboard, it would be a set of circuits in the motherboard.
Why make something easy to photograph when you can embed it an area that can only be seen in an x-ray?
Altering the flash chip would be too obvious. It's a textbook 101 supply chain attack...
Looking at the flash image (dumping it) or chip (x-raying it) would be the first thing anyone would do if they suspected something fishy. A tiny SPI man-in-the-middle chip sandwiched between the PCB fiberglass layers is a lot more discrete and more generic (same MitM chip fiddling with transmitted bytes can attack many different flash platforms, regardless of the sizes/pinouts/footprints of the flash chips).
Reading this makes me understand the decision about them licking down on repairs a little more. If a Mac won’t boot because of tampering (repairing) then it essentially solves this problem. I’m a little conflicted if it’s the case as I think we have the right to repair our own devices but distrust of any state actors (locally and internationally) is also pretty high.
As an ex-apple employee, all I’ll say is this means basically nothing. Everything was on a strict need-to-know basis and a condition of your employment was respecting that. I would be very surprised if more than a handful of Apple employees even knew what exactly what was purchased from Supermicro, so a random sample of employees absolutely would know nothing about this. Unless senior means Senior Vice President, it’s meaningless.
That being said, SVP level people did categorically deny it, and I can’t see them doing so unequivocally unless they really believe that will hold up on the court of public opinion for their entire tenure there.
Having worked at FAANG companies my experience matches yours. There are people doing good work in the information security teams, but they only tell you about attacks much later, with most of the details removed. One of the assumptions is that a certain (small) percentage of engineers are also agents of state-level adversaries.
>A senior security engineer directly involved in Apple’s internal investigation described it as “endoscopic,” noting they had never seen a chip like the one described in the story, let alone found one. “I don’t know if something like this even exists,” this person said, noting that Apple was not provided with a malicious chip or motherboard to examine. "We were given nothing. No hardware. No chips. No emails."
Not once have we ever seen a tiny grain of sand sized malicious chip on the motherboards we bought that I've seen in the racks I inspected in the datacenters I had access to!
Probably easier to build “smart” Cat5 cables that draw PoE to energise a chip in a connector to power a long-range antennae built-in to the cable (outside the shielded layer, of course) to broadcast whatever goes on inside of it to the outside - and put 500x of them in a cable box delivered right outside an Apple iCloud data-center under the guise of a typo’d purchase-order from Monoprice or Tiger Direct.
Why? Both the Israeli Intelligence Heritage museum and the Vault 5 leaks specifically show hardware implants that are intended to be inserted into cables like display, HID and USB cables.
Building a network tap into a CAT5 cable isn’t that hard you just need to essentially modulate the traffic into RF and have another implant near by that can intercept the radio signals.
We live in a day where we have demonstratable side channels attacks against RSA keys by listening to how a laptop squeeks when it’s under load and you think building a chip into a cable is a stretch?
Heck I have one cable like that atm which is a converter from a model M keyboard to a USB you can’t see the converter IC it’s built into the RJ45 connector the keyboard originally used without any additional bulk.
And this is likely not the only one the NSA has :)
It’s very easy to tap pretty much any electrical bus these days which uses a cable the tap itself can be very basic as it’s completely passive.
I’ve seen a demonstration of similar taps on VGA cables that transmit the entire image to a remove reciecer which is often implanted near by in a power socket, light fixture or anything else where you have a reliable power supply and enough wires to hide and even transmit a signal out side over the power lines.
You don’t want to do that since those broadcasts are detectable not to mention that with PoE power levels you will still not be able to broadcast outside of a datacenter also you can’t build an IC that can even take leverage of that power while being hidden in a cable.
All of these taps rely on external transceivers that will either record the traffic for later extraction or exfiltrate it through other means.
Also while PoE has more power it’s also more sensitive to voltage drop over the line which means the tap it self will be detected while normal Ethernet works from -+0.5v to -+2.0v without any issue the voltage range is to allow for voltage drop over longer cables so any drop would be ignored by both end points as they’ll just assume the cable is a few meters longer.
> That being said, SVP level people did categorically deny it
Again, as someone who worked directly w/ executive level people (and lawyers), the denials mean only that the statements were cleared by lawyers and probably corporate communications staff. (Obviously, Elon @ TSLA's an exception.)
After reading the story and the discussions here, I reviewed some of the statement language. There's lots of wiggle room.
All is further complicated if US gov't agencies etc., are concerned. Rules and regs that the average person thinks will apply won't. This is just how it works.
> Unless senior means Senior Vice President, it’s meaningless.
And not even then.
I suspect if someone did find something like this, they were told to bury it for plausible deniability.
"Okay, crap, you found a hardware security breach. We'll tell people inside to quit buying those servers, but we'll cough up some other reason. Don't breathe a word of this any further."
>"Seventeen individual sources, including government officials and insiders at the companies, confirmed the manipulation of hardware and other elements of the attacks."
Is it possible that many of these sources received their information from others on that list of 17, propagating imperfect or inaccurate information? I'd assume it's standard practice for journalists to confirm that multiple sources aren't essentially from the same source, but this doesn't look great.
Terminal sales are the metric Bloomberg cares about.
SuperMicro was delisted from Nasdaq in August [1] after failing to meet its reporting reauirements “amid an ongoing audit committee investigation” [2]. This is a name already receiving attention from the analytic parts of Wall Street.
When you drop a bomb of this size you'd better double check that. This case is exceptional and I'd like to believe it was treated accordingly by journalists.
Look I worked at a company that built boards in China. Every board is xrayed to verify every level of the board for every trace. They are matched vs. known good perfect board. If anything is wrong the board is destroyed. The boards I am talking about where complex 26+ layer boards which is way more then any standard motherboard. HW wise this is not impossiable, just improbable. The better method would be in software, replacing the on board system management software (intel ME) for example with a compromised version. That is very doable.
I would think it would be much easier to validate software via simple hashing than physical hardware, via x-ray. Sure, you can verify traces, etc, but with current lithography at 14 nanometers, I have pretty much no doubt that there is no economical way to validate tens of thousands of meter-long boards.
You could see every trace on the board at each layer. A chip like this story talks about would standout. Also at each point on the board you could probe (traces) end to end. It’s complex. Also the is the integery testing .. a machine that has 1000s of needles that pushes down on the top and bottom of the board at each contact point and test the resistance and conductivity end to end. Put something in the board in the path and the numbers come back wrong.
Signal integrity is really important as it can lead to grey failures down the line. It is really important to find them before you stuff the boards with $$ of components that you can not save if the board is bad.
> A chip like this story talks about would standout.
Sure, you could see it, but to know it was wrong you'd have to have a non-compromised board to compare against. Or knowledge of every design and supplier decision, which Supermicro/Apple do not have.
I absolutely agree. Putting a chip on the mobo seems the worse way of hacking into the hw and the supply chain. Replacing an existing part with a doctored one or patching the sw seems so much simpler.
Bloombergs "chipgate" fails Occam's razor and this whole story is losing credibility by the hour.
Yes, hardware implants would best be done by swapping out a BMC/IPMI/DRAC/ILO chip, eg, modifying it upstream in the supply-chain. This type of implant would be much less geo/politically-damaging, if discovered, and more difficult to notice (unless the client checks integrity of all Flash EEPROMs) and more difficult to track down.
I'm not saying it's a fake story, but the US is in the middle of a trade war. It could be a huge piece of propaganda. The sheer audacity of that though would be staggering.
If it was fake who decided that SuperMicro (down almost 50%) should be sacrificed for a trade war? Seems unlikely.
If the story is not true, occam's razor would suggest the journalists just got it wrong, maybe by turning a molehill (couple of hacked servers or server firmware) into a mountain (industrial scale espionage) .
What makes you think it would be the U.S. pulling the strings? In trade wars you want the enemy to sink their funds. What could be better to do that than to force every U.S. tech giant to audit their hardware?
Correct. I've expecting this to happen for years now, actually thought it odd that so much outsourcing was allowed at all.
People should look what was deemed critical infrastructure and manufacturing during war time (e.g. WW I & WW II). It isn't particularly difficult to understand why certain companies were/are continually bailed out.
In the parent's scenario, the initiator would be China. How would you make China pay the bill for the audits? By achieving the goals of the trade war, which was supposed to happen even without the audits? If China really pulled the long con like that, they'd win this round. But highly doubt they'd do that long con, there's too much to lose in terms of attack surface, like exposing your zero day to the world without using it first.
China has far more to lose here. Companies losing money to cycles spent auditing hardware is trivial compared to companies permanently losing business due to loss of trust as a part or the supply/manufacturing pipeline.
I'm not sure you're following this thread. Let's look at the whole conversation again.
Jedi72: Posits possibility that the story could be fake and planted by the US as a propaganda tool for their trade war.
steve19: Posits reasons for why it's not likely to be a propaganda lie planted by the US.
inetknght: Posits that it could be China that planted the fake story for the sake of making US industry waste funds (again, this is all still a hypothetical conversation based on Jedi72's original contemplation of the possibility of the story being fake).
topmonk: Posits that if China planted the story, China could lose reputation and should be given the bill for the audits. It's possible that topmonk misunderstood inetknght and was in fact referring to the US. But I'm assuming that topmonk understood inetknght to be referring to China and so posited that China should be left with the bill if the story turned out to be fake.
Me: First, questions how the US would make China pay the bill. Second, notes reasons why it doesn't make sense for China to start spreading misinformation like this (again, going on Jedi72's contemplation of the possibility of the story being fake). Ironically, your reply actually bolsters my original contention against topmonk's comment. It makes no sense for China to want to plant a fake story like this.
> Ironically, your reply actually bolsters my original contention against topmonk's comment. It makes no sense for China to want to plant a fake story like this.
I think you aren't understanding my comment, and you confused ra1n85 with your response to me.
It's strange that you think I believe that it would make sense that China would plant a fake story. As you wrote yourself, "topmonk: Posits that if China planted the story, China could lose reputation and should be given the bill for the audits."
I actually said "An argument could be made..." I meant that if there was a trade negotiation going on, the USA could bring this up as a way of trying to force China to give up something else.
But, back on topic. I agree with you, as you said, "It makes no sense for China to want to plant a fake story like this." Can you tell me where I suggested otherwise? I reread the whole thread and I can't see why you'd think I thought that China might have planted a fake story, or it would be a good idea for China to do so.
I understand you now. Sorry, it wasn't clear to me before. Looks like we're actually completely on the same page. I got completely twisted in the logic.
Ignoring ra1n85 (it seems the 3 of us are probably on the same page), my original question for you was how would you make China pay the bill. You say that the US could use this as leverage to force China to give up something else. I'm not confident that's great leverage by itself, but I just thought of something that may answer my own question.
IF China had the gall to plant a fake story, I would imagine it would be a sign of weakness. There's no reason to plant a fake story if their bargaining position is strong, so a fake story plant would be only the craziest of Hail Mary options, a poison pill that acknowledges, "hey, we're gonna lose, how can we cause the most damage we can before we go down completely?"
So if the US was able to confirm a fake story plant by China, they'd have more confidence in going full court press and getting everything they want because their bargaining position would just be that strong (discounting actual war).
I suppose discussing all the hypotheticals for what happens if China plants a fake story is getting out of hand and not worth all the typing....
If it had been used first, then the story would be true, not false. Jedi72's original thought experiment is no longer a thought experiment, and there'd be no point in asking what are the implications if the story was fake, nor who would have planted the fake story.
The fact that it came out at the “right” time, doesn’t mean it’s false. Previous propaganda ops (uranium from Niger, 45 minutes etc) were much flakier while the stakes were much higher.
I think it’s just a case of national-security actors sending out a message while leaving FAANG with enough plausible deniability to avoid tanking the whole market. As long as it’s semi-official, the only victim will be Supermicro, and everyone else will have received a message that they should pay more attention to their supply chain.
I don’t buy it. I know it’s fashionable to be cynical about the media at the moment, but Bloomberg is not a rag and it’s owned by someone who is no fan of Trump or trade wars. It doesn’t add up.
It would be something Russia would do, plant a fake story to massively discredit Bloomberg. It would go with their strategy of launching so many fake stories that nobody knows anymore what to belive.
The problem with this theory is that Bloomberg says all the sources were from the American govt.
If there was hidden hardware on a bunch of servers, where is the hardware now? Why doesn’t Bloomberg’s sources have the hardware or know which boards they were?
Bloomberg provides zero evidence this happaned, outside of their anonymous sources.
> Why doesn’t Bloomberg’s sources have the hardware or know which boards they were?
How do you know Bloomberg's sources don't? They're anonymous, and while they might know about the implants in detail, they may not have the authority to take examples on a public dog and pony show.
Also, if they want to keep their anonymity, they probably have to be careful about what gets released in order avoid exposing themselves. For example, if you have a limited-distribution report you want to leak info from, leaking a summary of the report is a lot safer than leaking the report text itself. At a minimum, the latter narrows down the leaker to someone who had physical access to a copy.
> This is as stupid as saying Xinhua/Caixin have anonymous sources with solid evidence showing Trump is from Mars.
That tone is pretty uncalled for. The Bloomberg story may or may not be completely accurate, but it's fairly detailed and plausible. While you may categorically distrust anonymous sources, it's not stupid to think they may sometimes be right and that you can trust reputable journalists to vet what they say a fair amount of the time.
This story is still young. I wonder what other news organizations can find out about it (beyond the press release responses).
Also worth noting that Bloomberg has demonstrated its willingness to put journalistic integrity ahead of profits in the past. They’re currently blocked in China because of a story they ran years ago about the business connections of the country’s top leaders.
> that story a few years ago was backed by solid facts, independently confirmed by multiple sources. what bloomberg has this time?
Again, how do you know this story hasn't been confirmed by multiple sources and isn't backed by solid facts? IIRC, Bloomberg claims they confirmed details with sources within the US Government, Apple, and Amazon. Apple and Amazon have issued denials, but its quite possible those denials may have been lies or the people who made them may not have had all the facts.
Reputable journalists don’t base an entire story on anonymous sources: you use anonymous sources for background, you don’t use them as primary sources. If they don’t yet have anyone on record, then they shouldn’t be publishing stories until they do. Deep Thoat wasn’t the only source for Watergate: he was used as a means to obtain further evidence and sources.
How do you fact-check anonymous? Plausible has nothing to do with it. Plenty of things can be plausible, but that doesn’t make them even slightly true.
Assuming credibility for an anonymously sourced story is a folly, especially when the allegations are both market-moving and completely unverified. It’s irresponsible. They should have held the story until they had verifiable info.
Anonymous sources are only anonymous to readers, not the journalists behind the piece who do know the identity and vet the information before publishing. That's standard operating procedure.
It doesn't mean there are no dishonest journalists and made up sources, but assuming a source is real it is never without any verification at all.
Apple says they provided denials of this to Bloomberg before they posted and the article doesn’t (or didn’t when I read it) reflect that at all. The author chose to exclude that information. Or Apple is lying.
Anonymous sources without corroboration or other verification might as well be made up. We could literally write anything with “anonymous sources,” but unless there is some other validation or evidence, it’s Shrodinger’s Cat: it is equally true and false.
You do have validation: that Bloomberg has in the past been reliable and that it would be harmful to Bloomberg to publish such a story falsely. It's literally what editors are there for.
Certainly I might not believe "briandear" writing an article with only confidential sources (and really, that should be the term, not anonymous), but you don't make your living by being a reliable source of news.
My thoughts exactly. They're making claims that virtually everyone in tech is denying and haven't/can't produce any evidence.
Not to mention, if this hardware had been trying to phone home, it's safe to assume it would have set off some kind of an alert at at least one of these places.
"...let us consider a hypothetical. What if:
1. Everything in the Businessweek story is true, Chinese spies planted hardware backdoors in computers built and used by major American companies, and the FBI investigated along with those companies and discovered the backdoors.
2. It is a national-security secret and the companies were instructed by the FBI never to acknowledge it.
3. The companies are patriotically but falsely denying the hack."
If it were Apple, they wouldn’t write a categorical denial because once the “truth” leaked, their credibility would be shot for a long time. The standard Apple answer would be “Apple could not be reached for comment.”
The people hypothetically demanding these denials have gone literally thermonuclear before. It’s an entirely different ball game when you deal with the guys with machine guns.
When you read the article, I believe they are alluding to the fact that Apple and Amazon did discover the vunerabilities.
“In 2016, Apple informed Supermicro that it was severing their relationship entirely—a decision a spokesman for Apple ascribed in response to Businessweek’s questions to an unrelated and relatively minor security incident.”
> Not to mention, if this hardware had been trying to phone home, it's safe to assume it would have set off some kind of an alert at at least one of these places.
Maybe at some big companies, but not anywhere I've worked. I hardly know anyone who audits outgoing traffic with dedicated hardware.
I wonder if there is some magical market cap boundary beyond which companies stop being grossly negligent. We know it's over 200B as Intel somehow never bothered fixing their products for decades, let's hope five times that is big enough.
The original article specifically says that they saw odd network behavior and issues with the firmware. If this is all true (?), that's a piece of how they found it.
even when vice is covering it up? there are nuances everywhere. if you don't trust the reporters/editors to be accountable, the whole system of news reporting falls apart
Imagine that the boards actually do exist, and represent an espionage by a foreign government. Why would the company not immediately hand it over to the FBI/CIA, and immediately be forced to sign an NDA?
> Bloomberg provides zero evidence this happaned, outside of their anonymous sources.
Bloomberg probably ran this hoping that now that people are looking, some folks outside the circle of anonymous sources will find the chip so that they don't risk exposing their sources.
Publishing fishing expedition stories is irresponsible journalism. Bloomberg is a market-influencing outlet: printing unsubstantiated rumor is unethical, especially given their influence.
It's not a fishing exhibition if you have a bunch of sources from different directions all corroborating the story.
The story is so explosive that I find it very difficult to believe that Bloomberg isn't on very solid ground.
Nevertheless, getting hold of irrefutable physical evidence may be very difficult. By breaking the story, they now have lots of people now looking for that evidence.
In addition, they may now have enough cover to be able to actually present evidence in their possession and claim that it came from an outside source in order to protect their sources.
There is no good reason for Bloomberg to lie about this as it will significantly damage their reputation and bottom line if proven false.
Now, that doesn't mean that Bloomberg wasn't the target of an operation and was given planted, false information to trace leaks. However, as this has been in the playbook very recently, I would expect the press to be on guard for this.
How many people at Apple or Amazon have the ability to steal compromised hardware and surreptitiously hand it to a journalist? That seems like a pretty lofty expectation.
I believe the Bloomberg story. Why? Because of the fact that the chip was originally found on hardware owned by Elemental. Elemental would have been a great company to target.
My guess is that Elemental was specifically targeted because the cost of doing so would be pretty small and with nearly a 100% chance of success. Back in 2015, Elemental was nearly guaranteed to be acquired by one of the greats (Apple, Google, Amazon, etc.) because they had grown too large to be acquired by smaller companies but were also unlikely to go public on their own. The company was doing very well, plus they had government clients.
Knowing that Elemental would likely be acquired and infecting their hardware beforehand would have been pretty sophisticated but also an easy thing for a malicious party to do. Even if the hackers didn't know/plan for Elemental's acquisition, they still would have been a great target based on their government work.
(I'm not trying to fault Elemental; I would expect the same thing to happen at basically any small company that employs maybe 10 hardware specialists)
And if the story were fake, why would Elemental even be mentioned? It's too small and obscure to be of note otherwise.
> My guess is that Elemental was specifically targeted because the cost of doing so would be pretty small and with nearly a 100% chance of success. Back in 2015, Elemental was nearly guaranteed to be acquired by one of the greats (Apple, Google, Amazon, etc.) because they had grown too large to be acquired by smaller companies but were also unlikely to go public on their own. The company was doing very well, plus they had government clients.
I don't think the acquisition potential had anything to do with it being a good target. It was all about the government clients.
I could see the acquisition potential as actually being a downside. Apple, Google, Amazon, etc. have histories of acquiring companies just to withdraw their products from the market.
I know Bloomberg isn't going to, and shouldn't, give up its anonymous sources... but it feels like Bloomberg's going to have to provide a lot more specifics if these reporters (and Bloomberg itself) are going to maintain their credibility -- concrete facts Apple can directly confirm or deny, as opposed to leaving Apple to guess at what it could be.
E.g. if Apple contacted the FBI about this, then who at Apple did so (or at least what was their role), on what date, and what FBI office? Or how did Apple detect it in the first place, what happened next, etc. Even if sources can't provide technical details, they should certainly be able to provide names and dates.
> E.g. if Apple contacted the FBI about this, then who at Apple did so (or at least what was their role), on what date, and what FBI office?
I'm pretty sure there is an open line between Apple and the FBI for these exact risks. Else I don't see how these cases do not get investigated multiple times in parallel.
Lost of news reports result in strong, aggressive denials. Bloomberg assembled extensive evidence, with around 17 sources. Perhaps Apple needs to provide more evidence than just a denial (though it's hard to prove a negative).
So anyone can just claim anything now and it is up to the person being accused to prove their accuser wrong? We're just going to assume guilty until the accused can prove themselves innocent? That is not a world I want to live in.
I'm pretty sure US law would protect Bloomberg against any lawsuits unless Apple could prove that they definitely knew the story was false when they ran it. Short of that, nothing - not even massive journalistic failures - would make them liable.
They don't have to know it was false, only be reckless in trying to verify whether it was true or not (which would also be hard to prove, but not as hard).
No, they have to provide evidence. Evidence is what distinguishes legitimate assertions from 'just claims' in everything from science to law to reason and rationality in general. Bloomberg assembled extensive evidence.
I have 10 Supermicro machines sitting in the room next door, bought through the years. If someone would tell me where to look I'd be happy to tear them down, but without any specifics, such as the serial numbers or SKUs of affected hardware it seems a bit thin. Though the hack itself sounds totally believable, compared to Van Eck Phreaking or powerline exfiltration it sounds pretty easy.
I've got an X10SAE and X9SRA I could check, but I would be surprised if the same thing targeted the consumer market. It seems like looking for redundant SPI flash and/or unpopulated/half-populated footprints would be a start. Although I've got to wonder if the implant was really using a redundant footprint for the flash, why it wasn't just in the appropriate package rather than the custom jobber Bloomberg implies.
Frankly I've got to reread that original article. It gave me a headache with the continual reiteration/illustration of just how small the implant was, and other anti-informative cruft. Wait until they find out about the size of transistors inside CPUs...
SPI works by paralleling all of the shared lines, and each chip having its own CS line. So you can't really enumerate like that, without already having enumerated the CS lines.
I2C works closer to how you're thinking, but even there a hostile implant doesn't need to have an protocol-dictated address to corrupt someone else's traffic.
I'm fairly amazed by the amount of skepticism here. A story this big is certain to get a lot of people looking into it. It would be pretty boneheaded for them to run a story like this with no evidence.
I suggest the skeptics keep an open mind, instead of categorically denying it could be true, just because a couple of for-profit companies don't want to see their stock plummet the way Supermicro did. Nothing reported so far is out of the realm of plausible, considering the value of a successful supply-chain attack against tech companies.
I am also shocked. Especially hearing the same BS talking points I see right now in US politics attacking credibility of news. No, reporters don't make up sources and an anonymous source reported by a big institution !== 'might as well be made up' as someone already commented in this thread.
I would love to see some research on accounts and comments on HN similar to Twitter analysis post 16. Seems to me any time China is broached the HN thread gets more comments than average. Many posts read to me as strongly defensive or taking straw man/obfuscation type tactics.
But then again that could be personal bias I don't know the actual human composition of HN comments - which is why I would love to see some research on HN comments/accounts.
With Iraqs WMDs stories the officials, not the reporters, lied. Also there were quite a few stories, backed by anonymous sources from the intelligence apparatus who were very critical about the official line and the pressure that they were put under to produce corroborating evidence.
> I am also shocked. Especially hearing the same BS talking points I see right now in US politics attacking credibility of news.
Tons of news outlets abandoned credibility in the Trump era.
This is either going to turn out to be an NSA gag order, or a total misunderstanding on Bloomberg's part. For me, this is the most interesting news story to follow in a decade.
The response is equally interesting. I wouldn't have assumed people would be so quick to jump to "Well, private US companies are lying to the public because the US government is compelling them to."
> I wouldn't have assumed people would be so quick to jump to "Well, private US companies are lying to the public because the US government is compelling them to."
I mean ask Joe Nacchio how going up against the NSA worked out.
Since everyone assumed IPMI was crappy and potentially backdoored, that's why the story seems fishy. Why go to science fiction lengths to subvert some easily subervertable thing?
I suppose I meant "presented as science fiction". I formerly worked as a hardware design engineer so I'm pretty familiar with what's actually science fiction ;)
New theory: the actual "attack" was a production mistake. E.g. an active component meant for a different product was accidentally loaded onto the PnP machine for these SM motherboards in place of some passive component. The changed component happened to trigger unexpected behavior in the BMC (e.g. put it into a TFTP firmware load mode sometimes). Those inspecting the hardware persuaded themselves they were witnessing a sophisticated attack, because that's what they expected to find one day.
Not really -- I have used SuperMicro servers in the past, and we had our own Super Micro part numbers based on the company -- so, they know who is ordering each piece of equipment and they know where each piece of equipment is being built and sent to.
If you're just ordering boards through a reseller, I wouldn't expect those to be infected, but, when you're ordering 10,000+ servers at a time, you'll get your own Part Numbers, your own specs, and your own build times/specs.
Or a truly massive act of trolling by Anonymous members inside government and Apple/Amazon, planting false information to sway politics, or just to prove that they can.
Somebody in another thread discussed the hack as a brute force strategy where the attackers compromised a lot of hardware without knowing where it would end up. If that’s the case, then I’d imagine that you could audit a bunch of this hardware from various places in the wild and see if any of them have the “extra” hardware. My understanding of the situation may be mistaken, however.
They managed to write 30 breathless paragraphs about how the UK police were inexplicably treating what they claimed was the Russian murder of a key scientist in the Litvinenko investigation as a suicide, and that the police had mysteriously testified that “no-one in his family seemed particularly surprised he had taken his own life" even though some of his relatives had suggested foul play, before they thought to mention that he'd been showing signs of depression for some time, his wife said he'd tried to kill himself the week before, and there was no signs of foul play and no evidence anyone else was present. They then argued this shouldn't have ruled out foul play because he could've been given some kind of secret Russian mind control chemicals. Seriously.
To make matters worse, from what I can tell he didn't even play the key role in the Litvinenko investigation that they claim he did. He merely recalculated Litvinenko's exposure in 2010 after it was discovered he was exposed twice rather than once - several years after all the announcements and further investigation the Buzzfeed story portrayed as a direct result of that work, and well after this had been clearly pinned on Russia. The original analysis was done by different scientists who are presumably still alive.
That's exactly his point. He's saying they always wanted to be legitimate news, but didn't have the money, so they built their war chest using clickbait.
This story is good news regardless of who is right.
Even if this attack actually didn't happen, you can be damn sure that the tech giants now massively will intensify efforts to prevent hardware hacks like this will ever happen to them.
Wow, didn't realize that is based in more then one year long investigation by Bloomberg. That seems to be quite thorough and am I already curious when companies will come forward with details.
Say you work for a company that uses SuperMicro boards on their servers. Is it possible to inspect them for the hack or you could have no way of telling if they're tampered? Because if there is then I guess in the following days we'll have confirmation from third parties.
Of course it makes sense for China. Assuming Bloomberg is right, China still is pretty much a manufacturing monopoly, the production isn't going to move any time soon.
Why are so many people ignoring the fact that Supermicro was delisted from Nasdaq over a month ago with continued delays and specious excuses from the company. Something really strange is going on there.
One interesting bit of fall out here would be companies moving out of China proof or no proof. One of the startups I worked at had fab in China and San Jose. The devices we sold to the government could only come from the US fabs. There were a few non-gov customers that insisted on the same.
The Apple timeline in the Bloomberg article doesn't seem to make any sense. Apple found an "accidentally" malicious driver on a Super Micro board in 2016, and that caused them drop them right away.
So if Apple found a batch of 7,000 manipulated boards a year earlier, why would that not cause them to drop Super Micro as a supplier?
A government gag order is plausible, but is a government keep-buying-malicious-hardware order a thing?
> So if Apple found a batch of 7,000 manipulated boards a year earlier, why would that not cause them to drop Super Micro as a supplier?
You don't want to tie the two events together. If the article is accurate, the hope would be that by waiting, the could garner support with comments like yours.
e.g. Let's wait 6 months before you do something in response to something today so you can say that this has nothing to do with the even 6 months ago.
In the case of investigating potentially state sponsored espionage I think it's plausible to keep buying the malicious hardware so the investigation can continue as they work their way up the supply chain.
It might be plausible this chip is not for spying, but rather a doomsday switch. It works like a switch which disables the whole system working correctly if enabled by a simple signal. There are many critical paths on the board which can be disabled with that very simple approach. You don't even need a sophisticated IC for that.
Given the impasse, I'm inclined to believe one of two things:
1. US intelligence planted & played along for this story, for a long time.
2. The story is true on all fronts: i.e. those inside Apple with knowledge about this are lying to senior executives under immunity protection from US intelligence/law
Personally, I think #2 to be a lot more likely - US intelligence has managed to sneak in backdoors into tech forever, in cahoots with sympathizers who probably have immunity agreements if outed.
Supermicro underperforms, especially Nvidia continues to win their traditional area of business. There is a small chance Nvidia will acquire Supermicro, but right now Supermicro is still too expensive for a company in decline.
Who, what, when, why, where? Bloomberg needs to tie it to specific SKUs of servers, get shipping records, see where they were sold after three years of use which is when they usually go out of warranty so researchers can get their hands on them.
Part of me wants to believe Apple and Amazon, but they're really under no obligation to tell us the truth. It's way more harmful for them to admit this happened.
In this instance, just to be fair, Bloomberg did not present any concrete facts that can be confirmed or denied by the industry. (Concrete facts would also have the benefit of being able to be confirmed, or not, by reporters not affiliated with Bloomberg or FAANGs.)
As it is, Bloomberg just kind of said, there is this issue that we're certain exists. So the industry is left to guess what the issue is in so many ways.
I think we'll all need to wait for the outside reporters and investigators to run some of this information down to get a better idea of what's going on. Because right now, even most of us are just guessing at what it could be.
They are legally obligated to their shareholders (you), so you could sue them for being untruthful, and the First Amendment says the government can't "tell you not to".
Technical possibility is one thing; proving the story has actually happened is another thing. Until now, what we get is a categorical denial of the story from all related parties. And all the evidence Bloomberg can provide so far is just vague anonymous sources.
Talk is cheap, show me the code/server/chip if they ever exist. Otherwise, the story is just a blunt lie fabricated by Bloomberg serving as a propaganda to bash China amid the Sino-America trade war.
> Talk is cheap, show me the code/server/chip if they ever exist.
Bloomberg is a journalism organization. They'll report and cross-check testimony that there's been a chemical weapons attack in Syria, but they're not going to go there to collect samples of the chemicals. They aren't going to have "the code/server/chip" to show you, and they shouldn't be expected to.
Furthermore, the people who talked to Bloomberg who may have access to the "code/server/chip" are anonymous and may not be able to have too many details released publicly without compromising their identities. Bloomberg may have more details than they have reported, but be unable to release them publicly while respecting their sources' confidence.
Bloomberg wouldn’t, however publish a story about a chemical attack based on an anonymous source. They’d have independently verifiable evidence and not just some dude saying so.
> They’d have independently verifiable evidence and not just some dude saying so.
This is where you misunderstand. An anonymous source isn't "just some dude" who called in "saying so." Journalists, in the case Bloomberg, knows exactly who their "anonymous" sources are.
Yeah. Compare how this article describes its anonymous sources with the Bloomberg one. It makes it clear that their sources were directly involved in investigating the claims Bloomberg made - including a "senior security engineer directly involved in Apple’s internal investigation" - and that their claims were based on that internal investigation. The Bloomberg article just refers vaguely to "senior insiders" who knew about Apple finding malicious chips on Supermicro motherboards... somehow.
Given enough information about an anonymous source, you can figure out how they are. Keeping it as senior insiders is a way to protect the individuals.
The question you have to ask yourself is whether you trust Bloomberg or not. If you trust them, then you trust that they did their due diligence. if you don't trust them, then nothing presented by them will get you to trust them.
I swear, it's like people don't know how investigative journalism works. Anonymous sources aren't anonymous to the journalists.
As for the companies denying this, this wouldn't be the first time they've lied in such a manner.
I'm tempted to give Bloomberg the benefit of the doubt.
Tech companies don't want to be hacked. And if they are, they want to be able to say "we cleaned things up and everything is safe now," not "we were infiltrated several years ago and have no idea what the malware does or even which systems it impacts."
Funny! So if you own a company and I say your company is dirty, there is literally nothing you can do except to admit that your company really IS dirty. Because if you deny, you'd be lying, based on your own logic.
No, but Facebook is already in deep trouble in terms of how the average American views the company. I find it likely that they want to protect their reputation at all costs (including lying to congress...)
Or, you know, it could be the NSA, since we've seen pictures of what they can do to a cisco router, and especially after the big web services made a push for TLS everywhere.
Does anyone more knowledgable know if this must be an at-the-factory thing, or if it's possible to do this afterwards, "interdiction" as the bloomberg story put it?
And now for wild speculation: What if the NSA or other US TLA was behind the hardware hack? While it would obviously require a coverup, I have no idea what leverage the government could have to keep it quiet - that would be a massive 1st amendment violation.
I would believe Bloomberg if they had some detailed reports/demonstration as regards the mechanism of how this attack actually happens, not some nebulous picturing of some vague concepts the reporters themselves seem don't understand.
Putting aside the specifics of this story for a moment: I really hope that we don't enter a new era of tech journalism where we get story after story written by anonymous government sources, because I am about to lose my mind over the constant barrage of reporting in this style on politics.
It's already creeping into business sections, just make it stop.
Some of the sources are said to be government. That same government is forcing anonymous sources by coming down hard on leakers, both inside and outside. That may or may not be what's happening here but it's certainly possible.
Yes I am scared of anonymous sources writing with an agenda that is bullshit as often as not. Not sure when this became the norm in journalism for people to just take everyone at their word just because they are senior and they have an axe to grind but it's quite tiresome.
If it's important stand up and put your name behind a story as a source, everyone just cowering in the corner because they want to keep their careers safe is making things worse, not better.
>Not sure when this became the norm in journalism for people to just take everyone at their word just because they are senior and they have an axe to grind but it's quite tiresome.
Anonymous sources have been around as long as journalism. It’s not like what those sources say is taken as a given, they are heavily corroborated against other sources of information, often documents/records/etc.
To know who is telling the truth you must know what the government policy really is related to NSA exploits and how bold NSA is when protecting secrets.
These things can be verified only when whistleblowers release documents. Snowden and other whistleblowers have revealed multiple lies, including that Director of National Intelligence James Clapper lied under oath.
That said, it's also possible that Riley & Co. rely on bad or unreliable sources.
This is exactly what happened when both the US intelligence community and media decided that Saddam Hussein had weapons of mass destruction in the 00's. All the sources traced back to a handful of Iraqi dissidents who had made the story up to encourage American intervention.
It's evidence that US intelligence community sources can be unreliable. That's directly related to this story because Bloomberg claims that many of their sources are from that community.
It's kind of weird to cite George Tenet in your claim since he's the one who famously told Bush they had a "slam dunk case" to convince the American people there was justification to invade Iraq.
1.The intelligence flaw is a high order lie in order to mislead public to believe the war is launched by wrong information. It's not. The intelligence flaw is irrelevant or is intentional. The war is predetermined and flaw intelligence has nothing to do with real cause of the war other than a cover afterwards.
2.The narrative that the Bush Administration launch Iraq war with flaw intelligence is another lie that the majority of public who support the war were not responsible because they were misled by their leaders. The war is launched by not only Bush administration but also UK. The main stream media were not orchestrated by government like a totalitarianism regime can do. For example, CNN interviewed a famous Iraq nuclear scientist again and again to sell the impression to the public that Iraq DO have nuclear weapon. CNN independently promote the war which happened to match the government agenda. US/UK are democratic countries. Both countries collectively(meaning enough portion of the people ) decide to over thrown Iraq regime
"the US intelligence community ... decided that Saddam Hussein had weapons of mass destruction"
I hope you realize this was all an invention of the US intelligence community — they knew all along it was bullshit. It's well documented at this point.
The other bullshit that has never been substantiated is the narrative that Saddam was working in collusion with Bin Laden. Despite the latter being quite vocal about his derision of Saddam's secularism and suppression of the Shia. But ask any average American today and "of course" they were in cahoots.
In that case the government led the charade, from claims of 'nuc-u-lar weapons' (as Bush pronounced it) to the claims of buying Yellowcake from Niger, the government led it and the media just went along.
In this case the media is doing it themselves from the start.
Far too often people mistake correct guesses with having actual knowledge. If I had a dollar for every time I've heard somebody be "100% certain" of something I'd probably have a lot of money by now, although I don't know for sure.
The aluminum tube claim was pretty obviously bogus. The tubes were narrow in diameter, variably finished (often rough inside), thin walled, and would result in laughably inefficient uranium hexaflouride centrifuges. It just didn’t make any sense to anyone with even a casual understanding of enrichment methods.
They also made perfect sense for missiles/rockets.
The chemical weapon claims were believable. I mean, didn’t we help Iraq manufacture chemical weapons during the Iran/Iraq war? Wasn’t that long-suspected belief later confirmed?
So, yes. I knew at the time, and I said so. I wouldn’t fundamentally have had much of a problem with invading Iraq (one could have that argument), but the justification, timing, and prioritization didn’t really make sense. Watching Powell pitch that goat rodeo was pretty sad.
You might find this a good read about knowing the iraq wmd story was a lie in advance. From someone who predicted it, they explain their reasoning. It's rather convincing.
It was very well known that G. W. Bush was itching for an excuse to finish what his old man started.
See the PNAC documentation, if nothing else. You don't need to be a card-carrying member of the Illuminati to understand the personal and political dynamics that existed between Saddam's administration and Bush 43's, or to foresee what was likely to happen.
You ain't kidding. Talk about trying to rewrite a narrative with bogus claims.
The 2003 Iraq invasion had nothing to do with warnings from informants about state secrets.
The idea very idea that there was any concern about Iraq's capability to wage war is a joke. Iraq was pretty well softened up by no fly zones and sanctions, so as to be sufficiently anemic, and decapitating the incumbent dictator for life (literally) was mostly just sour grapes for him going off script, and besmirching the sanctity of Kuwait.
It was gloves off for Iraq, as soon as the 9/11 hijackings unfolded. Literally next month people were whispering about Iraq, even though Afghanistan was well understood as the official point of origin for the attacks.
> It was gloves off for Iraq, as soon as the 9/11 hijackings unfolded. Literally next month people were whispering about Iraq, even though Afghanistan was well understood as the official point of origin for the attacks.
Not only that, even though Afghanistan played an actual role, the majority of the hijackers were Saudi. People sort of mention that in passing and then go back to pretending it has no relevance.
The point being that, capturing or killing the associated individuals still alive, to be held responsible, meant transgressing the territory of Afghanistan.
Nonetheless, I'd agree that waging war on Afghanistan, The Country would be just as silly as waging war on Saudi Arabia. It's like Canada waging war on both the United States and Italy, for something The Mafia perpetrated.
Meanwhile, war with Iraq was akin to Canada invading Norway for it's whale blubber, because the Norweigan king sunk a fleet of Danish whaling ships ten years prior, and was now suspected of hoarding a cache of illegal harpoons. Thus triggering a cascade of geopolitical events, whereby Canada stepped in to defend Denmark, thus angering a member of the Gambino family, who subsequently demolished the CN tower, for tampering with Denmark's sovereignty. As if to say that had Norway not attacked Denmark, the CN tower would not have been destroyed by a hijacked train derailment.
Bad US intelligence. Secretary of State Colin Powell gave a detailed presentation at the United Nations that was seen around the world. It was all based on faulty sources.
Big claims require equally big evidence and big sources, otherwise it's he said/she said. I'm doubtful about the story but not against it entirely until more evidence and sources can be presented. If not, it smells like clickbait trying desperately to get traffic, and they just flushed their reputations down the drain.
Also, some technical detail about what a chip with three pins can do. Was it working with another chip? Enabling intelME? There's just not a ton you can do with chip of that size.
Is the story that the board had an extra undocumented chip, or that the chip was used to exfiltrate data?
I'm not saying they can't do anything with a three pin chip, just that this was pretty short on tech details. That's (probably) not the kind of chip you'd be using to parse network traffic on a 10G link and upload secrets. Maybe a key logger, but I don't imagine apple or amazon having a keyboard plugged into very many of their servers.
If they had come out and said, this is what this chip does, this is how it interacts with the rest of the board, or it phones home to x.y.z or something, I'd be a lot more convinced. As it is, the technical evidence is a little light as are the sources. So I think I'll hold on conclusions until we get some more info about what they think the chip was doing.
but we don't know what it was doing, maybe the back door is else where in the motherboard and this things job is to prop the back door open every now and then.
There’s something going on here. I think the denials from Apple and Amazon are strong evidence that Bloomberg’s story isn’t the end of it if it’s even true in the first place, but it’s entirely possible that there are other layers to the other story that explain the evidence that Bloomberg and their sources have seen.
And, regardless of the factual accuracy of this specific story, the overall question of supply chain security, particularly when it entails depending on geopolitical rivals, is an important one.
> it’s entirely possible that there are other layers to the other story that explain the evidence that Bloomberg and their sources have seen.
I wonder if it could be a hypothetical - it's the sort of thing I could imagine coming up in a war games-like scenario (what would happen if China chose to use its access to compromise our supply lines?), and it was mis-relayed to Bloomberg or the documents they have seen (the sources or Bloomberg) do not clearly identify it as such.
Maybe a stretch, but I'm struggling to reconcile Bloomberg's story with the explicit denials.
I'm not sure why this is getting insta-downvotes. While there's no evidence to support it, there could be circumstantial logic to it. One government would very much like to divert focus onto the other, if possible, especially in a climate where they'd prefer to have the issue of election interference trumped by a trade war.
Lots of countries dislike each other. There's no reason to suspect that this is a "Russian misinformation campaign" and suggesting as much is just making things up to push xenophobic anti-Russian conspiracy theories. No different than the "QAnon" and "Pizzagate" conspiracies that the OP is deriding.
It was recently discussed in the news that Russian trolls may have exacerbated drama over The Last Jedi. Which is to say, there seems to be significant interest by those groups in sowing discord and distrust among Americans, not just interfering with the election itself.
I don't hold any given concern against the Russian people, or people who have Russian roots who live here. But pretending the Russian government isn't actively interfering with our media would be intellectually dishonest.
It's too specific to be fake. My money is on Amazon and Apple being under a gag order. Why would they be under such order? Maybe it's CHYNA, or maybe it's PRISM's big brother ;)
If it were a gag order why would they respond with specific denials instead of just saying "We take our customers' privacy seriously, your data is safe, we cannot comment on rumors" or something?
IMHO, if a company said "We take our customers' privacy seriously, your data is safe, we cannot comment on rumors" many people would interpret that as a confirmation of the original story. That kind of lingo doesn't sound specific enough or convincing nowadays. Maybe we're just too used to that. It seems like today's technical audience demands something that sounds very specific, airtight, watertight, germ-proof, shock-proof, etc
This is like believing undercover agents have to admit they're cops if asked. Who even spreads these rumors..
No, Apple would not admit being under gag order. The publicist wouldn't even know, so they wouldn't be lying. Like the PR dude would even have the clearance for that type of info..
People are pitching the gag order theory a lot but would they be legally able to not only flat out deny the events but also that they’re under any gag order?
The only gag order version that makes sense is "An incredibly small number of people at the companies were aware of this, and they've been lying to their own company."
It's possible... but that starts to be fairly chilling and I can't see them being employed after this shakes out if it's true.
As far as I'm aware, legal experts agree there is no way to compel you to lie. Apple's general counsel has said on the record he doesn't know what Bloomberg are talking about, so how would they even verify this gag order was proper?
I personally don't believe Bloomberg's reporting based on Apple and Amazon's strong denial and outright accusation of being misinformed, and that there's no third-party datacenter worker/homelabber posting that they've found this. But I'm glad this story has come up in the midst of the discussion on online voting machines. This kind of hardware manipulation would be much more powerful there and the hack Bloomberg describes is technically possible. The whole story reeks of this XKCD: https://xkcd.com/2030/
> I don’t know what that means, can you elaborate?
A political move where some external country is portrayed as doing something against American interests, without any proof, only to serve a political agenda by lying to their own populace.
There’s an easy way for Bloomberg to prove, or least provide a great deal of support for, their story. Show us an affected motherboard. If this problem was so widespread and they have so many well-informed sources, surely that shouldn’t be difficult.
Since when is it easy to steal corporate equipment walk out from the premises and surrender it to third parties to take photos so that a random hacker news user feels satisfied ?
As Chinese I think this is FAKE news. The people work in GOV in China know very little about technology. The smartest guy here don't work for GOV since those jobs are less paid.
> Apple has begun designing its own servers partly because of suspicions that hardware is being intercepted before it gets delivered to Apple, according to a report yesterday from The Information. "Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter," the report said. "At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there. Building its own servers with motherboards it designed would be the most surefire way for Apple to prevent unauthorized snooping via extra chips."