Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Undersea Cable System Cuts in Singapore (linode.com)
172 points by methodover on March 2, 2016 | hide | past | favorite | 129 comments


I know this is probably a slight tangent to the article, but I is anyone else fascinated by underwater cabling?

My first job out of university is working on the USNS Zeus for Military Sealift Command. I've been out on the ship and we're working directly with the cable handling equipment. I've walked in the big empty vats they fill with the cable. It's over 25 feet deep and 20 feet wide. They actually "walk" the cable up to prevent it from getting tangled. Seriously!! SO COOL!! I would love to post pictures, but I like my job a little bit too much to risk it.


Yea, I was pretty shocked when I learned about this when I was young.

"So you're telling me there's literally cables under the sea that connects continents together?"

http://i.imgur.com/f6V2jab.gifv


Neal Stephenson has a great little non-fiction piece on these cables, both overland and undersea, talking about the process of laying them down and negotiating what countries they go through, etc. It was unexpectedly tucked into the end of my version of Cryptonomicon, but it might be available separately. I agree, I'm fascinated as well.


It was originally published in wired. It is indeed a great write up on the topic!

http://www.wired.com/1996/12/ffglass/


That literally just took me a week's worth of free time to read. Absolutely worth it.

Several years ago I read a short book with a similar theme: The Victorian Internet by Tom Standage.

Another inspiration to me was a 25 minute video about the HVDC lien linking North and South Island in New Zealand. https://hackaday.com/2015/08/19/retrotechtacular-one-does-no...

I wonder if anybody will make a popular game out of estimating cable slack?


At the time that was one of the greatest nonfiction pieces I had ever read about


There is a great book, called "Tubes: A Journey to the Center of the Internet". It's about the plumbing of the Internet that covers pretty much all the physical aspects of what make the Internet including the undersea cables. The author even watched them land a new undersea cable. It's a quick read.

http://amzn.com/0061994952


Any comparable pics you can link to?

Perhaps some PR photos already exist on wikimedia commons.


They're usually not very secret, and you probably just have to search around a bit. Here's the cable bin on an unidentified ship: http://imgur.com/Oc24DNN


I have attached a few photos that were released by MSC. The first link is a general directory for the entire T-ARC fleet. The USNS Zeus (T-ARC-7) is the only remaining cable laying ship in the Navy.

Another cool thing you can do is use the 3rd link to track a lot of the ship! The Zeus is currently underway in Bermuda. It's not on a mission right now AFAIK. If it goes dark you know it's on a mission for a more secure cable line.

I can confirm that the drum linked above is not on the Zeus. The Zeus has two huge drums like that, but they are below main deck. The Zeus also has a cool as fuck hat that I really want, but so far no crew will give me one. A unidentified high officer said he would give me his if I kept the steering wheel for him after it's removal. Sadly, that'll be in the possession of NASSCO/Shipyard and then MSC after the work.

http://www.navsource.org/archives/09/27/27idx.htm http://www.msc.navy.mil/PM4/images/Zeusatsea2.jpg http://www.marinetraffic.com/en/ais/home/centerx:-70/centery...


There's a TV show called "Mighty Ships" that profiles various larger ships (mostly commercial, though some passenger or military). One episode focuses on the Tyco Resolute, a cable laying ship:

http://www.smithsonianchannel.com/shows/mighty-ships/tyco-re...

The full version used to be on YouTube.



In case you never read the famous Wired piece by Neal Stephenson on this topic:

http://www.wired.com/1996/12/ffglass/


Nice map of all cables --> http://www.cablemap.info/


If you find this interesting, you'll probably really enjoy this[0] great 1996 Neal Stephenson article, "Mother Earth, Mother Board".

[0] http://www.wired.com/1996/12/ffglass/


Along the same lines, the classic on Singapore is William Gibson's "Disneyland with the Death Penalty" @ http://www.wired.com/1993/04/gibson-2/

To bring that back to the cable area and forward 20 years, Singapore is a US ally and responsible for a lot of data flows in the SEA region. IIRC Optus, one of the major telco's in Australia, is majority-stake owned by Singtel, major shareholder Tamasek Holdings, which I believe is a vehicle basically controlled by the ruling family of Singapore. Seems the US uses one ally against another.

An associate who runs IP for one of the major wireless providers in .au told me a year or two back he was visited by the authorities and told if he accepted Huawei's bid on their infrastructure that their licenses may have trouble getting renewed.

"Free market" "democracy" at work. Politics and cables ... evil shit. Smiles from mainland China.


To be clear, Temasek Holdings is owned by the Singapore government proper, not by any one political family, though the company's CEO is the wife of the prime minister and it's a vehicle through which the government and the ruling party exert a lot of influence over the "private" sector in Singapore.


I had to close that, otherwise I'd be on it all evening ;) I have zero idea why I find the content so interesting!


I am reminded of an ex girlfriend that I had that once asked me why I liked a particular type of food. She demanded to have a reason for what I liked about it. (Let's say it was steak for example). I didn't have an answer for her and we ended up getting into a fight. I said I just liked it and I wasn't interested in defining what I liked about it and being quizzed on it. It just made me feel good to have this particular type of food.

Likewise I never question why I like big heavy machinery or constructions sites. I just do. Something about it is interesting to me.


Can't remember where, but I once read about research that clearly showed that having to explain why you like something (like a piece of art you selected) reduces the amount you like it.


Sigh. BP can afford to run fiber to their oil rigs, but Comcast can't be bothered to lay it through my neighborhood?


Oil rigs make a little more money than residential neighborhoods.


I thought they were currently operating at a loss?


Plus com-cash has exclusive rights to those telephone poles.


Offtopic joking serious: Be careful about mixing oil rigs and residential neighborhoods, because communities like Los Angeles are forced to deal with health problems and contaminated air and drinking water where environmental controls and inspections are weak.

http://la.curbed.com/2014/7/29/10067206/mapping-all-3000-of-...

http://www.latimes.com/science/la-me-next-door-20160221-stor...


I can say with confidence that they don't run fiber to all their platforms, even the ones that are readily accessible.


no NIMBYs on the bottom of the ocean.


The cost to "make good" the sea floor is much cheaper than the cost to "make good" sidewalks, roadways, lobby community groups and pay contractors to pull fibre. Plus the cost to BP is likely worth it for the reliability over VSAT, vs. what might be a marginal improvement to the ARPU for Comcast.


Not all the cables on that map actually exist yet, like the Hawaiki cable.


Un-tick "Future Cables" at bottom of page


I like the MENA line that connects France, through the Suez Canal, and to Mumbai. We have a constant stream of data flowing through underneath the boats sailing above it? Pretty darn cool.


Check http://www.submarinecablemap.com/ for a more complete map


So many cables going right by Cuba yet only the Venezuelan one connects with the island :(


I'm assuming that's going to change shortly (with normalized relations with the US in progress).


That assumes the Cuban government actually wants Cubans to have proper access to the Internet. The Internet isn't bad in Cuba because of some technical limitation.


Does it exist something similar for cables on land?


I'm in Kuala Lumpur and use Telekom Malaysia Unifi for my ISP (20 Mb/s bidirectional for $US62 per month, for anyone who is curious). This fault has existed for the past few days and is only now being reported.

It not a simple slowdown and is frustrating due to the architecture of the modern internet content hosting and I suspect however TM is rate limiting its customers. e.g.

1) some youtube videos play fine, others stall and simply never load.

2) I cannot play anything on soundcloud.com, plays first 2 seconds then never recovers (however if I switch off wifi and use Maxis 4G mobile connection it works fine).

3) Animated gifs take ages to load and stutter

4) torrents stream perfectly: I can fully utilise my upstream and downstream bandwidth

5) Netflix has no problems (I use a dynamic DNS to get USA content)

6) iOS App updates depend on time of day, during the day they are fine, at early evening peak hours they crawl.

7) Downloading any large file from a website is very slow at all times

It is a pity TM won't purchase/route capacity through an alternative link, having throttled internet for a month is terrible service...


Have you considered a VPN to a Singapore provider. e.g. PureVPN or alternatively PPTP on a DigitalOcean, Linode (maybe not in this case), Softlayer etc.. Generally VPS provider will have better international transit than your local ISP, however your local ISP will peer in Singapore so that part is usually congestion free.


Hmm, interesting suggestion for a temporary fix. I'll have a play around later.

But I'm loathe to add another point of failure in my transmission route, let alone one I need to maintain myself and troubleshoot myself (VPS).


Same here. Home connection has problems, sometimes work sometimes don't. Shared coworking space network is borderline unusable. Annoyingly, TM customer service does not inform that there's issue with cable and they tell you to "restart router for 10 minutes" :x

Offtopic a bit, but (since couple of days ago we've learned about it) Unifi has promotion now, you can change your 20/20 to 30/10 and pay 180rm instead of 260rm per month. Depending of your needs (we didn't need 20 up) you might find it interesting :)


Both YouTube and Netflix usually have local caches, so it's not surprising that popular content plays fine.


From SG here, funnily enough, I haven't encountered any issues. Any sites you noticed with unusually high ping?


Wow, that is better than in Australia. Grr NBN


in Geneva, Switzerland, 100MBit upload & download via Swisscom (but good luck finding server on the other side which would work 100% with it), costs cca 80 USD/month.

in neighbouring France where salaries are maybe 1/3 of Swiss ones, in rather small town which is in mountains, but not some secluded lonely speck but Chamonix, world premiere site for of piste skiing and alpinism (among other activities), its... cca 45 USD for 2 fraking MBits (plus world of pain of dysfunctional bureaucracy when dealing with Orange, main provider in the area - 3 weeks after subscribing via already existing landline, still not working albeit activated on their side).

this world can be a strange place when it comes to internet connectivity...


According to Telekom Malaysia, it will be fixed earliest end of this month: https://www.tm.com.my/OnlineHelp/Announcement/Pages/RESTORAT...


This reminds me of a funny story. I was a tech manager for the Winter Olympics in 2010, allocated to the mountain venues. We had this crazy test rehearsal day where we'd test our systems and processes. Proctors would actually walk around the various venues at the Olympics and just randomly create both big and small problems to see how we'd respond. Suddenly, our network connection to the primary datacenter went down, and we couldn't figure out why. We later learned that a boat had somehow (I don't know how) cut a main underwater cable that connected the mountain areas to the city. It was a stupid day. :)


Are you sure it was a boat? The fibre providing most of the connectivity from Whistler to Vancouver runs along the highway and has been cut by traffic accidents, but I don't think there's anywhere that a boat would kill it...


All I remember is what I was told. It could have been specific special-purpose fibre that was laid by Bell for the Olympics, don't know.


Maybe it's the NSA splicing a fiber with the USS Jimmy Carter.


How does that work in practice, anyways? I mean, cables these days don't just carry a bunch of fibre wires, but also high-voltage cabling to provide power to the repeaters. Isn't it risky / detectable to cut open such a cable?

Also, how is the spliced data actually exfiltrated back to land? The obvious candidate is either a leased dark fiber (which requires intimate knowledge of how the individual fibres are arranged) and laying a second cable (with the obvious disadvantage that cable-laying ships are rare and can easily be spotted).


>how is the spliced data actually exfiltrated back to land?

During the Cold War, they periodically returned to the site of the tap and changed the tapes and batteries. One of the taps was installed on a Soviet telephone cable across the Sea of Okhotsk in several hundred feet of water. It was a 20-foot long contraption able to eavesdrop without penetrating the cable. It was designed to fall off the cable if the Soviets raised it for repairs. It survived nearly a decade, until 1981, when Ronald Pelton sold its location (among other secrets) to the KGB for $35k.

http://www.military.com/Content/MoreContent1/?file=cw_f_ivyb... https://en.wikipedia.org/wiki/Operation_Ivy_Bells


An inflation calculator says it is $100k today. That sounds awfully cheap.


The cut is made 100's of miles away from where the taps are being installed and made to look accidental because installing the taps will disrupt service. When the technicians run the OTDR, a repair ship is sent and the distraction cut is repaired, but the taps have already been installed and sealed back up. Not all fibers pass through places where it can be done on land (i.e. not in US) so this is necessary.


submarine cables are significantly different than terrestrial fiber networks. a fault will be detected in/from both directions, immediately and automatically.

it has been suggested that the USS Jimmy Carter can tap submarine cables without disrupting service or physically damaging the fiber.

did you know when you bend a strand of fiber enough, you can make out the signal and only affect attenuation?


Yeah but still you need to remove the outer casing in order to, well, see the fiber. And that is detectable if you have to cut through a layer conducting high voltage.


Modern cables don't have high voltage to worry about. The signals are amplified optically:

https://en.wikipedia.org/wiki/Optical_amplifier#Doped_fiber_...

It isn't necessary to bend a fiber either. Optical fibers can be tapped via evanescent wave coupling. The fiber cores just need to be near enough to each other over a long enough span, possibly with some removal of the outer cladding.

Notable patents:

"There is a need for coupling light into and from an optical transmission line without interrupting the flow of transmission. Such lines may be used for communications, as well as other purposes.[1]"

1. https://www.google.com/patents/US3933410

2. https://www.google.com/patents/US4097117

3. https://www.google.com/patents/US4377403

4. https://www.google.com/patents/US4763272

5. https://www.google.com/patents/US4336047


Raman amplifiers still require juice to run the pumping laser. There aren't exactly 110V sockets sitting on the ocean floor.

There is a copper tube around the fiber bundle that carries high voltage (usually 10,000+ volts to reduce losses over long spans). The return is usually just coupled directly to the ocean.

If you're careful, use a non-conductive cutting blade, and have enormous balls, you can no doubt cut through the tube on one side and not cause any noticeable effects. At least, if you've brought along someplace dry to work on it...


Yeah it's just not that easy with the advent of coherent optical communications in subsea networks. You need to be able to do some very complex DSP in real-time at 50Gb/s. You also need to know the exact FEC encoding scheme, grey map, actual modulation format etc... It is very non-trivial. For OOK signals I think it's somewhat doable for coherent, very, very hard. Not totally impossible but I highly doubt even CIA has this expertise.


"Where there's a will (and enough black project money), there's a way."

With love, NSA


Submarine cable is nitrogen doped to prevent hydrogen intrusion.


Would make sense to just tap into the repeaters. That is the common sense approach from someone (me) that knows zip about the setup but that seems like the way it would be done.


They're optical repeaters.


> Also, how is the spliced data actually exfiltrated back to land?

In the operations against the Soviets that've been made public, they physically retrieved the tap.


Why bother when they have direct access to the US end of it anyway?


Why that submarine in particular?



Because the USS Parche (SSN-683) is decommissioned?


my thoughts exactly.


my friends and I always half joke about this too. It's likely the sad truth.


"TM: Work to restore Internet speed starts March 25" http://www.thestar.com.my/tech/tech-news/2016/03/02/tm-inter...


I'm in Singapore and this is not reported at all. Likely affects only Telkom Malaysia users only - one Malaysian telco lost their link to Singapore rather than Singapore losing link to the world as implied by the headline.


It definitely affects Thailand too, Sinet sent notifications about the problem, and speed is quite bad to some sites.


If I were a terrorist org with deep pockets, I'd just hire a black market deep see trawler with a sharp cable hook and have it drive north south a thousand miles out of Japan. Global everything shall halt.


Infrastructure damage that causes severe issues is such a low hanging fruit it's a mystery to me why it isn't attacked more by terrorist / anarchist / primitivist / et al. groups unless I adopt the Hansonian "X is not about seemingly related Y" viewpoint...


Thesis: The bogeymen don't really exist.


And/or they aren't very smart or very well resourced.


If a terrorist plots in his lounge-room and nobody ever sees repercussions, do they really exist?


That's way too general and easily disprovable by any one of the single counterexamples of groups identified as bad people (whether or not the ones pointing fingers are also bad) blowing shit up or killing innocents. The Hansonian viewpoint is at least specific and offers predictions. (Here's a good overview: http://www.gwern.net/Terrorism%20is%20not%20about%20Terror)


Yep, ISIS is a ploy for MIC global domination... We all in tech see it, fuck the deniers.


If you want to make flippant statements, perhaps you should find or research an infographic like: "Civilian deaths due to extremist groups vs. civilian deaths due to US military actions and sanctions".

Feel free to ignore the far longer history of one side, and the fact one side gave weapons and stupid quantities of free media airtime to the other.

If you recall, Wikileaks' war logs proved once and for all the real quantity of civilian deaths is a lot higher than you hear reported in the media by outing the US military's internally reported stats.

More people probably died in the first two months of the year in multiple African conflicts than civilian deaths attributable to terrorism in the last 10 years.

PS. Funny how as soon as I post this reply I get downvoted on both comments. American nationalism is alive and kicking on HN :)


You may be misinformed.

I might be.

However I believe that I can back my shit up better than you can.

Let's prove it.

Gimme links please.



Just because the MIC is winning doesn't prove me wrong. The usg incentive is continual war.

Chomsky backs this up, you're actually saying the same thing as me: that the system is fed by war. Show me how that isn't true?

However I will go through all your links and read them all.

But war is the only thing that keeps this country alive at the moment. War and financial fraud.


I'm going to upvote you without even reading your response for the sake of discussion


I am amazed that the undersea cable is not so thick at all. I was expecting a huge tub of cable. Just thinking about that and the amount of data we can equip across continents in less than 200ms is just incredible.

Is there a documentary on deep sea/undersea cable system? I am really interested in how they deploy the cables across the ocean. Have a ship sail across the ocean and start laying the cable down?



I see thanks. Now wonder how they pick up where it left off :-))


I can't seem to find it at the moment but ISTR watching a video (can't recall if it was actually a "documentary" or just some PR thing they put together) that Hibernia Atlantic put out. That may give you a starting point for finding one.


Fun fact. If you read the book The Victorian Internet, there's an interesting anecdote where a pioneer of the telegraph had a flawed theory that the thinner the cross-Atlantic copper wire, the better the transmission signal.


Weird, but in February 29, some of our customers experienced difficulties to connect to our Linode servers in Dallas (not Singapore) from some Internet providers in US (e.g: mobile carriers, cable ISPs). That day I checked the Linode status and it was normal so I forgot the issue until now.

Do you think these events can be correlated? May be they were trying to mitigate the issue and affected other regions as well.


Is it possible? Sure. Is it likely? No.

Without a lot more information -- preferably from both sides -- it's impossible to know.


Sorry, but I was expecting an answer from someone at Linode. Since their update page was not accurate I couldn't rely in their last report. I think the answer is: it is possible and it is likely.


It's so startling to see failures so...unabstract. We depend so much on the layers of the internet fulfilling their protocols perfectly, that seeing a vulnerability like this is just bizarre.

Of course xkcd is relevant as always: https://xkcd.com/697/


On a tangent, I'm curious whether events as the 8.2 earthquake that happened today, could affect these fibers?


I just recently read an article about this...

"Around 150 to 200 fiber-optic cable breaks are recorded each year. Between 65% and 75% occur in water depths of < 200 m and result mainly from fishing and shipping activities. In contrast, breaks attributed to geohazards comprise < 10% of the world average. However, seaward of the busy continental shelf and upper continental slope, geohazards account for at least one-third of breaks."

-- "Insights into Submarine Geohazards from Breaks in Subsea Telecommunication Cables"

PDF: https://www.iscpc.org/documents/?id=1794


An earthquake took out a section of undersea fiber in Alaska recently, so it has definitely happened before:

http://www.adn.com/article/20140725/59-earthquake-causes-tel...


I wondered about the earthquake as well. Context/maps:

http://www.bloomberg.com/news/articles/2016-03-02/magnitude-...


They are designed with this in mind, and certainly hold up better than oil pipelines, which also have to be engineered to withstand earthquakes.

I expect it is unlikely for an earthquake itself to sever a cable, but knocking out power / knocking over a building that then destroys the cables landing site is possible, if rare.


Singapore has a fair bit of redundancy of cables. Perhaps linode are only using a single provider (seems likely given they're cost concious).

The only Perth-Singapore cable has broken a few times on the past few years, latency really sucks then going from 50ms to 270ms (routing via the US instead).


Is there a global status of cut/degraded undersea cable systems? That would be very beneficial. It seems that the connectivity providers Level3, Hibernia/Atrato, HE, NTT etc.. only relay limited information, perhaps their notification from cable providers is very limited too, by the time it gets to downstream customers we're all guessing, e.g. on WebHostingTalk there are customers trying to determine latency issue to a Hong Kong provider, only to have a competitor announce a fiber cut 2 days ago in Jakarta may be affecting it.


You are wrong on one part: connectivity providers usually provide detailed information about failures to their customers [1]. The only thing is: you are not their customer — your ISP is.

Some ISPs will communicate to their own customers, i.e. you, for instance see the link NiekvdMaas posted in the thread [2]. Sadly, most ISPs do not relay any detailed info.

[1] I once saw a fully detailed report from a connectivity provider about a cable failure and repair, with pictures of scuba-divers included — the cut happened on fibers running at the bottom of a water canal.

[2] https://news.ycombinator.com/item?id=11213555


I hope this isn't China what with their recent land grab in the pacific [1].

[1]: http://www.independent.co.uk/news/world/asia/us-issues-stron...


I wonder if any countries have "just in case" sabotage-devices near the the undersea cables of possible enemies..


I'm sure that you're right. One of the first actions by Britain at the start of WWI was to cut Germany's undersea telegraph cables.

(Except for one which was under Britain's control. Germany ill-advisedly continued to send messages using this cable, confident in their encryption techniques. This led to the interception of the Zimmerman telegram which contributed to the United States' entry into the war.)


Undersea, on land, and outerspace, oh yeah.

Rumors abound of anti satellite/satellites (DS-P1-M and IS Satellites) and I think Russia, china, and the US have destroyed satellites with rockets. https://en.wikipedia.org/wiki/Anti-satellite_weapon

We have tinsel cluster bombs for power plants on land (BLU-114/b) https://fas.org/man/dod-101/sys/dumb/blu-114.htm

And undersea tapping is being done: http://siliconangle.com/blog/2013/07/19/how-the-nsa-taps-und...


I sometimes wonder how many countries _don't_.

(Both the ability to destroy cables moving data for "possible enemies" who are other countries, as well as for "possible enemies" who are their own citizens...)


or a fleet of submarines that run around tapping them...?


If you've installed a tap, it can just as easily be a tap with a limited self-destruct (for deniability) and/or a bomb (for sabotage.)


The point is that the comm lines are more valuable as an intelligence asset than they are as a strategic target.



is it though?

From the link:

The Asia America Gateway, which connects Vietnam and the US, will be serviced from March 4 to 6, the source told Thanh Nien.

That says, March 4 -> 6. Also only specifies Vietnam.

Edit: Spelling correction


Few people realize that to splice an undersea cable, you need two (big) ships. This is because if the cut was where the cable is a mile deep, by the time you bring the ends to the surface they will be approximately two miles apart.


Aren't the cables run with a bit of slack for just this reason?


What caused this? Sabotage? Animals? Submarines?


Let me check.

  $ bofh
  user to computer ratio too high.


how thick are these cables?


Depends on the cable, but it's not unheard of to be a foot or more in diameter. Here's a deep-sea fiber splitter:

http://i.imgur.com/qh4Gf.jpg


This is incorrect. A subsea fiber optic cable is about an inch in diameter. The splice cases and amplifiers are bigger.


It's not incorrect, the original comment is vague. I think it is more reasonable to assume they are talking about the entirety of the cable rather than just the carrier embedded within it.


Indeed I was. Thanks for the clarification.


Fucking sharks


Downvoted for being funny(?) but seriously, any speculation on what could have cut them? Ocean currents? Debris?



Sure, there's lots of speculation -- and we can speculate too, but it's pointless.

I'll go with solar flares.


Some ship dragged an anchor across them.


Discuss: Kickstarter to fund unmanned submersibles to follow a cable end-to-end, looking for taps


I'm afraid that you'll only be able to follow them for a short while before they slip below the ocean floor.


What about duplicating how they repair broken cables... hook onto it close to shore, pull the cable up to a boat, then follow the length of the cable across the ocean.

I'm assuming that's how they're repairing this one?


Likewise we're talking an immense distance in the deepest of oceans.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: