Yes, they put this in footnote 1: "Throughout this article “training” can refer to either pre-training, or fine-tuning." But the article is just talking about fine-tuning.
I posted this on HN back in 2023, reposting now because I don't think this article goes far enough:
I’ll make the bold claim that the following industries / companies would not exist without the USPS:
The Airline Industry: In the early days of American aviation, air transportation was unproven and not financially viable, until the USPS built the necessary infrastructure and gave contracts to airlines to allow them financial feasibility… starting in 1918! [1]
Machine Learning: In 1989 Yann LeCun wrote his seminal paper “Backpropagation Applied to Handwritten ZIP Code Recognition”, which used the USPS’s data set and has today become the hello world of machine learning tasks. More importantly this is the first commercial or industrial application of machine learning. [2]
Netflix: Before Streaming became a thing, Netflix was shipping DVDs via the USPS. The Postal Service adapted its processes and equipment to make this financially feasible, supporting Netflix through its transition to streaming. [3]
Amazon: Early Amazon was only a book vendor, the USPS offered special rates for books that made it possible for Bezos to be profitable from his garage … in 1994, thus birthing the behemoth it is today. [4]
Chickens: okay, not really. But the USPS ships millions of pounds of live chickens and other animals each year! [5]
The International Space Station was an international effort by multiple (of the richest) countries specializing in different areas. There is no one country that does it all.
You would think people like HN User corimaith would have learned from the ISS fiasco.
I've gotten to the point where I no longer even listen to people who underestimate the Chinese. Whatever points they're making can be safely ignored. Indeed, strategic sense demands we ignore them. The time for underestimating the Chinese is long past. That's not the reality we live in any longer.
On the contrary, the use of sanctions precisely recognizes the Chinese ability to innovate and the need to not give them our technology on a silver plate while they are it. Tens, if not hundreds of billions are already being funded into the chip industry's r&d and bringing manufacturing back home.
"Against the power of Mordor there can be no victory. We must join with him, Gandalf. We must join with Sauron. It would be wise, my friend."
I've seen this argument before with some pro-China voices and it's the same circular reasoning in arguments like Roko's Basilisk. It is is futile and unwise to oppose China's ascent at the risk of incurring their future wrath when said rise is inveitable, better that we pledge loyalty now for future favours". It's nothing more than intimidation.
The difference is that future of China's hegemony is far from inevitable. They are strong, but they're not stronger just yet. Their innovation hasn't really broken through in red oceans when paired against competive incumbents, and they are just as prone to hype bubbles as we are. And their massive investment is coming at the cost of a resulting maligned consumption balance and destructive price wars. So there still moves to be made that can radically alter the trajectory of events.
Which argument is right really depends on a realistic evaluation of the situation.
To me I just have a bad gut feeling when we (the west) force China to oppose us.
I also had a bad gut feeling when we outsourced our technology there. We let the buy our companies while China only allowed joint ventures with state controlled companies.
Why didn't we pick up the fight 10 years earlier if they are so evil that we cannot let them win?
Thanks for sharing your real world experience, it helps in seeing how regular folk are affected by policy decisions.
I understand from your post that you are a business person, buying product, performing value added services and selling for profit. Although I know little about business, I would guess that if one of your suppliers raised the prices on one of the inputs to your finished goods, you would likely increase the price of your product to preserve your profits and continue your business as a venture. I would guess that you would not pay the additional cost out of your own pocket.
My question is; why did you not expect the same logic to play out in the tariffs situation? That any country would pay the additional cost of doing business out of their own pocket and not pass it on to the consumer?
I read the article on archive and figured there was a big chunk missing. It really does not make any sense.
Sutskever and Murati were methodical, they waited until the board was favorable to the outcome they wanted, engaged with board members individually laying the groundwork... and then just changed their mind when it actually happened!?
The article says Sutskever was blindsided by the rank-and-file being on Sam's side. Presumably he thought the outcome was going to be business as more-or-less usual but with Murati or someone as CEO and then panicked when that didn't happen.
Or someone said "If you don't switch and back me, I am going to fight every bit of your compensation. Or you can back me and leave with favorable terms."
Thanks for writing this out, it's helpful for me as a layman.
Isn't part of the prohibition on trades among officers and directors also because of the inside knowledge they have? Public companies generally report quarterly but the insiders presumably have up to the minute information on sales etc.
And while we wait on the quarterly data, consistent insider selling is indicative of ... something.
The LLM community has come up with tests they call 'Misguided Attention'[1] where they prompt the LLM with a slightly altered version of common riddles / tests etc. This often causes the LLM to fail.
For example I used the prompt "As an astronaut in China, would I be able to see the great wall?" and since the training data for all LLMs is full of text dispelling the common myth that the great wall is visible from space, LLMs do not notice the slight variation that the astronaut is IN China. This has been a sobering reminder to me as discussion of AGI heats up.
It could be that it “assumed” you meant “from China”; in the higher level patterns it learns the imperfection of human writing and the approximate threshold at which mistakes are ignored vs addressed by training on conversations containing these types of mistakes; e.g Reddit. This is just a thought. Try saying: As an astronaut in Chinese territory; or as an astronaut on Chinese soil. Another test would be to prompt it to interpret everything literally as written.
Interesting... It took me 3 different attempts, but I found a set of custom instructions that allowed Claude to get the right answer on the initial prompt. Here's the instructions (I tried to keep them as general and non-specific as I could):
Carefully analyze questions to not overlook subtle details. Take each question "as-is", don't guess what they mean -- interpret them as any reasonable person would.
I made https://aimodelreview.com/ to compare the outputs of LLMs over a variety of prompts and categories, allowing a side by side comparison between them. I ran each prompt 4 times for different temperature values and that's available as a toggle.
I was going to add reviews on each model but ran out of steam. Some users have messaged me saying the comparisons are still helpful to them in getting a sense of how different models respond to the same prompt and how temperature affects the same models output on the same prompt.
Hey, this is pretty insightful! Wonder if, in the course of researching to build this website you reached any conclusions as to what’s the AI assistant currently ahead.
And to take a historic analogy, cars today are as wide as they are because that's about how wide a single lane roadway is. And a single lane roadway is as wide as it is because that's about the width of two horses drawing a carriage.
The story goes that this two horses width also limited the size of the space shuttle's boosters (SRB), so we ended up taking this sort of path-dependence off to space.
