What about the North Korean escapees who claim that, out of hunger, they have been extracting and (re-)eating corn kernels out of their own poop (implying there must be more than 2 for the effort to be worth it) - are they full of shit?
"226 of 521 FDA-approved medical devices, or approximately 43%, lacked published clinical validation data."
The lack of "published" clinical validation studies implies neither that the AI developer performed no clinical validation nor that the FDA hasn't seen it. So, it is not clear if the problem is with the lack of clinical validation or the lack of reporting.
For some reason the title exaggerates yet further (half of FDA-approved AI not "trained" on real patient data).
> AI developer performed no clinical validation nor that the FDA hasn't seen it
if the developer went through the trouble and expense of performing clinical validation you can be sure that they would publish the results __unless__ the results reflected negatively on them
Given that the design and endpoints of clinical validation studies is priceless information for developers of similar devices (i.e., competitors) applying for FDA clearance via the 510(k) pathway, it would not surprise me at all if this information was purposefully kept secret no matter how flattering it is. Especially for relatively new technology like AI
510(k) summaries are always a negotiation between FDA (which wants as much data as possible in them) and manufacturers (which want as little data as possible, as it is priceless competitive intelligence)
I was under the impression that with LLMs, in order to get high-quality answers, it's always best to keep context short. Is that not the case anymore?
Does Claude under this usage paradigm not struggle with very long contexts in ways as for example described in the "lost in the middle" paper (https://arxiv.org/abs/2307.03172)?
The conclusion you walked away with is the opposite of what usually works in practice.
The more context you give the llm, the better.
The key takeaway from that paper is to keep your instructions/questions/direction in the beginning or at the end of the context. Any information can go anywhere.
Not to be too dismissive, it's a good paper, but we're one year further and in practice this issue seems to have been tackled by training on better data.
This can differ a lot depending on what model you're using, but in the case of claude sonnet 3.5, more relevant context is generally better for anything except for speed.
It does remain true that you need to keep your most important instructions at the beginning or at the end however.
At the beginning it was true, the longer the context, the more the LLM was lost, but now, the new models can retrieve information anywhere in the context
I don't have the time to evaluate the effects of context length on my use cases so I have no idea. There might be some degradation when I attach the Qt book which is probably already in Claude's training data but when using it against my private code base, it's not like I have any other choice.
The UX of drag and dropping a few monolithic markdown files to include entire chunks of a large project outweighs the downsides of including irrelevant context in my experience.
No, you need to provide as much information in context as possible. Otherwise you are sampling from the mode. "Write me an essay about cows" = garbage boring and probably 200 words. "here are twenty papers about cow evolution, write me an overview of findings" = yes
Why are we so sure that a lot of "previously intractable problems" are/will be solved with this family of methods? (and I mean real-life/real-world problems, not toy problems constructed specifically to show the proposed methods in the best light possible in the research papers)
Of course others above have pointed out drug or protein design as a potential area, but there still seems uncertainty as to the practical impact on the real world. Other than that, I don't see areas of impact for these approaches so far.
One of the authors here. Thanks for your comment! While a lot of this research is theoretical and does not have immediate use cases, we have tried to summarize some of them in the last paragraph of the paper (VII. Applications of Non-Euclidean Geometry). See page 26. We present some in Chemistry and Drug Development, Structural Biology and Protein Engineering, Computer Vision, Biomedical Imaging, Recommender Systems and Social Networks and Physics.
Is geometric, topological, and algebraic ML/data analysis actually used in the industry? It is certainly beautiful math. However, during grad school I met a few pure math PhD students who were saying that after finishing their PhD they will just go into industry to do topological data analysis (this was about 10 years ago and ML wasn't yet as hyped up). However, I have never heard of anybody actually having success on that plan.
As someone who did an applied math PhD before drifting towards ML, it's worth pointing out that these applied math groups typically talk about applications, but the real question is whether they are actually used for the stated application in practice due to outperforming methods that use less pretty math. Typically (in every case i have seen) the answer is "no", and the mathematicians don't even really care about solving the applied problems nor fully understand what it would mean to do so. It's just a source of grant-justifiable abstract problems.
Thanks. That's certainly very interesting. Albeit it seems to me that the number of jobs doing geometric and topological ML/AI work in the drug or protein design space would be quite limited, because any discovery ultimately has to be validated through a wet lab process (or perhaps phase 1-3 clinical trials for drugs) which is expensive and time-consuming. However, I'm very uninformed and perhaps there is indeed a sizable job market here.
I think the job market in general for this kind of stuff is "small"; but you can find jobs. Look at Isomoprhic Labs for example. There are new AI/ML companies that have emerged in recent years, helped by success of things like AlphaFold. I think your question is really: does this research actually creates tangible results? If it did, it would be able to create more jobs to support it by virtue of being economically successfully and therefore growing?
Hyperbolic embeddings have been an interest of mine ever since the Max Nickel paper. Would love to connect directly to discuss this topic if you're open. here's my email: https://photos.app.goo.gl/1khCwXBsVBuEP6xF7
Not much to discuss really, I just monkey patched a different metric function, then results for our use case became substantially better after training a model from scratch on the same data compared to the previous euclidean model trained from scratch.
I'm currently working on massive multi agent orchestration so don't have my head in that side of things currently.
Can you share what kinds of problems were conducive to hyperbolic embeddings in your experience. Also, separately, are you saying companies are using these in practice but don’t talk about them because of the advantage they give? Or am I reading too much into your last sentence.
They are better at separating clusters and keep the fact that distances under the correct metric also provide semantic information. The issue is that training is longer and you need at least 32, and ideally 64 bit floats during training and inference.
And possibly.
The company I did the work for kept it very quiet. Bert like models are small enough that you can train them a a work station today so there is a lot less prestige in them than 5 years ago, which is why for profit companies don't write papers on them any more.
I don't think there's much use currently. But I kinda like the direction of the paper anyway. Most mathematical objects in ML have geometric or topological structure, implicitly defined. By making that structure explicit, we at worst have a fresh new perspective on some ML thing. Like how viewing the complex numbers on a 2d cartesian plane often clicks more for students compared to the dry algebraic perspective. So even in the worst case I think there's some pedagogical clarity here.
