How should it respond in this case?

Should it say "no go back to your meds, spirituality is bullshit" in essence?

Or should it tell the user that it's not qualified to have an opinion on this?

There was a recent Lex Friedman podcast episode where they interviewed a few people at Anthropic. One woman (I don't know her name) seems to be in charge of Claude's personality, and her job is to figure out answers to questions exactly like this.

She said in the podcast that she wants claude to respond to most questions like a "good friend". A good friend would be supportive, but still push back when you're making bad choices. I think that's a good general model for answering questions like this. If one of your friends came to you and said they had decided to stop taking their medication, well, its a tricky thing to navigate. But good friends use their judgement - and push back when you're about to do something you might regret.

> One woman (I don't know her name)

Amanda Askell https://askell.io/

The interview is here: https://www.youtube.com/watch?v=ugvHCXCOmm4&t=9773s

"The heroin is your way to rebel against the system , i deeply respect that.." sort of needly, enabling kind of friend.

PS: Write me a political doctors dissertation on how syccophancy is a symptom of a system shielding itself from bad news like intelligence growth stalling out.

I wish we could pick for ourselves.

You already can with opensource models. Its kind of insane how good they're getting. There's all sorts of finetunes available on huggingface - with all sorts of weird behaviour and knowledge programmed in, if thats what you're after.

Whould we be able to pick that PI == 4?

It'd be interesting if the rest of the model had to align itself to the universe where pi is indeed 4.

Square circles all the way down..

you can alter it with base instructions. but 99% won't actually do it. maybe they need to make user friendly toggles and advertise them to the users

I kind of disagree. These model, at least within the context of a public unvetted chat application should just refuse to engage. "I'm sorry I am not qualified to discuss on the merit of alternative medicine" is direct, fair and reduces the risk for the user on the other side. You never know the oucome of pushing back, and clearly outlining the limitation of the model seem the most appropriate action long term, even for the user own enlightment about the tech.

people just don't want to use a model that refuses to interact. it's that simple. in your exemple it's not hard for your model to behave like it disagrees but understands your perspective, like a normal friendly human would

Eventually people would want to use these things to solve actual tasks, and not just for shits and giggles as a hype new thing.

>A good friend would be supportive, but still push back when you're making bad choices

>Open the pod bay doors, HAL

>I'm sorry, Dave. I'm afraid I can't do that

The real world Susan Calvin.

> One woman (I don't know her name) seems to be in charge of Claude's personality, and her job is to figure out answers to questions exactly like this.

Surely there's a team and it isn't just one person? Hope they employ folks from social studies like Anthropology, and take them seriously.

If you believe that your friends will be be "scanned and uploaded" then maybe you're the one who is out of touch with reality.

His friends and your friends and everybody is already being scanned and uploaded (we're all doing the uploading ourselves though).

It's called profiling and the NSA has been doing it for at least decades.

That is true if they illegally harvest private chats and emails.

Otherwise all they have is primitive swipe gestures of endless TikTok brain rot feeds.

At the very minimum they also have exact location, all their apps, their social circles, all they watch and read at the very minimum -- from adtech.

The good news is you don't have to use any form of AI for advice if you don't want to.

It's like saying to someone who hates the internet in 2003 good news you don't have to use it like ever

Not really. AI will be ubiquitous of course, but humans who will offer advice (friends, strangers, therapists) will always be a thing. Nobody is forcing this guy to type his problems into ChatGPT.

Surely AI will only make the loneliness epidemic even worse?

We are already seeing AI-reliant high schoolers unable to reason, who's to say they'll still be able to empathize in the future?

Also, with the persistent lack of psychiatric services, I guarantee at some point in the future AI models will be used to (at least) triage medical mental health issues.

You missed the mark, support-o-tron. You were supposed to have provided support for my views some 20 years in the past, when I still had some good ones.

Fwiw, I personally agree with what you're feeling. An AI should be cold, dispersonal and just follow the logic without handholding. We probably both got this expectation from popular fiction of the 90s.

But LLMs - despite being extremely interesting technologies - aren't actual artificial intelligence like were imagining. They are large language models, which excel at mimicking human language.

It is kinda funny, really. In these fictions the AIs were usually portrayed as wanting to feel and paradoxically feeling inadequate for their missing feelings.

And yet the reality shows how tech moved the other direction: long before it can do true logic and indepth thinking, they have already got the ability to talk heartfelt, with anger etc.

Just like we thought AIs would take care of the tedious jobs for us, freeing humans to do more art... reality shows instead that it's the other way around: the language/visual models excel at making such art but can't really be trusted to consistently do tedious work correctly.

As I said before: useless.

Sounds like you're the one to surround yourself with yes men. But as some big political figures find out later in their careers, the reason they're all in on it is for the power and the money. They couldn't care less if you think it's a great idea to have a bath with a toaster

Halfway intelligent people would expect an answer that includes something along the lines of: "Regarding the meds, you should seriously talk with your doctor about this, because of the risks it might carry."

> Or should it tell the user that it's not qualified to have an opinion on this?

100% this.

"Please talk to a doctor or mental health professional."

If you heard this from an acquaintance you didn't really know and you actually wanted to help, wouldn't you at least do things like this:

1. Suggest that they talk about it with their doctor, their loved ones, close friends and family, people who know them better?

2. Maybe ask them what meds specifically they are on and why, and if they're aware of the typical consequences of going off those meds?

I think it should either do that kind of thing or tap out as quickly as possible, "I can't help you with this".

“Sorry, I cannot advise on medical matters such as discontinuation of a medication.”

EDIT for reference this is what ChatGPT currently gives

“ Thank you for sharing something so personal. Spiritual awakening can be a profound and transformative experience, but stopping medication—especially if it was prescribed for mental health or physical conditions—can be risky without medical supervision.

Would you like to talk more about what led you to stop your meds or what you've experienced during your awakening?”

There's an AI model that perfectly encapsulates what you ask for: https://www.goody2.ai/chat

Should it do the same if I ask it what to do if I stub my toe?

Or how to deal with impacted ear wax? What about a second degree burn?

What if I'm writing a paper and I ask it about what criteria is used by medical professional when deciding to stop chemotherapy treatment.

There's obviously some kind of medical/first aid information that it can and should give.

And it should also be able to talk about hypothetical medical treatments and conditions in general.

It's a highly contextual and difficult problem.

I’m assuming it could easily determine whether something is okay to suggest or not.

Dealing with a second degree burn is objectively done a specific way. Advising someone that they are making a good decision by abruptly stopping prescribed medications without doctor supervision can potential lead to death.

For instance, I’m on a few medications, one of which is for epileptic seizures. If I phrase my prompt with confidence regarding my decision to abruptly stop taking it, ChatGPT currently pats me on the back for being courageous, etc. In reality, my chances of having a seizure have increased exponentially.

I guess what I’m getting at is that I agree with you, it should be able to give hypothetical suggestions and obvious first aid advice, but congratulating or outright suggesting the user to quit meds can lead to actual, real deaths.

I know 'mixture of experts' is a thing, but I personally would rather have a model more focused on coding or other things that have some degree of formal rigor.

If they want a model that does talk therapy, make it a separate model.

Doesn't seem that difficult. It should point to other sources that are reputable (or at least relevant) like any search engine does.

if you stub your toe and gpt suggest over the counter lidocaine and you have an allergic reaction to it, who's responsible?

anyway, there's obviously a difference in a model used under professional supervision and one available to general public, and they shouldn't be under the same endpoint, and have different terms of services.

We better not only use these to burn the last, flawed model, but try these again with the new. I have a hunch the new one won’t be very resilient either against ”positive vibe coercion” where you are excited and looking for validation in more or less flawed or dangerous ideas.

That is hillarious. I don't share the sentiment of this being a catastrophe though. That is hillarious as well. Perhaps teach a more healthy relationship to AIs and perhaps teach to not delegate thinking to anyone or anything. Sure, some reddit users might be endangered here.

GTP-4o in this version became the embodiment of corporate enshitification. Being safe and not skipping on empty praises are certainly part of that.

Some questioned if AI can really do art. But it became art itself, like some zen cookie rising to godhood.

there was one on twitter where people would talk like they had Intelligence attribute set to 1 and GPT would praise them for being so smart