Can anyone explain why it is not possible to publish your work as a PDF or to a free/open platform at the same time you submit it to the journal? It this somehow prevented legally in the terms of the journals?
Yes, it is prevented legally. When a journal agrees to publish your work they require you to sign over copyright to them.
The terms may vary on the transfer of copyright, but in general you are not authorized to self-host a free version of your article. You can sometimes pay the journal a significant amount of money to make it open access to the public.
Some authors, myself included, ignore the agreement and still host free copies.
I'd also be interested in this answer. What I have done is email researchers directly asking for a PDF and they have been happy to give me one. Often some of the authors have their own site where they add links to their own papers.
Now, is this because these authors are exercising their own right to distribute their work, and many other authors don't want to expend that effort? The copyrights don't change after all, so collating these and redistributing could also be the same as sci hub.
It all depends on the terms of the copyright transfer, which is based on the historic relationship between author and publisher.
It used to be that when you published a paper and transferred copyright, you received a number of copies of the publication for yourself. You could send these out to others who wanted a copy. This was especially important before xerography machines were common, as it was difficult to make your own copies.
Even in the 1990s, an academic researcher might get a postcard asking for a reprint to be sent in the mail.
So research publishing has long had a tradition where the author has a limited right to redistribute copies directly. (To be certain, the publisher made the reprints so this wasn't part of copyright but rather a codified expectation of the author/publisher relationship.)
This tradition was carried over into electronic publication Why? Consider the 1990s and early 2000s when the transition to electronic journals took place, and imagine the authors up in arms for having that distribution ability taken away.
This is why you can get a copy from the author. It's sometimes also formally specified that author may also distribute a copy on a personal or research web page. Sometimes this permission is for the author's final draft version, and not the proof copy used for the publication.
But there wasn't a tradition of sending the paper to a tertiary reprint service.
