Hey all! This is something we've been working on for a while now. It's the future of native cloud storage on macOS. I run ExpanDrive (www.expandrive.com). We use a derivative of macFUSE (https://osxfuse.github.io/) on macOS.
Third party kernel extensions on macOS are on the way out. On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.
This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.
The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.
It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).
Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.
The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.
Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location.
Benefits
1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard
2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same
3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now
Hey all! This is something we've been working on for a while now. It's the future of native cloud storage on macOS.
I run ExpanDrive (www.expandrive.com). We use a derivative of macFUSE (https://osxfuse.github.io/) on macOS.
Third party kernel extensions on macOS are on the way out.
On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.
This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.
But that’s okay, they’ve got something better they’re working on. Just keeping it a little quiet right now since it’s not 100% done.
The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.
It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).
Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.
The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.
Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location.
Benefits
1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard
2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same
3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now
I'm playing with it. This is cool. I can't wait for SFTP support. (I know that is difficult for a myriad of reasons, but that's also why it would be so rad. Don't be afraid to assist it using SSH POSIX commands like find when they're available and compliant. There's no prize for SFTP implementation purity.)
Does the File Provider integration support block-level random access? Like, if I have a 1 GB .mp4 video, does it need to download the full video file before playback? This has always been one of the tricky scenarios for high-level OS file APIs.
the search feature does not seem to work. the web page shows an animation of typing a search into a drop down dialog box. When I install the trial, it does not allow me to enter anything here. Searching from the MacOS finder is very very slow. I have tried ExpanDrive in the past, and had to give it up for this reason... searching a mounted cloud drive is just too slow
Hey all! This is something we've been working on for a while now. It's the future of native cloud storage on macOS.
I run ExpanDrive (www.expandrive.com). We use a derivative of macFUSE (https://osxfuse.github.io/) on macOS.
Third party kernel extensions on macOS are on the way out.
On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.
This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.
But that’s okay, they’ve got something better they’re working on. Just keeping it a little quiet right now since it’s not 100% done.
The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.
It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).
Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.
The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.
Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location.
Benefits
1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard
2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same
3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now
B2 pricing is sooo attractive ... But I have been stopped by the much lower performance compared to S3 ... So unfortunately I can confirm this point :(
When you say performance, do you mean upload, download, or both? I can deal with poorer upload performance, and mitigate poorer download performance by leveraging a decent CDN like Cloudflare.
Please consider an Asia/Pacific data center. I am from India and my company was not able to use B2 due to high response times even from European DC. Even a DC in Singapore will be helpful for us.
Bandwidth in asiapac is very expensive for non-incumbents and India is no exception.
I find it bizarre how in India you can get 100GB of LTE for a few dollars but cdn bandwidth can cost content providers more than that - which is absurd.
Mobile broadband is witnessing intense competition in grabbing customers as millions of rural Indians are coming online. This started with a Petro-chemical billionaire starting Jio Network and giving free unlimited 4G data for a year(his company has 300million subscribers now).
Already 4 networks have exited the market and 3rd & 4th largest networks(Vodafone and Idea) have combined due to cash crunch. Airtel(earlier largest) has been raising outside money in hopes that it can survive the low prices. So there are only 4 networks remaining. Only recently they started increasing prices.
That billionaire is also going into Fibre(purchased his bankrupt bother company's infrastructure), maybe we'll see that competition extend to DC and interconnects.
I think the APFS snapshot integration is easily the coolest part of Arq 6. Arq now has access to a special Apple entitlement to take full-desk point-in-time snapshots of an APFS container and backup from that. It's like what time machine would've/should've been, for the cloud.
For the most part, I'd just be using macOS WindowServer. Which isn't very helpful. That's part of what is driving the question about whether one big monitor is ultimately better.
I’ve been using Infra.app for over 3 weeks. Love it.
Made by the founders of Kitematic and Docker for Mac and Windows.
Their care shows in the product.
Curious if anyone knows what Dropbox will do about SmartSync. KAuth extensions are likely out as of 10.16, and the Endpoint Security extensions don’t let you block a reply for more than 60 seconds, so you can’t dynamically page in large files anymore.
They have access to a kernel module and entitlement that only they have access to, com.apple.fileutil - and Apple has said they aren't giving out any more entitlements for it.
I would bet hard cash that it'll be out as of 10.17. Almost every major vendor is done, and the ones that aren't have a 2020 OKR (or equivalent) around this.
> more than 60 seconds, so you can’t dynamically page in large files anymore
certain vendors with legacy engines they've ported 10 years into the future are in trouble. But, assuming a 10.17 release, which they've worked out with apple? They'll be fine, mostly.
They currently offer no replacement for VFS. But I bet they will offer something fuse-like if they offer anything at all. Also: the file provider api they use for iCloud Drive that was supposed to ship in Catalina but got yanked is still likely to happen.
Third party kernel extensions on macOS are on the way out. On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.
This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.
The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.
It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).
Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.
The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.
Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location. Benefits
1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard
2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same
3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now
4. All sorts of other stuff, happy to keep going