I now wish there was a feature to make one's own comment less prominent yet still in thread, a self down vote in essence.
Because I took my first point in a direction away from the context, despite the reasons I wrote being the underlying reason.
I'm aiming at the extent to which data is susceptible to considerable performance improvements when you have the ability to align the physical store with the data structure.
Simultaneously, arguing as I did, to punch holes in the operating system to get to hardware comes with costs and the cost of losing backups is not trivial.
This is why I asked about the Weka FS in my comment, because if you were atop a system like that you would have theoretical redundancy already.
I ignored my dubious omission of this being another FS and not a hardware pass through my comment was about.
However the first point of contact in any case is going to be a requirement to partition a store for pass through access, taking my comment literally.
That's a great path to disaster and not what I meant but omitted the explanation required:
If you had commit access to your FS code, you could provide accommodation for the pass through request to vary metal, but just lose the point of the FS. I instead envisioned the FS providing you with hardware layouts that are suitable for the data you need running fast. I imagined a API to request that "on disk" format, and the FS to be able to indicate potential changes to balance between the requested layout and the performance of FS features such as cluster latency, computational efficiency of raid schemes and other features like T10 and other management of data overall. A scale between handing over the hardware and alerting to management consoles of the existence of a opaque store, and the agreement in tradeoffs the full features provide, where the application is placed in authority to size performance at a level of reasonable costs.
I'm thinking of serialisation that's a part of any larger system which provides reliability features in other components.
But the flexibility could exist to allow for flow under exceptional load, if doing so was critical to the performance of your overall system.
If you are buffering requests at a point where you have to pass the stream to the next component which is responsible for acknowledgement of the requests, and they will be able to resend if they have not received acknowledgement, then the opportunity to trade normal FS behaviour for raw speed, is possible under reported conditions known by the management instruments.
I'm assuming that the FS is not going to let you write data faster than it can function for jobs like replication, and then limits would indicate that the discarding of the FS roles are not always good value.
I haven't seen any sign of the purported advent of intelligence in file systems for the sort of thing I am interested in like this.
But certainly it must be a likelier possibility to cooperate with new FS companies, than the Windows design team and the weight their legacy brings.
I'm out of touch, but NTFS provides or provided interfaces for, e.g. sparse file layouts and there's nothing new in my central proposal. I've merely speculated how far it might be possible to go. And there's a entire field of data structure optimisation which can only be done in a worthwhile way with the FS cooperation. To just leave it at the hope of getting nice features in FSs else write your own driver, seems like a crude bipolar proposition in the present times
Because I took my first point in a direction away from the context, despite the reasons I wrote being the underlying reason.
I'm aiming at the extent to which data is susceptible to considerable performance improvements when you have the ability to align the physical store with the data structure.
Simultaneously, arguing as I did, to punch holes in the operating system to get to hardware comes with costs and the cost of losing backups is not trivial.
This is why I asked about the Weka FS in my comment, because if you were atop a system like that you would have theoretical redundancy already.
I ignored my dubious omission of this being another FS and not a hardware pass through my comment was about.
However the first point of contact in any case is going to be a requirement to partition a store for pass through access, taking my comment literally.
That's a great path to disaster and not what I meant but omitted the explanation required:
If you had commit access to your FS code, you could provide accommodation for the pass through request to vary metal, but just lose the point of the FS. I instead envisioned the FS providing you with hardware layouts that are suitable for the data you need running fast. I imagined a API to request that "on disk" format, and the FS to be able to indicate potential changes to balance between the requested layout and the performance of FS features such as cluster latency, computational efficiency of raid schemes and other features like T10 and other management of data overall. A scale between handing over the hardware and alerting to management consoles of the existence of a opaque store, and the agreement in tradeoffs the full features provide, where the application is placed in authority to size performance at a level of reasonable costs.
I'm thinking of serialisation that's a part of any larger system which provides reliability features in other components.
But the flexibility could exist to allow for flow under exceptional load, if doing so was critical to the performance of your overall system.
If you are buffering requests at a point where you have to pass the stream to the next component which is responsible for acknowledgement of the requests, and they will be able to resend if they have not received acknowledgement, then the opportunity to trade normal FS behaviour for raw speed, is possible under reported conditions known by the management instruments.
I'm assuming that the FS is not going to let you write data faster than it can function for jobs like replication, and then limits would indicate that the discarding of the FS roles are not always good value.
I haven't seen any sign of the purported advent of intelligence in file systems for the sort of thing I am interested in like this.
But certainly it must be a likelier possibility to cooperate with new FS companies, than the Windows design team and the weight their legacy brings.
I'm out of touch, but NTFS provides or provided interfaces for, e.g. sparse file layouts and there's nothing new in my central proposal. I've merely speculated how far it might be possible to go. And there's a entire field of data structure optimisation which can only be done in a worthwhile way with the FS cooperation. To just leave it at the hope of getting nice features in FSs else write your own driver, seems like a crude bipolar proposition in the present times