On the geometry side from the theoretical point of view you can repair meshes, [1], by inferring a signed or unsigned distance field from your existing mesh, then you contour this distance field.
If you like the distance field approach, there are also research work [2], to estimate neural unsigned distance fields directly, (kind of a similar way as Gaussian splats).
[1] https://github.com/nzfeng/signed-heat-3d [it works but it's research code, so buggy, not user friendly, and mostly on toy problems because complexity explode very quickly when using a grid the number of cells grows as a n^3, and then they solve a sparse linear system on top (so total complexity bounded by n^6), but tolerating approximations and writing things properly practical complexity should be on par with methods like finite element method in Computational Fluid Dynamics.
Playing chess with strings to build datasets for text generation.
I want to share this quick win.
The other day I was asking myself some theoretical chess questions, and wanted to answer them programmatically and needed to build some custom chess datasets for that.
I needed the chess basic routines, like getting the next legal moves, displaying the board, and some rudimentary position scores. I contemplated writing from scratch. I contemplated using some library. But instead I settled for a higher level choice : interfacing with Stockfish game engine over a text interface.
So instead of writing bug prone routines to check the validity of board positions, it turn the basic routines into a simple wrapper of parsing task to read and write UCI protocol to use a battle tested engine.
A chess position state is simply defined as a vector<string> representing the sequence of moves. Moves are string in long algebraic notation.
This architectural decision allows for very quick (LLM-powered development) prototyping.
namespace bp = boost::process;
bp::ipstream is;
bp::opstream os;
Here OpenERV use a push-pull ventilation design where air direction is reversed every 30s. This allows energy recuperation and dispense connecting the inlet and the outlet to each other, as each ventilation port alternate role simultaneously.
The alternative design is a counter-flow heat exchanger.
Using 3d printing and gyroids it seems possible to build quite compact ones. (metal 3d printed heat exchanger for helicopter https://www.youtube.com/watch?v=1qifd3yn9S0 )
3d-printing a counter-flowing heat-exchanger seems interesting but maybe there are some molding issues that need to be taken care of (maybe HEPA filters on the inside in/outlet are sufficient).
The main advantage of the heat-exchanger solution is that you won't need specific electronic control and can reuse the standard fans for controlled ventilation, but there is more thermally isolated piping required (and the pipes are quite big (~10cm diameter) because they need to move a lot of air even if the fans are weak).
The push-pull system is harder to DIY because most of the off-the shelf fans can't be reversed easily (and 3d printed fans are noisy and inefficient).
I find the idea of reversing the air flow direction every 30s simpler to understand than two counter-flowing pipe side by side.
Imagine a pipe filled with 3 metallic grid sections (such that the air temperature in the section will equalize with the metal temperature) separated by plastic grids (such that the heat isn't conducted through the metal), and you push air alternatively from one hot side at 20°C to a cold side at 0°C for 30s and in the other direction for 30s.
For symmetry reason, the pipe will passively (we don't count the energy required to move the air) have a gradient of temperature from the hot side to the cold side. The first section will be ~15°C, the second ~ 10°C, the third ~5°C. (Each section temperature is the temporal average of the temperature of the air flowing from previous sections : so because air switch direction, it means it's the average of left and right sections.)
From the point of view of the house, you only lose energy from the first section of the pipe which will be more like 15°C rather than 0°C.
The left-associative notation to remove unneeded parenthesis makes it even harder.
It could be explained so much better if the author made the 10 pictures corresponding to the transformation rules of the trees. Eventually highlighting the subtrees a, b, c, in the corresponding color before after.
Brains are used to pattern matching images but not abstractly defined syntax unless you have been trained in grammar theory.
Yes in a nutshell they explain that you can express a picture or a video with relatively few discrete information.
First paper is the most famous and prompted a lot of research to using text generation tools in the image generation domain : 256 "words" for an image, Second paper is 24 reference image per minutes of video, Third paper is a refinement of the first saying you only need 32 "tokens". I'll let you multiply the numbers.
In kind of the same way as a who's who game, where you can identify any human on earth with ~32bits of information.
The corollary being that contrary to what parent is telling there is no theoretical obstacle to obtaining a video from a textual description.
These papers, from my quick skim (tho I did read the first one fully years ago,) seem to show that some images and to an extent video can be generated from discrete tokens, but does not show that exact images nor that any image can be.
For instance, what combination of tokens must I put in to get _exactly_ Mona Lisa or starry night? (Tho these might be very well represented in the data set. Maybe a lesser known image would be a better example)
As I understand, OC was saying that they can’t produce what they want with any degree of precision since there’s no way to encode that information in discrete tokens.
If you want to know what tokens you want to obtain _exactly_ Mona Lisa, or any other image, you take the image and put it through your image tokenizer aka encode it, and if you have the sequence of token you can decode it to an image.
The whole encoding-decoding process is reversible, and you only lose some imperceptible "details", the process can be either trained with a L2Loss, or a perceptual loss depending what you value.
The point being that images which occurs naturally are not really information rich and can be compressed a lot by neural networks of a few GB that have seen billions of pictures. With that strong prior, aka common knowledge, we can indeed paint with words.
Maybe I’m not able to articulate my thought well enough.
Taking an existing image and reversing the process to get the tokens that led to it then redoing that doesn’t seem the same as inserting token to get a precise novel image.
Especially since, as you said, we’d lose some details, it suggests that not all images can be perfectly described and recreated.
I suppose I’ll need to play around with some of those techniques.
After encoding the models are usually cascaded either with a LLM or a diffusion model.
Natural Image-> Sequence of token, but not all possible sequence of token will be reachable. Like plenty of letters put together form non-sensical words.
Sequence of token -> Natural Image : if the initial sequence of token is unsensical the Natural image will be garbage.
So usually you then modelize the sequence of token so that it produce sensical sequences of token, like you would with a LLM, and you use the LLM to generate more tokens. It also gives you a natural interface to control the generation of token. You can express with words what modifications to the image you should do. Which will allow you to find the golden sequence of token which correspond to the mona-lisa by dialoguing with the LLM, which has been trained to translate from english to visual-word sequence.
Alternatively instead of a LLM you can use a diffusion model, the visual words usually are continuous, but you can displace them iteratively with text using things like "controlnet" (stable diffusion).
All pieces are there, we just need to decide to do it. Today's AI are able to produce an increasing tangled mess of code. But it's also able to reorganize the code. It's also capable of writing test code, and assess the quality of the code. It's also capable to make architectural decision.
Today's AI code, is more like a Frankenstein's composition. But with the right prompt OODA loop and quality assessment rigor, it boils down to just having to sort and clean the junk pile faster than you produce it.
Once you have a coherent unified codebase, things get fast quickly, capabilities grows exponentially with the number of lines of code. Think of things like Julia Language or Wolfram Language.
Once you have a well written library or package, you are more than 95% there and you almost don't need AI to do the things you want to do.
There is a huge gap in performance and reliability in control systems between open-loop and closed-loop.
You've got to bite the bullet at one point and make the transition from open-loop to closed-loop. There is a compute cost associated to it, and there is also a tuning cost, so it's not all silver lining.
>Once you have a coherent unified codebase, things get fast quickly, capabilities grows exponentially with the number of lines of code. Think of things like Julia Language or Wolfram Language.
>Once you have a well written library or package, you are more than 95% there and you almost don't need AI to do the things you want to do.
That's an idealistic view. Packages are leaky abstractions that make assumptions for you. Even stuff like base language libraries - there are plenty of scenarios where people avoid them - they work for 9x% of cases but there are cases where they don't - and this is the most fundamental primitive in a language. Even languages are leaky abstractions with their own assumptions and implications.
And these are the abstractions we had decades of experience writing, across the entire industry, and for fairly fundamental stuff. Expecting that level of quality in higher level layers is just not realistic.
I mean just go look at ERP software (vomit warning) - and that industry is worth billions.
On the geometry side from the theoretical point of view you can repair meshes, [1], by inferring a signed or unsigned distance field from your existing mesh, then you contour this distance field.
If you like the distance field approach, there are also research work [2], to estimate neural unsigned distance fields directly, (kind of a similar way as Gaussian splats).
[1] https://github.com/nzfeng/signed-heat-3d [it works but it's research code, so buggy, not user friendly, and mostly on toy problems because complexity explode very quickly when using a grid the number of cells grows as a n^3, and then they solve a sparse linear system on top (so total complexity bounded by n^6), but tolerating approximations and writing things properly practical complexity should be on par with methods like finite element method in Computational Fluid Dynamics.
[2] https://virtualhumans.mpi-inf.mpg.de/ndf/
reply