I didnt look into it yet but i already wanted to say - cuz of curiosity i build my own graph database in golang :) and i learned alot. so i absolutly understand why you did it and what experiences you probably made on the way :D
Kinda. So first i went for a standalone application back than to use it as storage/database. This part is archived now. it was released under:
https://github.com/voodooEntity/slingshotdb
At this point i think its important to mention that its nativly an in-memory graph database/storage.
At some point (4-5 years ago) i decided to change the way i handle it. Since im using the storage in many private projects, and there was no real traction (while also maintaining a database is quite some work) i decided i gonne change the way i handle it away from a standalone database to a library that works as an in-memory graph storage.
Also i added a custom query language including builder (the queries are 100% json compatible so you can in theory build queries from whatever language and transmit it via json.
Currently im completly reworking the query parser but since this is a privat project and im the only maintainer it may take some to to finish it. The update will takle some smaller bugs in the query parser that can occur on very complex nested queries and also will optimize query speeds. I cant provide a date when the update will occure i only can say it will (because i need it myself).
Finally said: The strength of gits is to be very fast (hence in memory handling) and easy to use from golang. Its threadsafe by default.
We build an Infrastructure with about 6 microservices and Kafka as main message queue (job queue).
The problem the author describes is 100% true and if you are scaled with enaugh workers this can turn out really bad.
While not beeing the only issue we faced (others are more environment/project-language specific) we got to a point where we decided to switch from kafka to rabbitmq.
Im a big fan of GraphDatabase's since about 10 years. I even wrote my own "in memory graph storage" in golang for a specific use case that none of the big GraphDatabase's could cover at the time.
That said - i WISH people would embrase the existing GraphDatabases more and make the hosters support them as standard, rather than abusing existing relational databases for graph purposes.
And to make it clear,i'm not talking about my own experimental one, i mean stuff like Neo4j, OrientDB etc.
> rather than abusing existing relational databases for graph purposes.
In my experience, the vast majority of graphs can be embedded in relational databases just fine and most people don't want general graph querying. People just don't like optimizing queries (or equivalently the schema to enable such queries).
I personally have never seen a pitch for graph databases that makes them seem attractive for more than data exploration on your local machine.
Yep, something like adjacency lists have solved a lot of dumb recursive queries/"graph queries" for me, usually because the graph doesn't change much and is just a few nodes deep.
Also, the last time i checked neptune, it could only be run in AWS but there was no public application image for local tinkering/dev. But its some years ago to be fair - maybe have changed.
I started with orientdb than switched to Neo4j. Orientdb was good for starting tbf, but we talking about 10+ years back. Now i would definately default to Neo4j
Thats like saying why do we need json we can represent it also in a textfile but accessing values can be more tricky.
Sure you can represent a GraphDB in a RDBMS - but graphdbs are optimized for their specific use case. Therefor for example locking of resources is optimized for said purpose. When going a "Edge" and "Node" table approach, locking is by default rather unoptimized in many RDBMS. Just one simple example.
> Thats like saying why do we need json we can represent it also in a textfile but accessing values can be more tricky.
If your text processing stack works good enough, and JSON only solves edge cases, then I think its a legit argument against adding another tech to the stack.
Its not like everyone immediately switched over to JSON even if they could have.
And the analogy may not be the best - in the case of graph databases, they don't do what RDBMS can do, at least not as efficiently.
So if we need some graph functionality in our RDBMS, do we split the database and use a "real" graph database or just incorporate the graph functionality into the existing DB? The choice seems quite easy for me, because there are whole host of issues in splitting DBs (duplication of information,
inconsistency, etc.)
Well i took the best example that came to my mind at that moment. may not be the best analogy but well its what it is.
My point was not about people enrichhing an existing RDMBS concept with Graph, it was about using RDBMS as Graph (as only purpose). So maybe i wasn't exact enaugh in my definition. Therefor:
"If your purpose is to use the benefits of Graph and you want to use it for this purpose only, use a GraphDB and dont use a RDBMS and make it a GraphDB."
I've down developing in private for >15 years now and even tho i wrote alot of stuff i never was motivated enaugh to "publish" something as product that i might need to support.
So first of all its a very cool project even tho i don't code ruby.
But in case the author of the page is in here, please read the following:
Some years ago i had a similar idea, just instead of ruby-cons i went for defcon videos. I build a page that indexed all defcon talks and even tried to categorize them in terms of topics and difficulty. Added a nice search, and put it out as "defcon.video". It didn't take google long to message me and tell me i had to take it down because its actually forbidden to host an alternative search index if you are using the youtube API. You are not allowed to store the results of youtube API (longer than 24hours if i remember) and as stated before not allowed to offer an alternate search.
While you might argue: why didn't you just state that you don't use the api to grab the data and shutoff the token usage : well because google is not really known to be friendly and i expected them to just close down my whole developer account if i don't follow.
So - in case not all the videos are hand inserted (and even then they might come at you) google definatly will reach out to you and try to force you down. Back when it happend to me i didn't have the time and patience to fight them back so defcon.video was shutdown.
> well because google is not really known to be friendly and i expected them to just close down my whole developer account if i don't follow.
and if their algorithm is in a bad mood that day, they'll shut down your personal account(s) as well, and possibly adjacent accounts. Hope you don't rely on personal email or Google Photos or Android if that happens.
This nuke-somebodys-life-from-orbit capability of such a gigantic company is IMHO one of the best arguments for regulatory limits on company size/scope. That's obviously easier said than done and comes with a whole column of cons, but the power imbalance is so unbelievably great and the reach so deep that in some cases I'd rather be on the wrong side of the state than on the wrong side of Google.
I tell people I left Google because of all the weird amount of data harvesting, and they always argue Apple's not any better, but I rarely hear of someone losing their iCloud to the level I hear of people losing their Google Account which is glued to too many damn things.
At this point I assume the only way Google will ever fix their ridiculous ban process is if someone passes a law that forces them to separate these comingled services from losing you access to your email, and essentially in some cases your personal business.
Well that's the reason why im running my private mailserver and the only thing im using google for is actual google services.
I always wondered why people would just give all their personal information to google in terms of mail and files (convenience i know but still..).
But than you see the posts of "google just shutdown my account without any reasoning and i lost everything help!!" here on hackernews and you know exactly why you don't use google for anything but their services.
We are just embedding the video using vlite.js and the YouTube Video ID (which is just embedding an official iframe in the end). So there isn't really an account to "shut down", unless I'm missing something.
I am currently working on building a curated collection of YouTube channels, with custom search index. Thank you for sharing this, I guess I can't continue. It's a shame.
As someone from the "minimalist prefered" guys, im just curious - what articles sparked your interested through this layout and why would you have skipped them on the normal layout?
(inb4 i think the herald is a very well done thing i just cant think of a reason why an article there is more interresting than on the normal layout)
From today: "Portland Airport Grows With Expansive Mass Timber Roof Canopy" is a good example. I wouldn't have opened this story on Hacker News, but the image displayed on Hacker Herald had me open it.
sorry for my late response. Thanks for giving me this example. I can kinda see why you would not click it in first place.
As for me - i have very limited time i spend on reading articles listed on hackernews so i always have to condense it down to those im very interested in.
That is an extremely impressive roof, and I agree that the title isnt grabbing. I too clicked on the picture but had previously skipped the article on HN.
While i have to admit, from all the alternative frontends i have seen over the years (and there have been alot) this is by far the best.
Tho as many others mention, one of the great things on hackernews is the minimalism / simplicity - and i have to admit its one of the reasons why i always like to go on hackersnews. Just a simple textlist, easy to read, no visual clutter.
Tho for people that like a more colorfull/stylish layout this might be a good alternative
For me, I think the lack of more generalized appeal of the interface is a net benefit, because it limits its appeal to only those that are most interested in the content. When it becomes too nice, and the audience too large, it will probably lose the thing that most of us are here (as opposed to reddit) for.
So even though I really like (and will bookmark!) this new interface design, I hope HN proper never changes.
I like the minimal interface, gives an old.reddit.com vibe. HN is horrible to use on mobile though. Everything is tiny, difficult to tap I miss tap every 3 taps
Looking at the pricing i dont see really why anyone would use this api. 2,5cent on first look dont seem to be "much" when thinking of the normal by hand usage of a search engine - but in terms of atomated usage (which is the idea for an API right? why else should i use an api...) this will ramp up very quickly to be very expensive.
Apart from that, while i don't like google for all the reasons we all know - still their support of search operators and the results based on them is a magnitude better than all other search engines i tried so far.
And yes my daily driver is duckduckgo atm, tho sometimes i still fall back to google if duckduckgo doesn't deliver and sadly have to admit that google often results in better results than the others... sad but true.
I gave a Kagi trial with limited searches to a friend and they reported using Kagi when Google failed.
Yeah. It's that much better than Google. Which is better than DDG. Time is money, and I have $10/mo to save so much time.
Really it's about incentives. The other engines want me to see what the advertisers want me to see. Kagi's customer is me; they'll show me what I want to see.
Ok, so i just created a test account on Kagi to see if you are right.
As i mentioned in my post, something i personally really value is search operators.
So i did a direct comparsion on Kagi to google with a simple dork (its not like this would be my daily search but its a simple example for the usage of search operators and i just had no better idea *shrug)
intitle:"Index of" intext:2024 intext:backup
Old and simple directory indexing dork for backup files/directories.
Kagi returns me 13 Results.
Google Returns me 108 Results.
The quality of the results is quite similar.
So i can't tell on other searches using operators, and since my kagi kontigent for my free account is 100 searches i will think about some better search terms and try it again tomorrow. But on first sight, i can't approve the Kagi > Google statement.
is your search operators stuff a little bit too specfiic? Kagi uses google, brave and others. i think the operators just dont work correctly and some you use, just dont exist. and your search this way is highly specfic. when you search regulary you will notice the difference. look at the listable filter when search normally. Your example just doesnt work.
btw this gives more, though some incorrecT: 2024 backup intitle:Index intitle:of
Thats interresting because it worked for me, and i only used operators i found in the kagi docs Oo (intitle / inurl - yes intext isnt in so i just added the term instad of with intext)
I also tried others after the post, i just didnt update here - for example i did a site:laughingman.dev (my domain) to see what gets indexed, and saw also that google finds more stuff than kagi.
I don't wanne badmouth kagi - please dont missunderstand me. And yes search operators especially in the way i use them is a very specific thing, but well thats something i do daily and i rlly like it - if often helps me to find stuff alot more precise than with just some basic stuff. It also helps to sort out false postive results more reliable. If this is not a case for kagi well than kagi might just not be for me - which doesnt mean its not good for others. Im only speaking for myself
dont you search normally too? When you do this, you notice the main difference. I will create a bug report later, but i think the api isnt allowing this operators on google site? So they arent included
To not fall under ,what is criticized by some people here, as making an ad for a competitor - but big parts of it are very similar to a project im working on for multiple years now, i not gonne name or link the project i gonne mention.
While my project is not meant to be specifically used with LLMs, what i build (or am building) is a system which has no specific defined sequence, but is rather a composition of "actions" of which each of them has a specific defined requirement in form of a data structure that is necessary to execute the "action". I build it in a way to be self supervising without one single task scheduling mechanism to enable people to write data driven applications.
Its nice to see that im not the only one (ye i didnt expect that to be the case dont worry im not elon-musk crazy) that tries to go such a way.
While i build it for completly different use cases (when i started LLM weren't such a big thing as they are now) its definatly a cool and creative way to use such an architecture.
congratz !