Could you say more about some of these times? I'm curious what it means in more concrete terms.
I only say this because I am a UI engineer who has been in some scrappy situations, so obviously I don't get to change the data model. What kind of structures do you find that "hit the database too hard" or otherwise invoke performance penalties?
When I say "hit the database too hard" what I actually mean is making too many calls to the database. If I have to make three or four calls just to get all the data I need, performance is going to take a hit. And databases are often the worst-performing part of the tech stack, so you have to craft your queries around the performance bottleneck of the database.
So you have to balance your table structure around getting the data you need in as few calls as possible while also only returning exactly as much data as you need and no more. Because pulling extra data out puts more load on the database and shipping more data across the network is slower.
In general, relational databases (RBDs) like Postgres or MySQL are great for making sure your data follows the correct structure (this field is a string, this is an integer, this field in table1 has to be the same as this field in table2, etc) but make it hard to make changes to the data structure later. What happens if that int should actually be a float? Now you have to write a migration that makes fundamental changes to the structure of the data and hope there are no negative side effects.
Someone else mentioned NoSQL databases, which offer a lot more flexibility at the cost of the data integrity that RDBs enforce. If you suddenly want to store a float instead of an int, go ahead. No one is stopping you, you just need to make sure your code is updated to handle the possibility of getting back an int or float (or coerce the value to the right type and pray).
Basically a database is integral to almost all applications but they're complex monsters with their own structure and rules and performance implications. If you're building an application you really have to know the data structure of the final product before you even _start_ configuring the database.
> but make it hard to make changes to the data structure later. What happens if that int should actually be a float? Now you have to write a migration that makes fundamental changes to the structure of the data and hope there are no negative side effects.
This is often stated, but it's not true. RDBMS make it easier to make changes to the data structure precisely because you only have to write migration and after that you are guaranteed to only ever get the data out of database in the new format. Which means your application code doesn't accumulate the cruft of dealing with two possible database data formats.
I strongly disagree that this pattern makes it easier to make changes. It makes it easier to use the data (because you know this data will always be this type), but changing the data is harder than if it did not enforce a type.
I'm not arguing that un-typed data in a DB is easy to use or a good idea, just that it's harder to change the structure of RDBMS if you didn't plan properly from the beginning.
Basically: I think data is hard and I wish there were better tools for interacting with it.
Well, this was maybe a bit entry level description.
Q: "databases are often the worst-performing part of the tech stack" - compared to what? nginx throughput? I find this to be a bit of a strange view, surely business logic is always the slowest part of the tech stack
Q: "What happens if that int should actually be a float?" - how often do you actually need to run migrations versus just extending the data? From my end, I have a small idempotent database schema-maintaining tool, and if I need a new column or a new table there's no need for a migration, and you know your whole stack will interact with the new schema or old schema identically assuming you set sane defaults etc. I've built a lot of medium-quality low-traffic apps so I'm yet to encounter a real-world case where a migration wasn't just bad planning
>I'm yet to encounter a real-world case where a migration wasn't just bad planning
That's basically the root of the issue. Poor planning in your code means you re-write some code. Poor planning in your database means you have to start restructuring data, and if it's already running in production you have to hope you don't accidentally corrupt production data. It's a lot harder to restore corrupted data in production than it is to roll back a code deployment. And the answer to the problem is obviously just spending more time thinking about the proper data structure, which is the entirety of my complaint: I want my data to fit my application, I don't want to have to write my application to fit my data. I don't want to have to think "does this field belong in the Users table or the Accounts table or the [insert table here]".
I'm not sure what you mean by "just extending the data"... if I'm writing a Rails app and I need to change an int to a float, the way I do that is by writing and executing a migration.
As for the speed... a database typically stores its data on disk and is often not hosted on the same physical machine as the web server. Meanwhile the app and web server store a lot of things in-memory on the local server and even when it has to read from disk, it's a local disk attached to that machine. Check these numbers for how long it takes to read from memory (or even local disk) versus reading a remote disk over the network: https://gist.github.com/jboner/2841832
> >If it's already running in production you have to hope you don't accidentally corrupt production data
Or you write use a read replica to transform your data into a non-live DB and validate it before you put it into production, with backups of your final old schema available? Plus I really don't think a migration is that hard. Much harder than having a litany of shitty backends you have to glue together in your front-end app, trust me as someone who's done both.
I mean, I hear you, persisting data is hard, but that's not the database's fault, it's because you pick two of three on data: performance, persistence, and flexibility
> a database typically
But that's not the "worst performing" part of the stack, that's the highest latency part of the stack. Is there some specific reason you can not have co-located web & db servers? Also, is there a reason you still reference 2012 disk numbers when SSDs clearly have reduced all "disk" operations by an order of magnitude?
> just extending the data
What I mean by this is if you start with a minimal amount of columns in your database, and someone is like "we need new property x", it's easy to add X outside a migration - adding new columns or new tables does not require migrations if you don't modify existing columns
So this is my approach. Use as few bits as possible to persist your data, and then you can generally add new features migration-free
I'm not sure if this is intentionally obtuse, but yes "tech" as a slang term doesn't mean "technology", it describes a type of company largely known for the prowess and scale of their user-facing software. This includes such entities as Facebook, Google, Apple, Microsoft, Netflix, and Amazon, but is not inclusive of course.
Tesla, on the other hand, clearly produces technological artifacts - quite impressive ones, at that - but they are not a "tech" company within the colloquial context you've stumbled into
> I'm not sure if this is intentionally obtuse, but yes "tech" as a slang term doesn't mean "technology", it describes a type of company largely known for the prowess and scale of their user-facing software. This includes such entities as Facebook, Google, Apple, Microsoft, Netflix, and Amazon, but is not inclusive of course.
Except I would argue that Apple is known more for its hardware than its software.
Also, I believe most people don't give a second thought to Netflix's software. It's the content they think about first.
It must be very context specific; I'm in IT, in an English-speaking country (Toronto,Canada), and don't even remotely think of "Tech" as "front-end user-facing software only". :-/
What kind of companies do you call "tech" companies then? I can't really understand the objection I'm facing, when people tell you their stock portfolio is heavy in tech do you really think companies other than the one I listed? I live in the Bay Area so maybe things are skewed to "local" companies but I held a similar opinion when I was still in Canada too...
EDIT: Just to hammer it home, search "is faang tech" and weep that "corporatefinanceinstitute.com" is on my side
of course faang is tech. Your assertion was not "faang is tech" it was "faang is all that tech consists of". These are not equivalent, and I recommend you view [1].
That's exactly the distinction I was making. Do you call Ericsson a tech company? I certainly never hear them in that bundle.
Again, "tech" != "technology" in this domain in my view. It's the double-great-gp or so that says it, but "tech" in my view leans towards companies with humungous software outreach and data ownership, so clearly FAANG but also Microsoft, Palantir, data warehouses, that's the kind of thing I put in this "tech" bucket. It's a type of company that probably knows more than it should about a lot of things.
And again, this isn't the "Wikipedia" definition (which itself has context-based caveats...), but my opinion and experiences involve this being the sphere of companies one calls a tech company today. I apologize for my inflammatory tone in the other comment and hope I have explained my point, even if you hold a different experience
I probably would call Ericsson a tech company honestly, though that would be a little iffy.
However, I definitely would call a hardware company (Dell, HP, Lenovo, etc) a tech company, and their software all sucks.
That being said, you have certainly made your point, and with your definition here I can see why you would think that way, even if I disagree. In addition I agree with the sentiment "It's a type of company that probably knows more than it should about a lot of things." Though I don't necessarily think this defines "tech", or that tech is the correct label, I do think these should get there own label.
As for your other comment, eh no big deal. We all make comments more aggressive then we intend every once in a while.
EDIT: I do think there is a difference between "tech" and "big tech", I think that "tech" would be as I described before, and "big tech"/"big data" is what you were describing.
I agreed with you at first, but I recall the Bay Area having one of the earliest shutdowns in the US, and that started March 17. While some companies were definitely sounding the WFH alarm about a week prior to that, I think the March -> April jump would've been more incongruent with April -> May jump unless COVID wasn't the only driving factor behind the August -> March growth in the first place.
Anecdotally, I saw a lot more people ordering online in the last two weeks in March than in late April and on. (And, based on the empty shelves then, probably an equivalent increase in in-person stocking up, too.) Remember that stores were running out of stuff before official shutdowns happened, too.
I'm curious what week by week data would look like.
Definitely one of the more rapidly growing parts of the convenience economy. With the much higher average spend and less time dependence, grocery delivery looks much more palatable as an investment and lasting market.
I think it will end up beat by curbside grocery pickup. What is more palatable: a grocery bag assorted with care by grocery store staff who sort things by size and weight when they bag all the time and spend quite a bit of time selecting and arranging the choiciest fruit in the store, carefully lowered into the trunk of your vehicle when you appear in front of the store on your way home, or a grocery bag left on your porch hours ago by someone incentivized to work as fast as humanly possible and pee into bottles in their high interest leased car while they hustle to the next gig?
I'm curious why we have this view that the in-store pickup order worker wouldn't end up just as "optimized" as the general online shopping warehouse worker.
Well, then you can just work out a deal to not monetize it.
Your objection is of course valid - extra workload on top of a normal job probably has its limits. But the fact one is willing to pay those who are willing to be paid for such an interview doesn't suddenly make it a strict requirement ha.
I must say though, I also view the interview standard I've faced - quick phone screen, 1h tech screen, 4-6h onsite (probably online now) - as adding up to a lot of time very quickly, especially as the "onsite" time is of course awkward when it means you take PTO to go to an interview. In this sense, I'd probably rather have an ~8-12h coding task than a 4-6h onsite. Maybe even a higher factor
>the lack of a standard library in Node.js is one of it's biggest pitfalls
>npm / yarn ecosystem sucks compared to what you get with Python and PyPi
So you claim Python has a better standard library AND a better package ecosystem? That's bold. Care to expand on why you feel this?
I ask because of this. From the perspective of a JS dev who helps Python devs at work, pip seems pretty weak, its publishing story even weaker, and then both languages seem to have all their strength in their package ecosystems (Python: tf, django, dataframes, etc; JS: react, typescript, etc). So now to see you say "npm ecosystem sucks compared to pip" has me wondering what I'm missing.
As someone who's done a lot of python and a fair bit of JavaScript, I'll agree with the preference for the python ecosystem. I guess my perception of the difference is a quantity vs quality thing. My experience has been that python packages tend to be more stable, better documented, and frankly more reasonable in scope. There's less framework churn (Django has been around for what feels like forever, as have lighter solutions like Flask) and libraries tend to include enough functionality to seem with adding a dependency for; the number of transitive dependencies on a typical JS project is alarming. Also much less of the ecosystem feels like it's just trying to make up for an absent standard library. You don't see stuff like underscore in python land.
Both languages suffer from patchwork build infrastructure; interpreted languages are great when you have one file and no dependencies, but as soon as you grow a bit beyond that rules about where to find modules make things complicated. Both languages have developed tooling around this that suffers from being an afterthought, and I'm not sure which is worse.
I guess this might not strictly belong to "the package ecosystem", but I strongly disagree that you can compare the npm packaging story to the pip packaging story. Publishing to npm, hosting your own npm instance, configuring npm to look behind proxies for certain packages, an opinionated `package.json` with real functionality beyond dependencies, a unified entrypoint, wow it actually boggles my mind to hear Python people say pip is even in the same room as npm.
Definitely let's be clear - babel & webpack and the associated ecosystem are a pain, but those are mostly borne of necessity to transpile HTML/CSS/JS from frameworks, and not because node.js itself isn't a decent language.
Finally, I don't really follow all this business about the lack of standard library. Can you explain to me what you mean? What parts of the Python standard library are a part of your everyday toolkit? I've built a lot of web apps over the years and the only time I've ever used lodash is for throttle and debounce.
An npm publish requires you to run `npm publish`, assuming your `package.json` points to the entrypoint of your code. Remember: `package.json` is pre-populated when you start your project with `npm init`.
Also, I was able to run my own NPM instance with a zero-config tool, I expect Python has parity there.
Anyways, pip... Just the other day, I went to use a Python project I had been regularly using, and without any changes to Python, my OS, pip, or the project, it completely started failing. Some errors that the `ssl` module wasn't available. How can a module not be available? There was no easy solution, all to something triggered by nothing. This install was literally done thru a source-controlled script too. It's a pretty awful situaton.
Anyways, NPM has never so wholly failed me, and in the worst cases where it fails it usually has helpful messages directing the user what to do.
That plus tools `npx` make it pretty clear to me, Python is literally version and dependency hell. NPM has had this all on lock for years now
Drug sensitivity is always such a strange thing to me. It strikes me as having characteristics of qualia - to you, going cold turkey on caffeine was apparently monstrous, but I've had the exact opposite experiences. I drink a lot of coffee, 3 cups a day average, but if I stop for a day or three or a week I certainly don't feel any pain, just a different focus.
I stop for a full week every year. I drink 3-4 rather strong cups every day (no coffee after 4pm), but only 1 cup daily on weekends. Only symptom for me when I don't have caffeine is being a little harder to focus, but that's not a problem for me when I don't need to code.
What would Hillary have done that would have avoided this situation? This is a dispute between two private companies, and there's not a clear piece of legislation that would prevent this, especially one that would be approved by Congress.
This seems like a pretty disingenuous representation of the progress made under the pressure to use inclusive language.
When I was young, it was really popular to refer to things via homosexual slurs to show how cool you were. Can you imagine how much trauma such an action imparts on someone who is a part of said slurred out-group when it is widespread and socially accepted?
I think that all social change is imparted through pressure, and the pressure to use language in a fashion minimizing slurring, outcasting, and denigrating structures is important. It's true that in this instance, it may not make sense and may be an intrusion on you and what you feel - and that's okay. It's also important to put the same pressure back that says "this piece of language does not have negative connotations and taking it away is policing free speech".
Surely you see how it can be appropriate or not appropriate to push back given the subject material.
I only say this because I am a UI engineer who has been in some scrappy situations, so obviously I don't get to change the data model. What kind of structures do you find that "hit the database too hard" or otherwise invoke performance penalties?