The tech industry nowadays is no longer about solving real-world problems and making the world a better place, instead it's all about screwing the end user in every single way possible, whether it's ads, stalking and privacy violations, spam ("marketing" as they call it) or just plain fraud where the company is happy to take the money but can't make it right if things don't go to plan and the customer is left holding the bag.
Most tech products nowadays aren't there to solve a real problem and aren't funded by customers buying them because they are good; instead they're funded by some VC scum and they're there to capture the market (or rent-seek) and prevent a legitimate business from starting (nobody can compete with free or below cost).
Technology-wise, we no longer use engineering as a means to an end to solve a business problem. Instead, engineering became its own thing and most companies encourage and reward those who opt for over-engineered solutions, which means you spend more time fighting with dozens of layers of abstractions and chasing the latest JS framework instead of actually delivering functionality. This is mostly a symptom of the previous point where showing "growth" and bragging about your (over) engineering is more important than actual profit.
Unfortunately there's just nothing out there that pays as well so I have no choice but to tough it out.
This describes a lot of software. No doubt. It's depressing and disturbing.
But not any of the systems I've worked on over 10 years. You may not get the absolute top salaries working on more meaningful projects, but there's plenty of them. Here are some examples that I worked on. To a fault, most of these places over-focused on the business and under-focused on exceptional engineering. Nonetheless, there's a lot of money to be made solving real business problems.
- Optometry practice management software
- Insurance benefits processing and customer service systems
- Professional soccer/football training iPad apps
- phone/pbx, Audio and video communcations, tie in with the web
- mapping systems to evaluate water scarcity and agriculture water use
- government benefits systems
Again, user Nextgrid isn't wrong. If you're willing to take say, up to 90% of the highest salaries available, rather than only the top 10% of salaries, there are huge amounts of non-advertisement problems to solve. Rightfully, this comment exposes the depressing cynicism which develops when building manipulative software.
In most all the systems I have worked on its more about screwing the user from a technical perspective than a deliberate business action, though there have been some of those too.
Either way it all comes down to some combination of mental laziness and no ethics. There is no ethical code, that I am aware of, for writing software.
* People (only software developers on security) who self-admittedly have no education, no experience, no credibility, and no practice of the art find themselves subject matter experts and cannot understand why an actual expert would disagree with their hastily formed opinion: https://news.ycombinator.com/item?id=22868620
* Hiring software developers is still somehow an arcane mystery few people can objectively figure out.
I have never seen anything remotely close to that in a place of corporate employment ever. I have been doing this in the corporate world since 2006, but if you count my time as a security analyst in the military than I could push that date back to 2001.
The military does force feed you (memorization verbatim) ethical standards, though, at a huge contrast to the loosey goosey corporate world:
Hiring is hard even in places where everything is quantified, such as baseball! Only a few teams have consistent success signing good value contracts even with all the statistics available.
Baseball is a bit different because there is only one winning team. Those people can still all bat (or whatever) much much better than the average qualified electrician can wire. But the point is that the average qualified electrician can wire. Whereas with the average programming interview candidate it seems to be a bit of a lottery.
JavaScript developers can't interface to the browser
I'm not sure I've worked directly with anyone recently who could pass a DOM test like that. Everything is React and and Axios and the browser (and HTTP, TCP, DNS, etc) is a mystery.
Edit: mystery isn't even the right word really, if you mention any of the other layers I've seen people get actually hostile.
From that particular post, this line stuck out to me:
Only 3 candidates were able to pass this code filter. The people that did pass either did well enough to pass or extremely excellent. The people that failed, maybe 19 of 22 people, all failed epically.
In practice I highly doubt that the population of candidates is actually this bimodal. It might lean heavily below the hiring bar, but I have trouble believing that it's bimodal.
My own personal experience has been that it looks a lot like a normal distribution where the hiring bar is just two standard deviations above the mean (which would imply that you give offers to about 1 out of 25 people who apply, which sounds approximately in the right ballpark).
As a consequence, if your question is producing bimodal results, I can't help but wonder whether it's a good interview question, because knowledge tests exhibit much of the same type of distribution of outcomes.
In my experience, if you give the candidate only one hour, access to documentation only helps with recalling past experience, because one hour is too much time pressure to actually calm down, pick through, and digest documentation on something you haven't seen before. If you knew the DOM functions existed and just needed their names, it'll work, but if you're unfamiliar with what the DOM already offers, even if you could learn it fine on the job, you won't learn it fine under time pressure. I recently adjusted one of my interview questions specifically to reduce the need for looking up documentation because I noticed exactly this phenomenon.
It surprised me just as much. I cannot speak to the capabilities or thoughts of any given candidate. Ultimately the test was a test of using a single standard API that is based upon a standard model. If it were just a matter of instructions or syntax I suspect a candidate would gotten over that by either asking the right questions, trial and error, using a reference of whatever. I think what really destroyed people was a complete inability to perceive the page as a series of nodes all interconnected with various relationships.
> In my experience, if you give the candidate only one hour, access to documentation only helps with recalling past experience
That is only partially true in this case, because the candidates knew coming in they would have a code filter accessing a page using vanilla JavaScript. I cannot remember if I specifically mentioned refreshing on the DOM methods or not. I might have. This was 8 years ago. But they had at least a day prior notice to catch up on how to interact with a web page using JavaScript using only their code. The time pressure would have applied to the specific tasks asked of the candidates.
I think what really destroyed people was a complete inability to perceive the page as a series of nodes all interconnected with various relationships.
Fair enough. :) I've experienced a similar issue with interview questions where they often floundered on the question because they hadn't ever spent time digging under the surface. For example, I notice that a lot of (typically junior) developers seem to only think of JSON as a representation for hierarchical objects but never really spent time thinking about the typed-ness of its keys/values or the limits of what JSON can actually represent. An interview is a bad time to be confronted with that realization.
I cannot remember if I specifically mentioned refreshing on the DOM methods or not.
Well, it was a long time ago, so I'll give you the benefit of the doubt and just talk about what I've experienced. I go back and forth about how specific I should be about the type of questions I might ask. On the one hand, it seems kind of foundational that someone writing web frontends is aware of the DOM and how it works, even if they never directly interact with it. On the other hand, I think questions need to control as much as possible for the "preparation" factor because some people simply interview better or have more time to study.
It doesn't seem too crazy to me today that a competent FE dev these days might have spent their whole career in React, and therefore they don't really have a strong grasp on the DOM. If you had asked when I first started doing web FE development, that would have been crazy! But frameworks got a lot better and a lot more comprehensive since then.
I recently had an interesting experience asking a question involving dealing with UTF-8 encoding. I figured "everyone knows about UTF-8 right?" I retired the question pretty quickly once I realized that bit manipulation is not actually something you have to be even remotely familiar with in order to be a functional developer today in the vast majority of programming tasks, and therefore it wasn't even a good fizzbuzz-type test.
Yeah, I have an in-depth knowledge of the imperative DOM APIs, but I can't remember the last time I actually needed to use them in my day job. React provides a very good abstraction that one doesn't need to break out of very often for a lot of work.
As someone who worked in the frontend space, and now doesn't, I think this has much to do with the fact that "anyone can learn to code" but often don't have any formal background. Lots of bootcamp grads, lots of people hearing about fat programmer salaries trying to get into the industry, people completing udacity courses and throwing buzzwords on their resume.
My experience in frontend feels like everyone learns ad hoc without digging deep into basics. Overall, I felt if you have aptitude or real passion for the work, you would just get a computer science degree and be attracted to other parts of the stack or other problems to solve.
This obviously doesn't apply to large tech companies which make it a point to hire talent (frontend, backend, whatever).
> Hiring software developers is still somehow an arcane mystery few people can objectively figure out.
This is actually one space where I feel the frontend world shines. Almost every frontend interview question has been very practical.
So this is a philosophical argument, but lots of those industries indirectly screw their customers. Optometrists who sell frames generally sell brands from companies like Luxottica who have made eyewear really expensive. The entire insurance industry is one of the major reasons healthcare is expensive and ruins the experience people have with patient care. All of this various from country to country of course.
I could go on, but sometimes your choice is working on a product that either screws the users directly (e.g. ads) or indirectly. Take your pick of where you want to draw the line.
That’s an unfair conclusion. Those examples were pretty specific and I, at least, see no reason to generalize (except maybe the insurance one), except perhaps along the lines of “serving exploitative monopolies is bad”
What I meant is that you can find an example in nearly every business of some person (or group of people) getting "screwed over", which is of course subjective. So subjectively, you could make the case that every business is "exploitative" (like the other replier who claims every capitalistic business is exploitative).
In fairness, there is no ethical consumption under capitalism. If you take that to its logical conclusion, then there is no ethical software development that supports capitalist endeavors.
If you take what I said to the logical extreme then sure.
What I was trying to say was that it's very black and white thinking to say something like ads = exploiting users and healthcare = not exploiting users.
Since Facebook usually comes up when people talk about unethical companies, let me use them as an example. Let's say there is a team that improves ad targeting by 5%, helping to reduce the number of irrelevant ads people see. Let's say there's another team that reduces data center electricity's utilization by 5%, which helps the environment.
Is the ad team more or less ethical than the data center team if they are basically both supporting the same end goal?
I don't work for Facebook if that matters. And I have no answers to any of the questions I posed above outside of my personal preference to not work in online advertising unless I need to do it to survive.
Luxottica is an unusual case. Most monopolies aren't that durable and Luxottica has challengers now. Also, the law in the US which says you can't buy glasses unless you have had an eye-exam in the past year is an anomaly amongst nations.
The functionality delivered by products is not screwing people unless you're talking about surveillence capitalism ala Google et.al. Capitalism and the pricing of goods is how we incentivize people to bring things to market and more importantly how we allocate resources. No one, anywhere, through any scheme has ever replaced the market for performing these functions and thereby driving civilization and invention forwards.
People keep believing they can (socialism/Venezuela), or give up on the project entirely after decades of trying (communism/ China / Russia). Maybe one day someone out there qwill have a genuinely new idea which works. Until then you have a right to feel good about contributing, in whatver little customer support / feature adder / managerial capacity you do, to the forward progress of society generally and the amazing, rocket-like increase in wealth absolutely everyone on this earth is experiencing relative to the past 30,000 years.
You've done well to stick with these things, and I do agree with your point, but I think almost any company can fall afoul of these practices...
- Optometry practice management software... targeted at maximising in-store sales with prescriptions that are difficult to use elsewhere, encouraging dark-patterns in sales tactics, etc.
- Insurance benefits processing and customer service systems... that attempts to minimise payouts, potentially by adding burden on the customer through the customer service process.
- Professional soccer/football training iPad apps... arguably sports doesn't bring as much benefit to the economy as equivalent spending elsewhere.
- phone/pbx, Audio and video communcations, tie in with the web... for premium rate phone lines that users with little choice are forced to use, such as those installed in prisons in the US.
- mapping systems to evaluate water scarcity and agriculture water use... for Monsanto.
- government benefits systems... that optimises to minimise payouts or contains unnecessary checks and hurdles.
These are great points. I'm reminded of a comedy TV show, The Good Place, where it turns out nobody has gone to "heaven - the good place" for a long time because there are too many unintended consequences of a person's actions. Buy an organic bamboo T-shirt? The company dumped waste into river. Vegan? You produced a bunch of single plastic plastic packaging.
It highlight the need to be diligent in daily decisions day-by-day. And when working somewhere that's doing something borderline, or totally unethical, working to move it in the right direction. I hope I can be aware enough to create a net benefit. The problem is knowing enough of the consequences of any action to evaluate if it's a net benefit. That's really hard, but I have to believe that toiling away _trying_ to make net positives happen will _actually_ result in net positives happening. Doubting I could ever make net positives happen leads to cynicism and ensures nothing good will come of my work.
(and thank you for bringing up point about benefit systems - that's the next gig for me, I will work to not be part of the problem)
Nice one. Just think about the user/consumer in any of these things. Does the thing I'm building help them, or only help the company I'm working for?
Be pragmatic. If the company provides a good service but doesn't survive, then that good service no longer exists! Some amount of optimisation for sales is likely necessary, but I think there's a line where things turn user-hostile, even in subtle ways.
Curious, how do you get jobs/contracts around these examples? My understanding is that most of these industries already have the talent they need (or have a preferred supplier they work with).
I freelanced for a few months for a local high-end cabinet maker. I helped them integrate their online ordering system with the automated fabrication equipment. It was, by far the most I have ever enjoyed programming. Coordinating with super skilled crafts-people to help them eliminate the redundant parts of their work was just so cool.
And by weird coincidence to this post I've recently been browsing around trying to find any other type of traditional businesses looking for equivalent help. But I've mostly found boutique places that wouldn't have need for a full-time developer.
Any tips on places to look for opening greatly appreciated.
I had a similar gig where a small local business wanted to integrate their Shopify store into their existing Django app. It was very easy and no BS to deal with like Kubernetes or Javascript build pipelines.
The problem is that I find it very hard to find these businesses, and indeed most of them don't actually have enough work to fill more than a couple of days so you would need a reliable source of these gigs if you want to survive (you can't charge insane amounts to compensate either as these businesses just can't afford it).
That type of work is too infrequent for most companies to justify hiring a dedicated developer.
You should work for a system integrator if you want to do that kind of work with regularity. The integrator contracts you out to those companies as needed, and you develop a niche and maybe customers start asking for you by name. In the best cases the integrator is your employer and offers full benefits, as well as absorbing the costs of downtime.
That business model can also be abusive for obvious reasons. I can recommend a good company if you're interested.
“Up to 90% of the highest salaries available”? I’ve worked on one of these and you’re more likely to get 60-70% of what you could otherwise be earning in some mediocre fintech company, let alone top salaries. Which is a much harder sell for most people.
You could always switch to a company that doesn't make its money with ads. For example I work for Snowflake and I don't feel like I'm building stuff to screw the end user at all. I just make our database better. I also worked for Siemens. There too I didn't try to screw any users, I just tried to make trains run cheaper, faster and more reliably.
Making its money with ads doesn't mean anything anymore.
There are plenty of companies that have paid products and still do user-hostile things including stalking, spam, dark patterns, etc.
I paid full price for a Playstation 4. I still had to opt out of "enhanced data collection" and spend 15 minutes disabling every option on their bullshit attempt at a social network. Even the games themselves had their own telemetry crap I needed to disable buried deep in some advanced settings.
Spotify still insists on ratting me out to Facebook by loading their SDK even if I'm paying for the app. They also interrupt my flow every so often with in-app popovers about some stupid feature I don't care about.
A client I worked for that has a paid app had an insane amount of analytics and marketing SDKs in the app (including Facebook of course), so much that I actually refused to install the app or use the product myself even though they gave me a voucher.
> There are plenty of companies that have paid products and still do user-hostile things including stalking, spam, dark patterns, etc.
This is a good point and I'm glad we're finally seeing some pushback. I would have a problem working for a company that does unethical things and I was surprised that so many people don't care. And before you tell me that people don't have a choice — perhaps some don't, but that certainly doesn't apply to all the crowd that moves to Silicon Valley to work for Google.
The examples of paid products you used to support your point are not very good. The PS4 console was sold at a loss and Spotify is barely profitable. In short, you might have paid for them, but neither company made money from you based purely on those purchases. Both companies perform data collection to guess what, generate more ad revenue (through lots of channels).
Sony isn't having money trouble, they're huge. That's a sorry excuse.
And Spotify doesn't make much money because most of their users are freeloaders. Their poor business model should not become the problem of the paying users.
>I also worked for Siemens. There too I didn't try to screw any users, I just tried to make trains run cheaper, faster and more reliably.
So I say this based on personal experience. The Siemens' sales force uses their size and influence (especially with existing customers) to absolutely screw over customers and smaller competitors with better products. Siemens is no different than many other big companies who can use all sorts of unethical (but legal) tactics to sell.
It's very possible to work as an engineer in a big company and have zero idea as to what everybody else is doing to screw people over.
Not only that. Siemens bribes people from the governments in third world countries to get the work from the government and totally destroys the local competition an the taxpayers.of those countries.
> You could always switch to a company that doesn't make its money with ads
I always thought that I would be able to switch to any programming job I wanted, and that the only decision would be between making a lot and roughing it up (like the GP), or working for a better cause for less money (as you propose).
But now I find it -- after 25 years of professional experience -- nearly impossible to find any job. Maybe it is due to the coronavirus, or the budding depression (or both), but I've never seen the market so slow... :(
This is my observation as well, and it was even before the coronavirus. The majority of tech jobs in London are bullshit startups chasing after AI, blockchain or the latest JS framework. It's very rare that a real company hires (presumably because they have all the talent they need and don't need to hire for the sake of "growth") and when they do it's often a bad fit for my skills (Python/Django).
That strikes me as odd because I've found that nearly every company of a certain size has a programming team these days.
You might have to change tech stacks and rebrand yourself to appeal more to enterprisey jobs, but that's more of an issue of marketing than anything else. It will be a small transition, but once you do that you'll be fine.
All the permanent Django jobs I see also have over-engineering such as insanely complicated front-ends (that actually perform worse than good old HTML) or Kubernetes (when the entire product could be served off a single bare-metal server) or similar, not to mention they are BS startups doing user-hostile things.
Here in Boston I'm hearing that the market is still pretty strong. Lots of smaller companies are having trouble but bigger ones are hiring like crazy for senior and higher roles.
I also think B2B is a much less predatory place - most of the businesses have real money, want a real thing to help them make more of it, and are engaged in the process of making that work.
There's plenty of bad actors (like any biz) but most of the time when I ship features everyone is happy, and no one is taken advantage of.
I went to Stanford undergrad and honestly, the Bay Area tech scene (and Seattle from what I've noticed) looks unappealing to me. The culture is a turn off.
I'd much rather have a 40 hour a week job somewhere else, where everyone understands it's a job and you can be your own person at home. It's hard to articulate, and I could write much more about it, but it's almost like the identity you have to don to fit in there feels too all-consuming.
I don't want to pretend I'm changing the world instead of just installing a new social hierarchy run by the meritocratic elite. I don't want to pretend that programming is my only hobby -- I want to do other things than learn new Javascript frameworks at home. I like my nice 3 bedroom house which is just minutes from downtown and is less than 3x my annual income.
It's admittedly an outsider's perspective but the only reason I would go back is to save up money for a few years then go somewhere else.
I think you would be surprised just how many people see their job as a means to an end in Seattle/Bay Area, especially outside of the startup world. At FAANG and other big companies I think there is actually very little Kool-aid drinking compared to the smaller companies. And at a subset of those companies, people work normal hours and don't brag about things like working on side projects at home. Especially many of the parents I've worked with have been great engineers but at the end of the day just want to go home and spend time with their family, and it's fine from both a career and social perspective.
1) I switched from being a software engineer to a data scientist in 2010, in part because of the culture. I totally get not enjoying the culture, though I do recognize how much it varies from person to person and company to company, so I try not to stereotype. But on some level, I get it.
2) We're opposites in some way. I've been writing code since I was 8 years old, and enjoy it. I'm the type of person who loves to find a fun and challenging problem and dive in so much I don't turn off. However, people like me are rare. In the last 15 years in the Bay Area, I've only worked with one other person who fit that criteria. Everyone else treated it like a 9 to 5 job. Me, I've never been one to judge or think negatively of that. That's how people are. It's normal. I get there are problems with SE culture, but some people being super passionate about what they do, to me isn't one of them. I don't believe I've ever annoyed anyone, bothered anyone after hours, or done anything unreasonable, and if I did I would appreciate people letting me know in a 1-on-1 fashion. The last thing I want to do is bother my coworkers.
3) You're on ycombinator, the center of Bay Area tech culture. Frankly, I'm baffled you like hacker news, for the people who like this kind of stuff after work hours, yet at the same time are anti that. YC is the heart of tech passion, and if you don't like it, why are you here? Not judging. Your comments and views are valuable. I'm asking out of genuine curiosity.
Well, don't paint with a broad brush. I love solving problems too, and I love learning new things. I too played with computers as a kid and my middle school friends and I built websites for fun. I wasn't exposed to real programming (beyond neat Javascript tricks) until CS class in high school.
However, science has always been my #1 thing, and programming is my probably my #3. Interesting, but not the only thing I want to do. Working 40 hours a week at it is enough for me. I still love learning new things and getting better at the craft, but when I go home, I have other hobbies.
I'm hacking my treadmill with Arduino for fun. I'm brazing metal. I'm riding my bike more. I'm reading a fiction book that just came out. I just read a book about world development. All of that is a lot more interesting to me than going home and sitting at a different desk doing more programming, you know?
And while I'm not offended by your comments and this is entirely non-personal -- really, I'm just curious as well and enjoy talking to people with different perspectives -- the attitude that if I'm not always programming, I'm "less than" than other programmers seems oppressive to me. I'm still smart, I'm still curious, I'm still good at my job. But there's still the perception that if I am not programming more, I am falling behind.
So the solution for people like me is to pretend to be someone like you, who loves programming above all other things, and it's mildly exhausting. It's great that you love programming so much and I kind of envy you. But not everyone needs to be exactly like you to be a worthwhile contributor. (Again, not personal, I just enjoy putting these thoughts into words.)
>the attitude that if I'm not always programming, I'm "less than" than other programmers seems oppressive to me.
Who says that? It's so backwards from everything I'm accustomed to in the bay area. (In the bay area I've done 10 years of data science and 5 years as a software engineer before that.) Maybe it's because I didn't go to universities out here and it's university culture? People at MIT can get very passionate about their projects.
I've had companies (many of them) advertise to me saying I should join them because they have a healthy 9 to 5 work culture. But what company doesn't? Maybe a young startup, but you sign up for that at that point.
Everyone I know who regularly works more than 40 hours a week (who isn't a CEO) is almost always because their boss asks for X quantity load of work done, which might be 50 hours of work, so they do the 50 hours instead of communicating they can't do that much within 40 hours. Sometimes it's because they're a junior and are afraid of not fitting in and are afraid of getting fired, but sometimes it's a senior who has their tech down, but not their communication skills down. It happens, and it's sad to see, because it almost always leads to burn out on their end. It's not a good thing.
I love what I do, but I do 40 hours a week. Want more? It better be a rare event (crunch times are a real thing in some industries) or you need to be paying me more, and I'm not cheap.
>So the solution for people like me is to pretend to be someone like you, who loves programming above all other things, and it's mildly exhausting.
I love programming, but it's just a tool. It's like saying you love hammering. It doesn't make a lot of sense when you take a step back. What I love is the projects I do. You mentioned hacking your tread mill, so it sounds like you do more tech hobby stuff after hours than most software engineers.
Frankly, I rarely do projects outside of work, despite loving what I do. Why? Because companies give me inspiration to do specific tasks that I love. Without working I have a hard time finding things to do. If I could find infinite things to do, I'd be a CEO, providing things for everyone else to do. The only people who truly don't turn off, end up being CEOs, not software engineers. (Unless they don't know how to capitalize, or some other edge case, of course.) So it's unrealistic to believe you have to be doing non-work tech to work in tech. That's just absurd.
Also, I can't say from country to country, but within the US software engineer culture, while it varies from company to company and team to team, is generally somewhat similar across the country. On the NYC side, working more than 40 hours a week is common (eg, quant work), where on the bay area side only working 40 hours a week and turning off (unless you're at a startup) is common.
Is it possible this has to do with too small of a sample size? It's easy to stereotype. We naturally do it unless we learn otherwise, sometimes through learning statistics, sometimes through other routes.
> You're on ycombinator, the center of Bay Area tech culture
Citation needed. I think YC users think they are the center of tech (or the universe?).
What would be that actual proof that it was? Do we know how many users HN has? What % of VC actually spend time here? What % of meaningful founders actually spend time here? Ya the CEO of github will drop a post on a major thread concerning his company, but so what?
Honestly the more time I spend here the more I see two notable demographics: cynics trying to feel superior (like myself), and people trying to justify their own salary and lifestyles via a fat FANNG paycheck.
P.S. and college students / recent grads who know everything.
It could be trending away from that, but historically https://www.ycombinator.com/ is the center of bay area startup culture, and then those startups grew into bay area tech culture as a whole.
Ycombinator comes from Stanford which centers around creating CEOs, creating most of the companies in the bay area. Most non-yc bay area companies still have Stanford at its roots. I don't have a direct data point, but almost every company I've interviewed for in the last 10 years and have worked with, their CEO or CTO or some other c-suite "club" is strongly tied to Stanford and from that directly or indirectly tied to YC.
It's a multifaceted question and has a multifaceted answer, unfortunately.
There is how to get into a data science job, and how to make that switch.
There is how to get into doing data science as a hobby or a skill/experience, and how to make that switch.
There is how did you make the switch to data science. That is, what is your story?
And there is the deeper dive, most people do not consider enough: "Will I like doing data science work?"
My story:
When I was a teen, I wrote a stock market bot that was pretty profitable. That may be a quant researcher job and skillset, but it is a kind of data science too, as data science is a research position, where creating models is part of the job spec, identical to a quant researcher.
So I got into it, out of hobby, not out of taking a class or reading a book, but by being infatuated with the plots I see in the stock market, seeing patterns everywhere. When I'm doing a 9 to 5 I have to put a timer on when I'm looking at plots, or I can easily spend days just taking in information. I just like it, a lot.
In 2010 I applied for a job as a software engineer, passed all the interviews, was going to do another SE gig, and then I was talking to a manager during an interview and I said something like, "With enough time and effort, nothing is impossible. That's what I love about programming." So, he threw a curve ball at me. He said something like, "We have a team of roughly 15 people classifying web sites, labeling if the site is porn, education, business, and so on. There are a little over 60 categories." This team worked on the web filter schools and dmvs all over the country paid a service for, to limit what websites people can go to.
I jumped on it, "I can automate that!" So he started a new team and put me on it. What did I do? I looked up how SEOs work and how Google's search engine worked. Certainly they were classifying websites in one way or another, so it was just a matter of research and figuring it out.
...in the end the project succeeded. What strongly surprised me at the time was how it ended up categorizing websites to a higher degree of accuracy than the people we were paying.
That's the non-business side of data science in a nutshell. It's figuring out how to do something often suspected or assumed to be impossible, and making it possible. It's not just researching how to do something, but using cutting edge research from published papers and often doing the next that no one has figured out yet.
Programming is a tool that automates thoughts. If you can figure it out you can put it into code. If it's a complex enough of a problem you can use ML to automate the writing parts of that algorithm you could write by hand. ML is often thought of as a way to generate a process, that can't be written by hand, and to some extent that is true, but only in the sense that it would take too long to write by hand. Eg, where writing by hand would need an if statement for each of the millions of hypothetical scenarios.
ML is also useful for giving insights. You have a hypothesis -- an idea of how to do a task -- and you run ML over it, and either it solves the problem, or it doesn't, and what it comes up with that is incorrect will always teach you something, so data science is often an iterative loop, where you put some ML in just to see how the data varies from your hypothesis giving better ideas in how to write a better model.
Sometimes someone will say data science is an analyst + software engineer, but that's not quite right. An analyst looks at data, plots it, organizes it, and then reports on their findings about what is to management. A data scientist looks at findings and applies predictive analytics to it, finding solutions to potential problems. Eg, "We'd like you predict future customer churn, and propose solutions that will eliminate this potential future customer churn, before it happens." This is done through identifying or classifying what the causes of customer churn are, then finding solutions to those causes (hypothesis), then writing some software that automatically enacts that solution. Then you can look at if that solution is working, and validate your hypothesis or invalidate and learn more of what is really going on. Hopefully that makes sense.
In summary, if there is a task that other software engineers can't figure out how to do but you can, regardless if you're using ML or not, it may be data science. (Keep in mind, software engineers can use ML, so while data scientists often use ML, ML is not data science.) The path from software engineering to data science is usually just that, taking what is deemed impossible by those around you and making it possible.
How to get into DS as a career: As a software engineer, your best bet is to transfer internally. It's rare for a company to higher a fresh data scientist who previously was a software engineer. Data scientists can help all companies, from startups to large companies, so you don't have to be at the "right" company to switch, you just have to find a problem that could benefit the company, advertise your hypothetical solution to management, and start doing it. Data scientists are the ones finding projects to benefit the company, and doing it. They're usually not being given projects from management like software engineers are, though ofc they can be given projects too. Keep in mind: I'm a lead data scientist, and my first data science roll I ended up leading a data science team, so my views are biased towards my own experience.
A friend of mine works at a B2B business. Every week they have a "lead forensics" meeting where they somehow figure out which companies visited their website and then spam them with unsolicited sales calls/emails.
Spam is not sales, and you dont get that many conversions by selling to people who didn't want to buy your thing in the first place.
Most of the enterprise sales cycles I have worked under go from 3 months to a year anyway, and the implementation time might take 8 weeks to a year+ - so drive by spamming doesnt really get results.
I think DataDog does this. I've vowed to never advocate for their products because of their relentless salespeople and however they got a hold of both my business and personal emails.
I've been at companies without ads that still screw the users. At one they had us remove the unsubscribe option, for example, to force users to call and argue their way into a cancellation. The call center was graded on how many users they got to give up.
> Instead, engineering became its own thing and most companies encourage and reward those who opt for over-engineered solutions, which means you spend more time fighting with dozens of layers of abstractions and chasing the latest JS framework instead of actually delivering functionality.
I've been thinking about this in relation to the concept of incidental complexity lately. Every year there is more and more tooling and configuration to do before actually coding anything.
I have always considered myself a full stack developer, but each field gets more specialized, and when issues arise I'm just not on top of the suddenly very complex nature of the surrounding ecosystem, even though the goal is to eventually write the same code in the same language I normally use.
>Every year there is more and more tooling and configuration to do before actually coding anything.
I've become increasingly worn-out from this as well. It's strange that as an industry we parrot the "premature optimization is the root of all evil" line, yet we build out our tech stacks as if we were all part of FAANG.
I am going to start showing everyone the Stack Overflow architecture, a beautiful example of scaling up rather than the monstrosities generated by scaling out. There is the promise of cheaper hardware, but the developer costs must cancel this out, no?
>Technology-wise, we no longer use engineering as a means to an end to solve a business problem. Instead, engineering became its own thing and most companies encourage and reward those who opt for over-engineered solutions, which means you spend more time fighting with dozens of layers of abstractions and chasing the latest JS framework instead of actually delivering functionality. This is mostly a symptom of the previous point where showing "growth" and bragging about your (over) engineering is more important than actual profit.
And yet, some kind of new and improved framework gets upvoted to the top of this website every week.
Seriously though, it truly disgusts and depresses me to browse Github and see projects which could have been accomplished with (literally) a single file being split into 10, 15, 20+ files, all wrapped up in some god-awful package managers.
No, developers; your JavaScript snippet does NOT need to be accompanied by two dozen .grunt, .yaml, .composer, .gulp, .jenkins, or .fellatio files.
I have been dealing with this at work today. Docker on Google cloud with a VPN. A two line change took at least half an half an hour to test out. Then log into the container and I can't even run less on the logs to try and work out what was wrong. I can't even remembr the problem that docker was supposed to be solving but someone added it in.
I can relate to your point of view, which is why I don't work for "the tech industry", but instead have my own SaaS. This lets me stay close to tech (I love programming in Clojure and ClojureScript!), do what I enjoy, and not have my soul sucked out by evilness and stupidity of today's adtech world.
I have a product which solves a real problem and I'm getting paid by customers who appreciate that their workflow is somewhat easier with my tool. No tricks there.
So, you do have a choice. I'm not saying it's easy or quick, but the choice is there.
How do you propose finding those real problems to solve? I know that every problem I encounter in my life (or work) has already been solved 100 times over.
If I were to start again, I would look outside of my tech bubble. Go see a local workshop, baker, restaurant. Look into niche industries. There are plenty of inefficiencies and unsolved problems there. I think the best ones are where you deal with physical things (e.g. not just advertising or getting customers).
There are plenty of unsolved problems.
An alternative is to look for a problem that has been "solved", but not in a certain niche. For example, inventory control and tracking production seems to be a "solved" problem, but it turned out it wasn't a "solved" problem for small/medium-scale electronics manufacturing, which is the niche I'm in.
Yeah in honestly shocked at how many niche problems. I encounter that have ready solutions. If I ever find a truly unsolved problem, in likely to assume that its just too hard to properly solve at the time.
Why do you think you need to find a truly unsolved problem? If all you want is a profitable, sustainable business, look at what companies are already paying for, and build a product that solves one of those problems. It doesn't have to be a carbon copy of an existing product. You can differentiate by price or combination of features. Or you can target a different niche. Competition is a signal that there's money to be made in the market.
It won't be easy, and you will have to learn sales & marketing. But it is doable. If there are X companies in the market, there's likely a place for another one (as long as this isn't a winner-takes-all type of market).
Just find a pain point. One of my products sells just because people really don't like certain aspects of my biggest competitor. So I capitalize on that and make migration easy.
So solve it for the 101st time, but better. Everything is broken and nothing works. So, build something that isn't broken and does work.
I just found out yesterday that AWS Amplify doesn't have a manual deploy button. It's only job is to build and deploy an app and there is nowhere on Amplify itself where you can press a button and kick off a build! Junk like this is everywhere now. It's a golden opportunity for people who know what they're doing.
Your points remind me of something I've been saying for a while: We don't have a shortage of developers. We have too many developers and so they are all working on the same things for different companies, or on boondoggles.
(Also, I would really, really encourage you to invest some time and effort into living below your means. It's easier to retire or take a more fulfilling job when you only 'need' 75% of your salary and you already have a big chunk of the surplus sitting in an account somewhere. On the flip side it's made me complacent wrt to demanding raises I deserve)
And here we have the product of today's VC backed tech.
There is a whole other world of bootstrapped/profitable businesses out there where these things aren't a focus or problem. You might not even have to take much of a pay cut.
You have a choice. These companies pay so well specifically because of all of the terrible practices you mentioned. You're currently choosing to prioritize your lifetime earnings over these other priorities, which IMO can be a fine choice to make, but it is a choice.
Get out of the VC backed world then. Plenty of stodgy old enterprises that make actual useful stuff need good IT talent, and they pay well enough. These environments come with their own set of issues, but I've never felt like I'm working for companies where I am scamming the end user.
Just stop working for shit companies producing stupid shit, money isn't everything.
Even better, get out of the private sector (if you can). I'm very happy with what I'm contributing to the world at the BBC, and I'm paid 'well enough'.
The BBC might be an exception here but typically government projects always go to some BS consultancy that will take lots of money and deliver a shitty solution, or sometimes the problem or spec itself doesn't make sense but you're not allowed to talk to them and clarify/rethink the spec because reasons.
There are some notable exceptions aside from the BBC in the UK. The Government Digital Service is held in fairly high regard. I’m not sure about the states. There are no doubt examples at the BBC of what you describe, but mainly it’s pleasant smart people building useful things. There’s dysfunction, but it’s bearable.
> The tech industry nowadays is no longer about solving real-world problems and making the world a better place, instead it's all about screwing the end user in every single way possible
I think you're overstating it. This is true of much of the industry, maybe even most in certain bubbles like San Francisco, but it's far from all.
> Technology-wise, we no longer use engineering as a means to an end to solve a business problem. Instead, engineering became its own thing and most companies encourage and reward those who opt for over-engineered solutions
I think there's more truth to this, although I still think it's far from all of the industry. There's also a little bit of a good reason for it: tech companies have figured out that when you let engineers work on things they find interesting or exciting, they tend to work much harder and longer. In some sense this isn't a bad deal for those programmers either, if they're getting to do something they love instead of something they hate.
What it does lead to is a weird situation where doing something in a way that's only 80% efficient in terms of effort because of over-engineering, may still end up being economical because it leads to 200% effort from the people working on it. I think this is the root of a lot of the technology decisions being made today. Which, again, is really weird and seems questionable, but it also seems to work out pretty well for all parties most of the time.
Moralizing that, scumbag vcs this, screwing users that. Unfortunately although everyone else is terrible and I'm a victim, I'm not willing to stop any of it because I love the money!
> The tech industry nowadays is no longer about solving real-world problems and making the world a better place
Your complaints are legitimate problems with the industry, but it should come as no surprise that a free market does not always make ethical decisions.
> Unfortunately there's just nothing out there that pays as well so I have no choice but to tough it out.
You always have a choice. Nobody ever promised that a choice which "makes the world a better place" would necessarily be the one that's best for any individual's own wallet. It usually isn't.
As long as tech workers continue to choose to do work purely based on salary, even when it "screws the end user in every single way possible", employers are incentivized to continue to pay them to do so. The job market works both ways.
It's hard to muster much pity for someone who uses "toughing it out" to describe making a great salary and (by their own admission) screwing their customers.
Maybe join my company, or another company like us?
We're funded from customer revenue. Privacy focused. No VCs, so no moral flexibility needed on our end to meed VC expectations. We don't do any dark patterns. All remote.
There's GitLab, Wikimedia, Cockroach Labs, etc.. there are opportunities out there if you make it a real priority to work at such place.
> we no longer use engineering as a means to an end to solve a business problem. Instead, engineering became its own thing
I hadn't though about why we over-engineer things this way. But I agree
I might add, based on the over-engineered (and, in general, badly designed) software I have worked with, that it usually comes about because it's never possible to change any aspect of the underlying architecture with the purpose of simplifying things.
We wind up with over complicated 'solutions' because we are simply unable to go back to simplify anything ever for any reason. The 'stake-holders' just don't think it's priority.
So all that's left is keep stacking clever solutions on top of bad decisions such that every new clever solution makes it even harder to go back to simplify.
In my case the problem is less about changing things because the product became over-engineered after several years, and more about products starting out over-engineered because some idiot wants to put React and Kubernetes and "managing a large team" on his resume when the business problem at hand could be solved by a handful of PHP files sitting on a shared host (not saying you should go to such extremes, random PHP files and shared hosts are bad for other reasons).
> Technology-wise, we no longer use engineering as a means to an end to solve a business problem. Instead, engineering became its own thing and most companies encourage and reward those who opt for over-engineered solutions, which means you spend more time fighting with dozens of layers of abstractions and chasing the latest JS framework instead of actually delivering functionality. This is mostly a symptom of the previous point where showing "growth" and bragging about your (over) engineering is more important than actual profit.
This seems to vary company-to-company. I've seen the extremes on both sides of this. I recommend finding a new company that shares your views on tech-debt and pragmatism.
Two threads from replies here that are helpful: look at B2B SaaS or join enterprise IT. There are a lot of companies that are trying to modernize their tools and adopt agile ways of working, and if you're coming from a startup background you will bring a really valuable perspective and set of skills.
I know Fortune 500 developers who are working on super cool DevOps enablers and internal libraries using modern tech. The good places are not your dad's IBM WebSphere configuration shop anymore.
I worked for a good B2B SaaS, and while I was able to tell myself I'm not doing anything unethical or immoral, and while some of the customers were indeed very valuable (in terms of value to their customers) Fortune 500 companies - a large chunk of customers were adtech, dodgy mobile gaming, etc.
So you're still just providing infrastructure for these guys and not really contributing anything useful to the world, you're just one level removed from that, and hopefully you have enough customers who actually do good.
If the bar is "I don't want any code I write to be used for purposes which I do not personally endorse" then yes, it's probably a good idea to go work for a mission-driven non-profit and never make open source contributions to anything ever again
SRE/ops especially feels like a huge job security grift.
Every cloud provider is wrapped in HTTP REST APIs but the cloud era version of a UNIX neckbeard has a DSL or CLI tool to peddle.
Sorry, gang. Doing just fine with a monorepo of JS, Python, and Go packages that wrap the APIs for a variety of providers. New infra is just a new language dependency file away, and I don’t have to learn your semantic wank.
So glad you had a job at FAANG, but outside ITs reality distortion field, we’re plenty productive and you’re just another human.
My outfit has five separate teams to protect us helpless devs from SRE/Ops cloud CLI's and dashboards: devops, appops, SRE, infosec, and IT technology. The things is cloud providers and cloud wrapper companies all pay bribes to managers. The company becomes all about SRE/ops instead of app development. The infosec team is currently running the company's cloud strategy, lol. App developers are not even allowed to deploy code. Naturally though, on-call pages still come to us. I just look at the errors and think, "that's nice, whelp I can't deploy anything so I will close this." I would quit but Covid-19.
I would add there is a conflict of interest in the software industry. Big companies which could invent products to (really) optimize software development would not do that because they will have less developers (business channels at the end) to sell their offerings. Most developers around the world are wasting time doing the same thing over and over instead of having good secure, robust frameworks and UX/UI builders/tools.
I think this a biased point, speaking specifically about startupish business models, there's a lot of different models around the world in a lot of places. There's work to be done with critical applications or slow paced startups, or niche applications on premises. All of them pay well too, maybe you are just saying what you are seeing, but there's a lot more options to it
I agree with everything but your last point. Having worked on software projects where nobody was thinking about how things are done and they were only thinking about deliveries, I can say that you don't need more than 6 months to make the solution a completely unmaintainable mess.
An example I often see is database normalization. Why use 5NF when we can simply do with 3NF? Fast forward this a few months later and you end up doing pattern matching on blocks of text instead of querying indexed columns.
Ahh, it's far more uncommon for me to feel like a database is over engineered. It's much more common to find a codebase has 7 levels of indirection but the database fails to accurately and completely represent their business domain.
When I do feel a database is over engineered it is almost exclusively adding complexities to the database to handle a contrived or rare examples. With the most common being "splitting one table into 2 because a one to one relationship is actually a one to many once every 5 years", or "adding a join table because in Joe's 30 year career there was that one time multiple companies jointly sold us a single product"
> there's just nothing out there that pays as well
Tech pay will flatten out/drop in the next decade.
Skill requirements are diverging. Tech skills are being commoditized by influential, huge, monopolistic companies. There is no licensing body to keep supply in check (like doctors, lawyers do) and there isn't one likely to be either. Put that together and you have a recipe for excess skill supply, replaceable talent and incredible effort required to switch roles.
This seems like an issue which is heavily skewed towards start ups and tech companies, which are ironically generally viewed as the most attractive companies for tech workers to work at.
The pay will be lower but there are opportunities out there where you can genuinely work towards helping people with technical solutions. Both the local research hospital and world class children hospital in my area always have data science openings and positions like that for instance
I have no inside look and can only comment as a user, but isn’t a company like gusto providing a genuinely valuable service? As a small time startup operator I was happy to pay for their product.
Companies like that are relatively niche and usually already have all the talent they need. Not to mention, even if it looks good on the surface there could still be bad things hiding under the hood. A friend of mine works at a niche business selling network hardware, sounds innocent enough - and yet they are using creepy analytics to infer which companies visited their website and then spam them with sales calls.
Time to regroup on other domains. There are probably a lot of things that could benefit from good tech but not through mainstream channels and trendy ideas.
I'd argue that it's also contextual. The previous generations were probably helped a lot by the capitalist lever, but it seems that since the 90s we're putting the lever first and the needs second, trying to find growth beyond anything else, which leads to the equivalent of p-hacking at the economic level.
The tech industry nowadays is no longer about solving real-world problems and making the world a better place, instead it's all about screwing the end user in every single way possible, whether it's ads, stalking and privacy violations, spam ("marketing" as they call it) or just plain fraud where the company is happy to take the money but can't make it right if things don't go to plan and the customer is left holding the bag.
Most tech products nowadays aren't there to solve a real problem and aren't funded by customers buying them because they are good; instead they're funded by some VC scum and they're there to capture the market (or rent-seek) and prevent a legitimate business from starting (nobody can compete with free or below cost).
Technology-wise, we no longer use engineering as a means to an end to solve a business problem. Instead, engineering became its own thing and most companies encourage and reward those who opt for over-engineered solutions, which means you spend more time fighting with dozens of layers of abstractions and chasing the latest JS framework instead of actually delivering functionality. This is mostly a symptom of the previous point where showing "growth" and bragging about your (over) engineering is more important than actual profit.
Unfortunately there's just nothing out there that pays as well so I have no choice but to tough it out.