Are you just interested in the training part and managing the trained models, or you'd actually like to productionize the models and serve them at scale?
A lot of end-to-end platforms are available nowadays that try to cover the entire lifecycle of a model from data prep, ETL, to training, serving, monitoring, operating. However, I found none of them really robust enough to cover all these cases perfectly, so I resorted to using different pieces from different vendors combined with my own stuff to make the entire platform suit my needs. This is still not perfect, though, and I think there's a lot of room for improvement in the space to enable really easy to use and scalable MLOps.
Still some of the tools I found to be ok: TensorFlow TFX, Kubeflow (to some extent - ops are a nightmare), Feast, MLFlow, GCP Vertex and AWS Sagemaker can get some work done, too.
I'd say for the foreseeable future I simply want to focus training and running trained models, I don't plan to do anything at scale like launch a business, so the creating and training aspect is the one I want and probably should only focus on at first either way.
But I like your approach of stitching together various vendors so they fit your use case, I think it can be really flexible but also probably more expensive and slightly harder to manage... I think it can be worth the tradeoff though.
You make a good point there. Personally I’ve struggled quite a bit moving from one off models to taking them to production. Would you mind elaborating on what you mean by none of the platforms being robust enough?
Wow. That's a real bummer -- an end of an era. I've been relying on Alexa (as part of various browser extensions over the years) to get a sense of a popularity of a given website for years. Not sure what's the good alternative. If you're aware of some in form of a Chrome extension, let me know.
There is a local company here in Seattle called Pulumi[1]. We've been using their tool extensively over the last 18+ months and it's pleasure to use.
It's built on top of Terraform, but it simplifies IaC because you can now write your infra components in one of few high-level languages they support (JS, Go, Python, etc). CDK is similar, but biased towards AWS. With Pulumi you can provision your infra stuff in multiple clouds easily.
This is a known and hard problem. A lot of companies are trying to internally do some sort of analysis of their own content libraries and intelligently tune their encoders. Netflix is definitely the most known for their tech related to video analysis and introduction of VMAF[1], but other metrics also exist which enable you to compare the original/master and the encoded variant (PSNR, SSIM, etc). Bottom line is that you need a lot of trial and error and fitting different curves on your bitrate/quality graphs and often times what metrics consider to be good quality human visual system doesn't agree 100% with. It's a very interesting problem nonetheless, and I recommend [2] if you want to learn more.
It's not that simple. Especially if you deal with videos that have open GOPs and have multiple B frames in the GOP. The encoders don't do so great job in those cases. Also breaking the videos up into short chunks is easier said than done -- you need to understand where it makes sense to split the video, make sure the Iframes are aligned and generally try to keep the consistent segment size -- which with very dynamic videos encoded with different types of source encoders could result in very inconsistent performance of the encoder itself across the chunks. For that reason, it's always best to have some sort of two/three pass encoding where the analysis step is integral and then based on it the actual split & encode is performed. Which of course does not work for low-latency live streaming scenarios.
The problem is that most of the big tech thinks they are smarter than 1000's of codec engineers who worked on libx264 and they try to reinvent the wheel. They realize too late that all the money they threw into building their own stuff (or variants) do far worse in general case than the baseline libx264. They just don't want to admit it.
I don’t think it’s that they don’t want to admit it, it’s that the cost savings from something disruptive may make it worth it at scale. Or that somebody wants to risk their career on it haha and big tech has tons of cash to throw around.
Absolutely. In fact, I interviewed a guy last week that comes from very big tech and he basically himself admitted that they're basically wasting time and money chasing unknown "ideal" codec while almost always those projects hit dead end. He loves the money, I am sure, but he wants out anyway. :)
I've been to a project where, because the development and "hardening" budgets were separate, we knowingly pushed out buggy code so that we could take advantage of the latter.
Had I known that half of the job would be to game the system, I wouldn't have joined.
I am very excited to see a company built around Cadence! I am just trying to understand what would be the main differentiator between Cadence and Temporal, except providing a SAAS solution?
Do you plan to maintain both projects, or plan to deprecate Cadence in favor of Temporal?
The short answer is that Temporal is a fork of Cadence by the original founders of the project. The Temporal fork also has a VC backed company behind it. The Temporal is still an open-source project under MIT License. We believe the success of the open source project is essential for the success of the company, so we are putting significant resources into its development.
We spent almost year working on various improvements before declaring the first production release. The most important technical difference is that Temporal uses gRPC/Protobuf when Cadence is TChannel/Thrift. One implication is that TChannel did not support any security and Temporal supports mTLS. We are currently working on a comprehensive security story.
Awesome! What's the plan for Cadence? Do you plan to maintain both projects moving forward or you'll start shifting community to Temporal and deprecate Cadence?
I ask because internally we just deployed our first Cadence cluster a few weeks ago -- it'd be good to know what to expect and what the founders of the project suggest to do now that Temporal is officially released to production?
So the short answer is that we love Cadence but it is no longer our project. Uber team has full control and we are not part of Uber basically. Keep in mind Cadence is a project for Uber first and foremost. But Cadence and the team are super great.
The long answer is that we see Cadence as a great part of our lineage. We felt that our vision for the open-source of Temporal was much different than what Uber sees for the project.
Now that we have a production grade release, we are slowly phasing out Cadence support. We will still provide free support to any Cadence users who need help migrating to Temporal.
I can’t speak to your situation without knowing more. I will say if you would ever like a hosted version of this tech, Temporal is the way to go. I am happy to answer any specific questions too!
I had 7 meetings in total today, starting at 8am. Almost the entire work day (8 hours) spent in meetings. I am principal engineer and usually have say in many technical discussions and decisions.
But on a light day I have around 2-3 meetings a day. One of them usually being an interview with the candidate (either phone or onsite).
This is exactly what I said in my retweet to PG's essay. Having hackable bad tests in the very tech industry proves the point that artificial tests are still the way of thinking for many and something we collectively need to unlearn. The question is how.
When I see a company conducting these kinds of tests during interviews, it's a signal to me that they are probably not a very forward-thinking company, and probably not a good fit for someone like me who tends to think outside the box. If it's another cog in their machine that they want, these kinds of interview tests are probably pretty good. But if they're an innovative company trying to change the world, they'd be much better off with a different, more pragmatic approach designed to attract the independent thinkers.
Beyond that, in my opinion, nearly anyone with average programming experience can hack a typical programming/whiteboard interview, so these kinds of interviews are definitely not the best way to find the best fit for particular roles. It's a good way to filter out the inexperienced, for sure, but it's a pretty low bar to set, especially if it's a company looking for top tier engineers, people with the ability to see the bigger picture.
It's also likely that a great engineer would pass the test with ease, but because their skillset and abilities have been inaccurately assessed due to the poor interview process, the company could have initially assigned them a much more effective role where both the company and the employee could have benefited to a much greater extent, but didn't. Overall, relatively speaking, the "hackable test" approach wastes time and hurts everyone in the long run, both employers and employees. A little investment up front with more personalized, specific interviews can go a long way.
I think we should start refusing these impractical interview processes, or at the very least, from an interviewee standpoint, turn down jobs (if you can) and let the potential employer know that their interview process is the reason.
> Beyond that, in my opinion, nearly anyone with average programming experience can hack a typical programming/whiteboard interview
I agree with your conclusion that all it does is filter out inexperienced/incompetent candidates. This is valuable, but to me it's the "first question" - can they code?
I don't think there's really much value between an "Amazing coder" and a "can get the job done" coder. Very rapidly you're trying to answer much softer questions like "does this person work well others?" or "how fast can they learn new systems?" and there isn't a good way to do that.
The interview question I usually give is algorithmically very simple. There's no linked lists, no graph theory, none of that. What I actually look for is mostly how they interact with me and how they go about solving it.
There's a pretty straightforward gotcha that more or less everyone hits. If the candidate hits the gotcha and then keeps piling on special cases without ever taking a step back, that's a big minus. If they keep doing it even after I suggest there's a simpler solution, that's worse still. I much prefer somebody who takes a step back but can't quite figure it out without a little more help than one who just remains confident in their solution no matter what.
Yeah it's inherently shitty that I have 45 minutes to figure out if this person will benefit the org for years to come, but there's a lot more signal there than just "Did they open their CS 201 algorithms textbook in the last few week?"
>When I see a company conducting these kinds of tests during interviews, it's a signal to me that they are probably not a very forward-thinking company,
Can you list all 'forward thinking' companies for the benefit of all of us?
I agree with all you said. However, 99% of major tech companies employ exactly the same process of hiring software engineers. I've been on hundreds of interview loops and unfortunately just one bad interview in a loop of 5-6 could make the candidate look like a "bad" fit and thus receive no offer. That particular candidate might be in fact great fit, but the way he was tested could not reveal that.
I don't know the solution to this problem. Eliminating tests whatsoever and just talking with the candidate about his experience and probing his knowledge on different topics is not efficient either. There are a lot of talking heads around who when given a simple task fail miserably.
We as a tech community need to come up with better ways to assess other people's competency while also making sure those people fit within our company's culture, work efficiently with others and after all create value. This is a hard problem. So, we try to simplify the problem by imposing the "proven" way of finding such people -- give them arbitrary tests and hope they pass them.
A lot of end-to-end platforms are available nowadays that try to cover the entire lifecycle of a model from data prep, ETL, to training, serving, monitoring, operating. However, I found none of them really robust enough to cover all these cases perfectly, so I resorted to using different pieces from different vendors combined with my own stuff to make the entire platform suit my needs. This is still not perfect, though, and I think there's a lot of room for improvement in the space to enable really easy to use and scalable MLOps.
Still some of the tools I found to be ok: TensorFlow TFX, Kubeflow (to some extent - ops are a nightmare), Feast, MLFlow, GCP Vertex and AWS Sagemaker can get some work done, too.