A simpler, if perhaps less effective, idea would be to just take advantage of ensembling. Get many estimates from everyone qualified to give them, then just fit a normal distribution to your final results. The average of many estimates tends to actually be quite accurate sometimes, with the famous example of visitors to a county fair guessing the weight of an ox[0]:
"At a county fair held in Plymouth, 800 visitors had participated in a competition to guess the weight of an ox. While most people's estimates were too high or too low — falling an average of 37 lbs. away from the true weight of 1,198 lbs. — the median of everyone's guess was off by only 9 lbs., or less than 1 percent of the true weight of the ox."
Curiously there is also some "within individual" averaging that helps too. If you would ask the same person to estimate the project timeline a few times, spaced temporally, then take the average, it'll probably be more accurate than a single estimate.
You're right! Ensemble forecasting is great! Vistimo proper (https://www.vistimo.com) does both -- it gets individual high/low estimates from multiple people, and creates a combined distribution from all of them (though each estimate is a log normal, not normal, since it's a time estimate).
But Vistimo is a full project management tool; that space is full and switching project management tools is expensive, so it's hard to get people to adopt.
Vistimo Quotes is a much lighter-weight way to get many of the same benefits without having to talk your whole team into signing up.
I'm curious if you allow the project to enter actuals after the fact, then you can compare them to individuals/aggregate estimates so you know how biased your model/individuals/the organization is on these types of estimation efforts. You could give that as feedback as an additional service: "You're constantly overconfident on X type tasks..."
Not at the moment. This is the dream, but it's statistically much harder than it seems!
For each task you have one distribution, and one sample. What's the likelihood that sample was actually from a different distribution?
Instinctively it feels easy to judge "your usually closer to the worst-case than the best-case" but mathematically it's not so straightforward, especially when the estimate doesn't include sick days, splitting time with other projects, etc, but both the projection and the reality do.
We tried this at $OLD_JOB and what I've found is that when there is a larger disparity between estimates, it hints at information asymmetry, but that asymmetry can still exist when everyone has similar ideas about timelines, and so you can still get drastically wrong estimates when everyone thinks they're in agreement.
How was 1 std dev 37 lbs for that??? I feel like if I guessed I might be off by 500 or more lbs. I would have guessed "slightly less than a small car".
37 lbs seems astronomically accurate unless this is a county fair of farmers...
I think Steve McConnell was calling for statistical modelling even earlier than that. Definitely not a new idea!
I'm working on the assumption that it hasn't caught on because it's a pain to do without proper tooling; but it's also true that many engineers believe estimation is useless, impossible, and a waste of time or worse. It can be hard to overcome the pain of having been burned by a too-tight timeline.
Nice service, lots of ways to grow. I think the ability to make pretty pdfs for clients is actually a strong pull.
In my experience, statistical modeling of potential timelines is not the bottleneck to accurate quotes: it’s the ability to actually give good guesses for “min time” and “max time” on critical tasks. I.e. the majority of timeline variability is due to just a few “tasks” that totally blow out of proportion because I underestimated how much work was actually involved. Idk, maybe that’s just me.
It would be nice to track my estimates vs actual time spent to improve the “human error” part.
Yep, it's down. Y'all have just about tipped over my baby redis instance with a few thousand project simulations all at once. Should be back as soon as I scale 'er up.
Feels like a good idea, but the hard part here is the data input. Do you have any plans to import from Jira or something like it? Ideally bi-directional sync after a given simulation is run?
I said a similar thing about having no basic import (even csv or such) function making it completely unusable for large projects. Got a really snarky response from the developer.
Oh wow, my first job in the industry was building something very much like this!
It was actually quite powerful: the UI and simulation were done by Microsoft Project (https://www.microsoft.com/en-us/microsoft-365/project/projec...), which I would describe as "the Excel of project management" in that it was an ENORMOUS bag of features for building Gantt charts. The coolest part was that it integrated with Outlook's calendar, so if a task started on June 1st, was assigned to Bob, took two weeks, and Bob was on vacation from June 8th-22nd, it calculated the end date for the task as the end of June.
The Monte Carlo part was run by Crystal Ball (which was a startup then, but has since been acquired by Oracle). It was a mature tool, so there was a big library of probability distributions (https://docs.oracle.com/cd/E57185_01/CYBUG/define_assumption...), you could enter correlations (if X task takes a long time, Y is going to take a long time too), and you could analyze the output in all sorts of ways (how sensitive is the deadline to this task, how frequently is this task on the critical path, etc.).
The implementation was quite a hack - a hidden instance of Excel running Crystal Ball and lots and lots of VBA to manage everything. I'm not sure that it was ever used for real :(
I really like this idea and wish all project management tools included it by default. Fwiw Fogbugz has been using it for over a decade, called Evidence Based Scheduling:
It turns out the project management space is full of neat tools with clever abilities, but it's very hard for a team to switch between those tools. It's expensive and time consuming!
Vistimo Quotes is an attempt to pull just statistical estimation out on its own, so you can use it no matter what you use for your project management.
(A note: this and EBS are pretty different in their actual math -- they just seem similar because the next closest thing is "everyone agrees on a single number per task and we add them up" which is so limiting that just having a statistical model puts EBS, 50/90, and monte-carlo into their own category.)
Heh, that note is why I decided that "project management" was just a bunch of nonsense (like any other field using maths without taking uncertainty into account...)
Yep, I'm not giving my email to something before I have a decent understanding of what it does. Even the "Try it out" redirects you instantly to a signup modal.
I was interested in it a bit, but not interested enough to fill the form. It's not even about the email address, but just the effort. I know it's ridiculous, but that's how it works. "Don't make me think".
It also has no data import functionality so if your project is over 50 tasks (basically anything remotely business scale) you’re gonna be sat doing data entry for a while.
As a package this seems perfectly reasonable, but I'm skeptical of the use of Monte Carlo being a technical buzzword rather than a choice made because it was the appropriate tool for the task. I wasn't able to find any technical details in the "How it works" section, but if you assume the distribution of time to complete a task is uniform on the interval [worst case, best case], then at any time point the underlying distribution seems to be a composition of sums and minima of uniform distributions. These are available in closed form. I see in another comment the distributions are assumed to be log-normal -- that's fine too, there are good approximation to these. On the website it says thousands of trajectories are generated. It seems likely that one can straightforwardly compute the result that the Monte Carlo approach would give in the limit of infinitely many generated trajectories.
I promise, this is the result of real constraints! Monte carlo is used because of the limitations of analysis when using log-normal distributions. Normal distributions can be summed algebraically; log normal distributions cannot.
Note that Monte-carlo is not used prominently anywhere on the site, just here because I thought HN would find that part of the implementation interesting.
It is true that there is no closed form expressions for the sum of two independent log normal distributions. However, like I said, good approximations exist. Moreover, if you were to ask me to compute some expectation E[f(X)] where X is the sum of independent one-dimensional log-normals, it seems simplest to convolve the densities and simply compute the expectation by trapezoidal integration.
I think I misled myself into thinking your selling point was the technology (Monte Carlo), whereas as you say there is little about that on the website. And your comments below are right: building the tool around MC affords you a lot of flexibility without having to be increasingly clever.
The only thing I would worry about then are events which have low probability but potentially very high "weight" in whatever measure of risk you are using. Sometimes the weight can be so large that your MC estimates seem to converging, and then the addition of a single trajectory completely blows up your variance. These are related to the "black swan" events people sometimes talk about. I don't think this is a problem in your model.
Very interesting! Is there a sample report / projection available to view? I'd like to show this to a few coworkers without having to spend time loading in my own sample data
I've used this technique before at an employer. I thought it was crazy at first but I came to really like it. No more story points and estimating the size of tickets. The monte carlo simulation handles the forecasting for us so the engineers can focus on completing tasks. The PMs focused on user stories and acceptance criteria. The stakeholders had a continuously updated forecast for when things would likely get done.
The idea seems simple enough and the math to do these kinds of predictions is fairly straight forward unless there is some secret sauce under the hood that isn't obvious. I guess the value is slick UI / tool that was created around it.
One question that I do have about these types of tools... how does one evaluate how well it worked? I guess in the end as long as the timeline wasn't too far off then you call it a win.
I think a useful tool for project planning is not necessarily simulating possible time-lines (or a estimated window of time for the project to be completed) however to intelligently identify a set of critical tasks within the list which would have a high probability of totally blowing up your time line their deadlines are missed. In project management / scheduling there are often critical path items listed. However often these critical items are semi-arbitrarily picked by the PM and not backed by and deep insights.
About 5 years ago, when I first started using this method, I gave a talk about it that went into detail about how to do it, and the benefits. I did a live demo showing how well it modeled the variance even for really simple tasks like "take off your shoes and then put them back on".
The audience reaction was very positive, and I was pleased to have spread the word.
No one actually put it into action, though. It turns out if you need better estimation tools, you don't have time to make them -- you're in crunch time, because your estimates were bad.
The benefit is not so much slick UI -- that part is just me having fun. The benefit is that the work, however easy, is done.
If folks "steal" this idea because it's easy to implement, that's great! It's not a new idea, just an underused one. I just want to stop being hired onto contracts where the first thing I have to do is explain that the deadline is impossible.
I agree with you that a linear backlog is not sufficient to do really sophisticated planning -- Vistimo Quotes is a pared-down tool designed for easy adoption, because my kitchen-sink version requires too much buy-in for many teams.
Sorry, I did not mean to imply that the only value that the creators had was making a nice UI. I truly meant it as a compliment. There is hard work involved in creating a product out of an idea. Seems like this will be a very useful product. Congrats to all of the people who have worked on it.
There's a product called LiquidPlanner built around this idea thats been around for years. A friend used to work there but I have no other affiliation. Looks like they're still kickin' https://www.liquidplanner.com/
Trying to get correct estimates feels like it violates some basic quantum principle of physics. Were venturing into Schrodinger territory here folks.
On a more serious note, good idea and nice site. I may give this a try. I have been thinking along these lines myself, using simulations to complete different dev related tasks
Out of interest, why not make this a jira app and sell it in the atlassian marketplace? You get low cost distribution, the tool would fit right into the existing product everyone is using, installation is easy etc, data sync is easy, you can still generate the nice pdfs.
I'm a consultant, and it happens that hardly any of my clients use Jira. So this is the result of scratching my itch (and the itch of people who already give me money) first, and dressing up the result for others to use.
Lots of people have expressed interest in Jira integration, and that's definitely on my list to accomplish one way or another. The Atlassian marketplace is like most app stores, though -- if people already know your name, it's easy for them to find and install your app. But if you're a relative nobody, there are 50 other estimation apps, and very little room to pitch what makes you unique.
Honestly, the Atlassian app store is pretty bare. There’s pretty much only one or two solutions for everything, and they’re likely 2+ years out of date.
The idea is interesting, but it appears to involve using Vistimo's UI to break down projects and tasks. It feels incredibly waterfally, and also will involve re-entering mass amounts of project management data. I hope it integrates with Jira/Trello.
I like this! I would consider using it for some of our projects.
Biggest issue I have is with the task list, for any reasonably complicated project we would at least need ability to group tasks into features or milestones. Task lists can get pretty granular, but communication to stakeholders or clients does not usually need to be that detailed, higher level groupings are preferable (and also makes it easier to organize the tasks).
As others mentioned, integration with Jira and such would be nice, but not a deal breaker.
Also I think I broke my demo project, I changed one estimate and the projections just say "New projections are being calculated". There are only 4 tasks :)
I totally agree, personally I want more hierarchy; but user testing with milestones was pretty rough. Vistimo proper (https://www.vistimo.com) is less slick-looking, but has milestones and story-mapping and bugs and enhancements and all kinds of hierarchical organization tools. Most users I've tested with have found Vistimo Quotes' approach easier to pick up and use on existing projects, even if it's less powerful at expressing the full richness of the work.
Re: the long running calculations, I will take a look, but it's possibly just a traffic jam. There are a few more "test" simulations running simultaneously than usual this morning.
Estimation. Great if you can fix (a) who is working on it (b) the visual and technical design and (c) the scope and (d) other interruptions.
You can fix those things, have fully signed off design docs, have a CCN process for every little change etc. and watch everything grind to a halt while you deliver the wrong thing on time.
I’m not saying don’t estimate. Do! Have a plan and realise it won’t survive contact with reality.
Interesting idea, and very nice and simple web page! I have no idea if people are willing to pay for it, but good on you for making an attempt.
A question though, isn't Monte Carlo a bit overkill? Would you lose much by modelling each deadline as a normal distribution? Then you could just calculate things analytically.
And once you're committed to using monte carlo, you're free to include one-off events -- sick days, vacations, social loafing, etc, that are really hard to model analytically, since they are roughly uniform in time -- the longer the project runs, the more days off make the project run longer.
As another alternative, could you convolve the probability distributions numerically? I’m not sure of the trade-offs between that approach and Monte Carlo, but if it’s something you’ve considered I’d be interested to hear your thoughts.
Once I took a few steps down the simulation path, it seemed so easy and powerful that I didn't look back. There is some computational cost, but for me it's more than made up for by how easy it is to, for instance, say "I'd like to model sick days";
* let the government tell me how often people are sick (https://www.bls.gov/cps/cpsaat47.htm, see "Computer and
# mathematical occupations", absence rate, due to illness or injury),
* make my simulated workers take days off at that rate.
And everything else, all the statistics etc, can stay exactly the same. And I can have a lot of trust that it's producing sane results because I can look at a run, and say yes, that's a plausible number of sick days.
Monte carlo requires you to be clever once, setting up the simulation, and then you almost never have to be clever later when you want to introduce new twists. Any other statistical model you have to keep being cleverer for every additional complication.
Neat idea. This looks somewhat similar to Actionable Agile except it's up front rather than relying on ticket cycle time.
Are you able to share the task list so that each project member is able to fill out their best and worst case estimates on their own in an async fashion, preferably anonymously?
It's been my experience that most business needs involving estimates can't wait until the job is partway done. "Can we afford this" and "should we do this now" really need some up front answers :)
Vistimo Quotes is specifically for producing rapid estimates that don't require team buy-in, sign-up, etc.
Vistimo (https://www.vistimo.com) does anonymous, asynchronous team estimation -- but the additional labor costs (sufficiently describing tasks to async folk understand them, managing team permissions, requesting reestimate when requirements change) have made it not particularly viable as a product.
Finding a balance of power and simplicity is never easy.
Yeah that was the only downside I found with Actionable Agile. It didn't answer the question up-front. However, I think it can work if you're able to estimate the number of tickets instead of time frame. So instead of a best/worst case estimate in time, you'd estimate the best/worst case in number of tickets a task will need to be broken up into. I did this one time when I needed to try to get a better understanding of estimation up front when the work wasn't well defined yet.
It's possible that people may give better estimations when thinking about the number of tickets because then you immediately have to think about the number of individual components whereas with time, you're going more by feeling.
That makes sense! I am constantly interacting with new teams, so I like time because everyone already knows what a day is. A ticket, or a point, or other abstraction changes from team to team and project to project.
Haha, look closer! These buttons have a continuous curvature profile, rather than a rounded rectangle. It's a big part of what makes them feel so "70's calculator" chic :)
I'd like to see some kind of explanation of how exactly you "express the various uncertainties" of the project. I'm not going to click the CTA until I understand what I'd actually be doing to use the product.
I tried to use this but once I start entering tasks I don't see how to proceed. There's no way forward from neither there nor the project list page. Am I missing something?
The UI is slick, but why should I trust this black box?
The "how does it work" section doesn't even mention
* modeling assumptions
* training data (if any)
* validation (if any)
Why is this better than, say, giving the 25-75% range around my "best guess" assuming log-normal duration and the variance of prior project completion times? Can you demonstrate that it's better than that?
It's not better; it's almost exactly that, as detailed in many comments below. I choose slightly different thresholds -- 75% is not very high, you'll be wrong one time in 4 -- but that's the approach.
If you're already doing log normal modeling of your estimates, and including sick days, social loafing, etc, then you don't need Vistimo Quotes. Just keep doing what you're doing!
This is something I would buy if I had confidence in the simulator.
Why not write a blog or a brief white paper giving some hint of what's going on behind the scenes?
Generally speaking, every bit of energy someone spends being cool (good UI) comes at the expense of energy spent being good. Maybe you've done some serious work on the simulator, but I can't tell from your marketing material.
How do you distinguish a nonsense model from a good model when you don't have a writeup, you don't have validation data, and it is not possible to run cheap experiments?
I'm not saying you can't make money selling an RNG with a slick UI, you certainly can.
I thought this would be brilliant, so I clicked 'try it out' only to get through and be asked to manually write in the tasks. I have a huge backlog already in an external tool.
I can't comprehend I'm expected to write them out one by one with no ability to import that I can see.
Not trying to be snarky, I just didn't understand what you needed. Json import is not something any other users have brought up, but I'm happy to look into it.
I'm working on CSV export, first, but I will absolutely keep this in mind.
Until you have some reasonable way to quantify the likelihood that the CEO will see something shiny and insist on changing course, this just doesn't seem like it would be useful in the real world.
There's an "unknown unknowns" factor specifically for including guesses about new features or last-minute changes.
And the great thing about being able to run estimates quickly is that you can tell the CEO how much their new idea will send the timeline off course as soon as they propose it, so they can decide if it's really worth the cost.
I've been using this tech for years as a consulting engineer; this is just the dressed up version. It's been useful and effective for me, at least.
"At a county fair held in Plymouth, 800 visitors had participated in a competition to guess the weight of an ox. While most people's estimates were too high or too low — falling an average of 37 lbs. away from the true weight of 1,198 lbs. — the median of everyone's guess was off by only 9 lbs., or less than 1 percent of the true weight of the ox."
Curiously there is also some "within individual" averaging that helps too. If you would ask the same person to estimate the project timeline a few times, spaced temporally, then take the average, it'll probably be more accurate than a single estimate.
[0] https://www.npr.org/sections/13.7/2018/03/12/592868569/no-ma...