How To Estimate Software Development Projects Realistically

As practice shows, many software developers can't estimate software development projects adequately, which usually results in blown-up buffers, overheads related to missed deadlines and other issues that may have a detrimental effect on the overall project success and client's budget.

If you ask any average PM or software engineer about what client's question is most annoying to them, the vast majority will definitely answer with "When will the project be completed?" So, what's the matter with realistic project estimation? Why does project estimation process top the list of developers' most hated challenges? The answer is really simple: it's unclarity and uncertainty that surround the process!

On the one hand, there's a formula saying that project duration = scope/efficiency. On the other hand, we don't know either a numerator, or a denominator in this formula. Why? There're several reasons:

• Client's approval of a technical requirements specification doesn't guarantee the document contains a realistic work scope. You'll only understand what a realistic work scope will be like after you take over the project from the client.
• Any task assignment has bottlenecks that can be interpreted in several ways.
• Developers' productivity and efficiency levels can be different within a team, as each of us has own work tempo and habits, so it's just impossible to make any clear predictions about this or that employee's performance during project development.

Anyway, when evaluating project scope and resources, we need to make a lot of predictions and bear responsibility for our forecasts. We need prediction as a way to get detached from unclarity and better manage reality.

All known math models that aim to answer the question "When?" (i.e. Gaussian curve, Poisson Distribution and Murphy's Law) are based on simplistic methods that are almost impossible to use in real-life practice. What we need to do is to build an adequate and realistic model of handling unclarity and uncertainly. This model will allow us to create an assessment algorithm, know its limitations and constraints and learn how to use this model for proper project estimation.

If your development team possesses the following features, you'll most likely be able to predict to the project scope and timing with minimal deviations:

• You have a tight-knit team of seasoned professionals with a very low staff turnover;
• Your client provides you an adequate and clear feedback (read generates a predictable number of bug fixes) and doesn't change project requirements much or at all;
• Your team already dealt with similar tasks before on previous projects;
• Your workflow, technology stack and development environment remain unaltered all the way down the project;
• Your team has a shared vision of the project objectives and goals;
• You estimate project in relative rather than absolute value (i.e. no $ and man-hours are involved)
• Your team has sufficient knowledge base and best practices acquired from previous projects/iterations, OR the client is OK to work in iterative way while receiving regular forecasts based on the actual estimation of team efficiency in previous iterations.

How likely are you to have such a team? Well, the chance is slim unless you have a legacy software and a team of developers who're about to retire and, thus, don't want to pursue better projects and IT career opportunities. The reality is such that development teams are becoming younger and more dynamic, and it's getting more and more difficult to retain them on the project if they don't find it cool and challenging enough. Most of today's teams are interrupt and change driven and, thus, don't match the above team profile. Therefore, we need to build a realistic and adequate project estimation model to convince the client they've chosen the right software development provider / project team.

How to Build a Realistic Project Estimation Model

A realistic estimation model is one that provides estimation precision acceptable for practical implementation. While math concepts don't seem to work well in this domain, we still have to learn how to provide estimates that are at least 80% to 90% realistic and how to handle and minimize all possible risks so that the client doesn't pay through the nose at the end of the day.

First off, each client purchasing custom software development from a 3rd party provider must understand that project estimation is impossible without time buffers, i.e. safety margins in the project schedule, and safety margins do not equal cheating the client. That's the reality! If your client is too opinionated and doesn't want to see any buffers in the spec, just remove the lower limit! Yet, making buffers requires their thorough management and control. If 1/3 of your buffer time is wasted while the project is still far from its completion phase, it's high time for expediting and pushing! To visualize time-spending, web developers normally use burn-down charts. If your processes are in place, expediting and pushing will account for only 5% of the work scope or less. If they account for more than 5%, you're in big bad trouble!

Hire talented developers from LATAM, Canada, and Europe

In complex software projects the key focus should be on finding bottlenecks and controlling buffers that "feed" them.

The principles of realistic project estimations

There're only two ways of a realistic project estimation:

1. Past experience and historical data;
2. Expert assessment and intuition.

Past experience and historical data

In practice, only typical and repetitive operations can be estimated based on past experience and historical data. While still existent, these operations will soon be automated and managed by robots.

Using past experience and historical data can have some shortcomings that affect precision of estimate:

• Developers who complete repetitive or very similar tasks on a regular basis master their skills and gain significant speed of problem solving. So, you can't say that it'll take the same time for 2 different developers to complete the same task. Moreover, we normally get sick and tired of doing routine job and, thus, try to find ways of making our job less routine and more creative. Making a change to a habitual process can increase the risk of errors and missed deadlines. But it's OK as long as you can control it and allot buffer time for experimentation.
• We're inclined to forget details and nuances of operations that we don't execute regularly, and need time to tune in and ramp up. It means our past experience based estimate will always be overly optimistic.
• Contrary to forgetting details, we have a very good memory when it comes to punishment. If you had a situation with screwed-up project deadlines in the past, you'll most likely remember it forever and will eventually become obsessed with expanding your safety margins in the project schedule to prevent missed milestones in the future. In this case, the estimate will be based on your fear to repeat the mistake rather than common sense.

It's critical for any good PM to not only discuss project schedules with the customer, but also discuss methods and ways of reaching project goals, otherwise there'll be surprises down the road. Everything that's rational to automate and standardize should be automated and standardized indeed! Tasks that imply experimentation with new technologies or development methodologies should be scoped separately as experimental tasks (with separate time and budget buffers).

As we can see, experience alone may not suffice for effective and realistic project estimation, but expert assessment and intuition can come in handy.

Expert Assessment & Intuition

The more we imagine how the project will run, the more precise our predictions will be. Key here is not try to deceive yourself! If you can foresee any technological clashes or discrepancies down the imaginary project road, you'll definitely bump into them down the real project road.

Any project task estimation process should be split into four stages:

Would you like to learn more

about talent solutions

1. Task setting, analysis of requirements and preliminary assessment (deadlines estimation)
2. Research
3. Estimate clarification based on research and a small chunk of real-life project work
4. Task completion

Further task changes are only possible as a force majeur case and all milestones should be coordinated with the client.

Absolute Estimating vs. Relative Estimating

Relative estimating is less problematic than absolute estimating. How large is Jupiter? We can't describe it manually, nor can we easily remember its dimensions we learned at school. But if we're asked "How large is Jupiter compared to Earth?", we can always find good analogies: Earth is just like a pea, while Jupiter is like a watermelon; Earth wears size S and Jupiter - XXL.

Our brain perceives relative values much better than the absolute ones, so use this principle in your project estimate. Divide each big task into a dozen of smaller ones and estimate each separately. Do compare sub-tasks: which one wears S and which one - XXL? Use your imagination to better understand the approximate scope of all project tasks and exercise often to boost this skill.

Although each customer in IT is definitely looking to get a ballpark figure along with the project estimate, it's still possible to estimate some tasks in relative terms. For instance, new tasks can be estimated in relation to one simple and well-known solution, which also allows to compensate for natural deviations in estimates and definition of problems.

Planning (Scrum) Poker

If none of the above works for you, start playing Planning Poker, a gamified consensus-based technique for estimating efforts or size of project goals in software development. Team members make estimates by playing numbered cards face-down to the table, instead of speaking them aloud. In this case numbers equal hours: 1/2, 1, 2, 3, 10, etc. The cards are revealed, and the estimates are then discussed. If numbers played are the same or close, determine its simple average and put it in your project plan. If the difference is significant, ask respective team members why they've provided such an estimate. As you get more details and insights into your developers' mindshare, you'll learn how to estimate more realistically.

Hire talented developers from LATAM, Canada, and Europe

If you still can't estimate adequately using Planning Poker (because of lack of experience), add a research stage where developers execute a particular function element, estimate work complexity and scope, and improve understanding of how many hours will be needed to get the job done.

Decomposition

Decomposition is a method of splitting a huge abstract task into several small yet clear sub-tasks that are easier to estimate. Break your project scope down into separate components to make your developers better understand the requirement. Software developers should be skilled in splitting a holistic project or technical specification into phases and estimating them separately. The more they exercise this method, the more realistic and adequate estimates with fewer safety margins they'll deliver to the client.

To nurture this skill in developers, we recommend that PMs / team leads do the following:

1. Explain what is meant by "adequate and realistic estimate", how it'll be used on the project and why it's critical for the estimate to be realistic.
2. Explain why safety margins buffers should be minimized.
3. Make sure your team is inured to reporting issues with task resolution in a timely manner and encourage mentorship.
4. Make each team member estimate their assigned tasks individually.
5. Familiarize your team with past projects' estimates and actual project cases they can learn from.
6. Let your team estimate real-life project specifications, discuss their estimates and why they come up with such bottom-lines. The more bottlenecks and inconsistencies they come across, the more realistically they'll estimate in the future.

If you train your Agile team in adequate project estimation on a daily basis, you can nurture this skill within just 2 or 3 weeks. And what's your take on this?