Sunday, December 26, 2010

Joel test for software companies

Today I discovered Joel test, a test to evaluate software companies. While the article is pretty out-of-date, and there are some points that are exclusively for companies working on compiled programming languages, the article is still very interesting.

I think every software company should took the test. Also I think it can be useful for programmers, to evaluate if a company is a good place to work.

The original article is at:
http://www.joelonsoftware.com/articles/fog0000000043.html

Let me provide a summary of original questions here:
  • Do you use source control?
  • Can you make a build in one step?
  • Do you make daily builds?
  • Do you have a bug database?
  • Do you fix bugs before writing new code?
  • Do you have an up-to-date schedule?
  • Do you have a spec?
  • Do programmers have a quiet working conditions?
  • Do you use the best tools money can buy?
  • Do you have testers?
  • Do new candidates write code during their interview?
  • Do you do hallway usability testing?
My personal update to the questions would be:
  • Do you use a distributed source control system?
  • Do you use a bug database where users can report bugs directly?
  • Do you have a testing protocol, and specific resources for testing?
  • Do you fix bugs before implementing new features?
  • Do you have automated build or deployment procedures?
  • Do you have a roadmap, and you don't make important changes to the short term priorities?
  • Does your team work in good conditions (quiet environment, flexible schedule, freedom to choose development software, fair paycheck...)
I think those questions can give you an idea on how efficient your company is, and indirectly, about the quality of your software.

69 comments:

  1. Why does a centralized company need a distributed source control system?

    ReplyDelete
    Replies
    1. Distributed VCS offers a much larger feature set than conventional source control. The number one being the ability to commit into the personal repositories that reside on the same system.

      Delete
  2. # Do you use a bug database where users can report bugs directly?

    If so, you fail the test? It takes a very special kind of user to put up with a bug database and all the hoops they force you through. And, since customers are always reporting dupes, your bug database will be full of noise and everybody will ignore it.

    GitHub gets this one right: place a human being between the customer and the programmer. Customers file support requests or issues, not bugs.

    ReplyDelete
  3. The old test is still valid. It is not outdated, just because it is old. That "concept" is one of the things wrong with "young" programmers. "We are the first generation to have these types of problems" NOT.

    Distributed source control: Not needed in a centralized environment and may actually make things worse. However, systems like git/mercurial etc. do have some very nice features over many non distributed systems. But those are not bound to being distributed at all. (e.g. local commits, that's a great tool, even when not distributed)

    Direct bug reports don't work with most types of user scenarios as someone already pointed out.

    Joel already mentioned testing right?

    Fixing bugs before implementing new features is a direct duplicate, you just reworded it.

    Automated build and deployment procedures are not necessary and depend on the type of company.

    Roadmap and long term planning go hand in hand with having a spec. It's not necessarily a detailed technical spec, but something (not even a document in the traditional sense any more).

    He mentioned a quiet working environment already right? Of course some additions like good pay etc. are a good thing, but even your list doesn't encompass everything I would look at (e.g. medical benefits if you're in the US etc.)

    The beauty of the Joel test is that most points are still valid and general enough to be applicable in newer environments. Also, where is his test not applicable to non-compiled languages? It's nothing about compiled vs. not, e.g. you need testing with both...

    ReplyDelete
  4. A centralized company needs distributed source control because any project with more than one developer is distributed among the developers. Distributed source control is about more than forking projects or remote development or offline checkins. It's about having room to breathe and branch and a reliable way to merge back up.

    ReplyDelete
    Replies
    1. Hello Bill,

      You obviously never experienced how it feels leading a team of 50 developers where all of them "take some room to breathe".

      You also seem to not have experience with branching large projects (meaning projects involving about a dozen Git/HG repositories). Git and/or HG are quite poor on branching in sync and on enforcing one part of the code base to be read-only for a particular team (from a set of development teams working on the same product), meaning branching some part while preventing other parts to branch.

      The need for centralized version control didn't go away. It is just replaced by some "social engineering" for some big OSS projects (it's a quote from the HG book). Could be theoretically done the same in large software companies but then it requires quite some man power. It is significantly more cost effective to do it with a centralized version control tool.

      Respectfully,
      Adrian.

      Delete
  5. This isn't much of an update, and just seems to be a restating of the original article with a few tweaks. Some things that get me:

    >> Do you use a distributed source control system?

    Sometimes a DVCS is the right decision, sometimes it isn't. If the rest of your work flow is monolithic (or not all devs have access to the whole source, or it's huge, etc.) then you're just shoehorning a DVCS into a centralized workflow, which does nothing for anyone.

    >> Do you have a testing protocol, and specific resources for testing?

    Joel asked 'Do you have testers?' There is nothing in this new question that isn't directly correlated with the original - if you have dedicated testers you'll have these other things, and if you don't you probably won't.

    >> Do you fix bugs before implementing new features?

    In most large pieces of software the answer will be 'no' - and it shouldn't be 'yes'. Sure, fix some bugs before moving on, but when you have that many small bugs (that impact no one) then it's often not worth it to fix them if it means sacrificing a new feature (which will effect many users).

    >> Do you have automated build or deployment procedures?

    If you can make builds in one step and make daily builds (two of the initial questions), then you have automated build and deploy procedures.

    ReplyDelete
  6. This "update" seems to simply reflect the author's personal preferences rather than having any real demonstrable basis in productivity and employee satisfaction.

    ReplyDelete
  7. I think a DVCS is a great thing for open source projects, but it's a distraction and buys you very little for the added complexity in your typical corporate environment.

    In my opinion, the original questions are fine and don't need a major update as they cover the basics nicely. I'd just add "Do you use continuous integration?". The final question about usability tests doesn't apply to all projects though.

    ReplyDelete
  8. "why does a centralized company need a distributed source control system?"

    Because the peer-to-peer model has many advantages over centralized (client-server) source control. Especially when it comes time to merge, branch, or keeping commits local without forcing them on everyone else.

    ReplyDelete
  9. Bill, Even with one developer you need a distributed source control system if a user's working on multiple features at the same time. Otherwise your central repository will be littered with way too many development branches.

    ReplyDelete
  10. I don't agree at all the DVCS somehow was the solution for merging. Perhaps for the Open source world, but for companies using propriety solutions merging hasn't really been an issue (part from the fact that no merging tools can fix conflicting changes or even trigger on incompatible changes in the same file done by different authors) for many years. Most of the larger known VCS:es have support for merging and branching and it has been stable for ages.

    Even if you can branch you should really try to aviod branching just because you can unless you can keep your branches short lived. Branches is not a solution for long lived changes, incremental, small step progress is.

    ReplyDelete
  11. @Niclas - I completely disagree that branching is a non-solution. Because if a person can't branch then they'll maintain multiple UNVERSIONED (or at least uncommitted) work-spaces which is infinitely more dangerous (I have numerous examples of people losing access to their local work-spaces and having to start over), or worse, they freaking email files to one another or collaborate over shared desktop, all because of fear of branching in a crippled subversion system (e.g. --reintegrate hell).

    That being said, I've yet to see a VCS that properly handles file-renames within branches. And in my tools, 'refactoring' makes heavy use of file-renames.

    ReplyDelete
  12. I'm in the process of switching my team to GIT (distributed vcs). Reason? Significantly better merge tracking than svn, tfs, vss, cvs, ... .

    ReplyDelete
  13. @Michael - try ClearCase. Monolithic, sure, but it handles branching, merging, and tracking file renames so easily.

    ReplyDelete
  14. Holy shit I've done the impossible. I've create branches AND merged with regular subversion. I pray to god the universe doesn't implode.

    ReplyDelete
  15. @tyler

    The original Joel test was pretty much personal preferences as well. You could certainly point to companies that were shipping software and making money that would have passed none of the original test questions. The questions were meant to demonstrate Joel's preferences for development, and many people agreed with him.

    I suspect many people will agree with most of these new ones too.

    ReplyDelete
  16. I'm with Niclas on this one: branching is for short-term diversions/developments, not for long term project strategy.

    Long-lived branches lead to management hell, well before you hit reintegration hell. If you have a multiplicity of branches, how are and the rest of the project team supposed to know what features are on which branch, and what bugs need to be cross-filled, back-filled and forward-filled to which branches?

    Branching is a short-term work-around for managing multiple developers - it just formalises what having a local copy of all the source does in the first place.

    ReplyDelete
  17. "freedom to choose development software"

    Fail. Even in open source shops, the management may want to standardize on one environment to keep things consistent.

    "fair paycheck"

    Fail. Everyone's definition of fair is different.

    ReplyDelete
  18. The only useful update is "Do you have automated build or deployment procedures?" in place of "Can you make a build in one step".

    It's less confusing for web developers who don't understand the point of "make a build in one step". The point is to automate the process so that you don't make procedural mistakes when updating the code. It doesn't matter if the code is compiled or not, this is critical for effective debugging.

    The rest are simply restatements of the original questions tailored to a much smaller subset of the software market. As such they are completely inappropriate for a general evaluation of any software company.

    ReplyDelete
  19. I don't think it is crucial to fix all bugs before adding a feature. You have to triage. Some bugs just aren't worth fixing now, but still worth keeping around in case a later rewrite can fix them easily. It's much more important to have a test suite that covers all code and make sure all tests pass for every build.

    ReplyDelete
  20. I also disagree that DVCS is essential. Certainly not for a co-located group. I agree that good branch/merge tools are essential, but you don't need the complexity of DVCS to get that.

    At my last gig, we used Perforce. It's not free, but it's not outrageous either (at the most expensive tier, it's $740 initial + $160/year for each developer). For that you get an excellent product with tech support which is second to none.

    I'm currently using Hg (we spent a little time with git before switching to Hg). It's fine, but there's a lot of complexity too. Not that p4 didn't have complexity of its own, but we never had the kind of head-scratching WTF moments that seem to keep cropping up with DVCS.

    So, yeah, VCS is absolutely essential. Easy branch/merge is absolutely essential. Getting people into the habit of keeping everything in your VCS from day 1 is absolutely essential. Getting people to use good check-in practices (useful commit message, don't mix all sorts of random unrelated crap into a single commit, etc) is absolutely essential.

    But DVCS, in an of itself essential? Absurd.

    ReplyDelete
  21. DVCS is absolutely essential. Centralized VCSes all assume that a change is done when you check it in. There is no ability to do experimental work and have it turn into something real without mucking up the version history for everybody. Making a branch is a major operation that creates a new public name and requires that name to be managed. It slows things down and makes work more difficult that it should be.

    I am nearly to the point where I will no longer work for a company that insists on a centralized VCS. They are horribly constraining and painful to use.

    ReplyDelete
  22. Try this for GIT people:

    http://nvie.com/posts/a-successful-git-branching-model/

    ReplyDelete
  23. @Michael -

    Bazaar handles renames correctly, full-stop; it's something they put a huge amount of effort into very early in the development process.

    GNU Arch / Tom Lord's Arch (where many of Bazaar's developers came from) did likewise.

    ReplyDelete
  24. DCVS is a must. they provide much more features, and ways of performing work.

    Of course, a decent centralised system is a bare minimum and would work really well, but DCVS adds another dimension to that.

    I do not believe someone can prove that DCVS is absolutely better than CVS, people are really different.

    it is jus a personall fact for me that most top developers who are savy on technology, on doing things _constantly_ faster, better, smarter are going to DCVS.


    What i believe most people would agree is that there are really terrible CVS systems, such as ClearCase and VSS. ClearCase, by the way, is a definite winner, and it is still being used nowadays :))

    ReplyDelete
  25. careful not to be too critial people, these comments seem to be heavily censored ...

    ReplyDelete
  26. So whats the update here?

    ReplyDelete
  27. Watch Linus' google talk on git.
    He presents specific work flows that are nearly *impossible* with non DCVS, and are thus discounted as options. Once they are made possible (via either git or hg), it opens up whole new work flows that can allow far more productivity.

    http://www.youtube.com/watch?v=4XpnKHJAok8

    ReplyDelete
  28. In terms of questions to ask developers during the interview process. I ask one of the 4 questions, and if there is time left a 2nd question.

    * Describe what the optimum software life cycle is end-to-end, its strengths, it weaknesses, its scalability up, down and out

    * Describe how a software team achieves success on a day-to-day basis and what kinds of processes are key to achieving that success, and what tools best support that processes

    * Describe what a product architecture and software life cycle would look like that is highly aligned with the business team

    * Give examples of symptom identification type processes in engineering and the tools used to support that process, and then describe processes and tools that address root cause.

    Change Management is key, and DVCS is one component of this.One of the most valuable aspects of a DVCS system is the ability for developers, contractors and vendors to share code prior to integration. DVCS has been used by OS teams since the 80's, and came into full swing by the early-mid 90's.

    Context switching is the number #1 killer of developer productivity. I look to integrate tools that minimize contact switching as my #1 priority. This means I do not want to see the developer having to submit tickets to IT departments or make request to the Build/Release person for branches or builds.

    Here are some key principles I apply to engineering workflow and environment.

    * Loosely Coupled/ High Cohesive versus Monolithic / Tightly coupled.

    :( if the source tree is monolithic, the team will never achieve better than mediocre results in terms of productivity and responsiveness to change. throw a centralized VCS on top of that and now its a double wammy. productivity and responsiveness cannot even achieve mediocrity.

    * Empowered Software Stacks versus Tight Rope Stacks

    :) show me an environment where a developer can easily setup their work environment, make code changes, test, and deploy into their own installed product instance, whether that be embedded or web, prior to integration, and any developer can maintain and extend the continuous integration workflow, that is a team that has got many things under control

    :( show me an environment where the developer has to go to anybody else to get things done, and that is a team that rarely if ever hits schedule, takes days and weeks to do tasks that are done in minutes by the empowered team, and generally has attrition due to the brutality of the work environment

    :( 90% of all software automation is dealing with symptoms. This is why software development efficiency is not better than 30 years ago. People keep developing tools and processes to automate symptoms and bad process... Most automation is, "Automating Bad Processes". Often what occurs the environment is hacked such that no one on the team really understands the process end-to-end. Often the policies of process are embedded directly into the scripts. Most of the time the policies have to do with symptoms. For example. Continuous Integration as it is practiced by the masses is a symptom identification process. It would be interesting to calculate the cost in terms of software/hardware and personnel required to deploy and manage a typical CI deployment, and what the context-switching factor is. What the A-Teams have known for 30 years is that continuous pre-integration is where the big win occurs. When the developer can validate their change relative to the rest of the teams prior to integration, productivity goes way up. Impact and context switching to team members goes way down. If the developer cannot from their laptop run a private build, test, deploy cycle in parallel to their development effort on the next task, then the process is inefficient.

    Not sure if you are automating bad processes? If the process defends against failure, it is a symptom identification process.

    If the process facilitates success, it is a pro-active process, and deals with root-cause, versus reacting to bad practices

    ReplyDelete
  29. Comfortable temperature. We've been working in a building where the temperature ramps up to the low 80's in the afternoons. And I've once worked in 40 degree temps at a hospital, horrid! Typing with gloves on!

    ReplyDelete
  30. DVCS may be a must for individual developers, but it is not essential for a software company as a whole. There are successful and fun place to work that are still stuck on CVS...

    There is very little that DVCS adds to the team's ability to work together... The parts that I see people use are the parts that make the devs more productive individually.

    On top of that, using git will exclude employees that use Windows. (though other DCVS have much better support)... Enforcing an OS on an employee(when not required by the project) is a sign of a POOR working condition. (keep in mind, artists, designers, analysts, managers and testers should be using the same VCS as developers)

    ReplyDelete
  31. On the contrary, I'm very hesitant to hire developers that think DVCS is a must for software development at a software company. All the arguments I seem to hear (and the ones I see here, too) are that DVCS allows you to branch/merge easily without having to "inflict" your experimental work on others.

    Or phrased more accurately, DVCS allows you to hide/hoard your work from the rest of the company (and the history involved in producing it) until you choose to share it.

    How is this an advantage?

    Centralized version control has just as effective branching and merging, it's cheap to create a branch, and as a bonus, you get running insight into the forward progress of your peers. If you're embarrassed by your experimental work then you need to re-evaluate your working environment -- are yours peers too judgmental?.

    ReplyDelete
  32. "Do you have a bug database?"

    Unless you are using such a system purely for logging and statistical analysis, you probably have bigger issues. Specifically, you should be very worried if you are buffering reports until your developers can get to them. It's an indication of management problems.

    ReplyDelete
  33. This "update" mostly smells of so much fresh paint for the bikeshed. Why? Because it's mostly FOSS-type buzzwords, personal preferences, or already in the original. The original is a baseline, this certainly isn't.

    ReplyDelete
  34. @Anonymous

    "All the arguments I seem to hear (and the ones I see here, too) are that DVCS allows you to branch/merge easily without having to "inflict" your experimental work on others."

    The main argument I've heard against centralized version control is that it *requires* one to interact with a central repository in order to perform even the most basic tasks. For some organizations this has value, for others it's an impediment. Distributed systems still permit this approach but do not require it.

    ReplyDelete
  35. > Centralized version control has just as effective branching and merging, it's cheap to create a branch, and as a bonus, you get running insight into the forward progress of your peers. If you're embarrassed by your experimental work then you need to re-evaluate your working environment -- are yours peers too judgmental?.

    It does not. In 6 months, after it's been integrated into the mainline, will anybody ever care about the 'feature foo' branch? I've not yet met a centralized VCS that allowed a branch that did not have a permanent name. You can remove the branch, but that makes your repository very messy and difficult to understand later.

    No matter how simple the branch and merge commands are or how reliably they work, requiring a global name for a branch is more than should be required.

    At my workplace we have branches from the released tip for each feature. When any given feature is ready, it's integrated and becomes the new released tip. The other branches are expected to merge with it before they can be released.

    This allows testing of each feature in isolation. In addition, as the build manager, I can maintain a branch that only I have access to that mashes all of the branches together to test them as one. This allows merge conflicts to be detected and dealt with early on. This also allows a full integration test to be done before any feature is released.

    Additionally, if need arises, I can begin demanding that each and every change along the way be an atomic transaction in which the software passes all unit tests before and after each change. And I can refuse to allow a developer to push a set of changes to the public repository until that's true. That would be completely impossible in any reasonable and safe manner with a centralized VCS.

    In a similar vein, processes that require code reviews before checkin to the public repository are greatly enhanced by a DVCS. Now everybody can see the _exact_ change that will be published before it's published. And if it needs to be re-worked, that re-work can happen before the change hits the public repository so there is still a clean, transactional history for the change in which each step has a logical, self-consistent reason for exiting.

    When people used a centralized VCS, I see changes live privately on their hard drives for weeks or months before they're checked in. It makes for a difficult situation if they should leave, and a very bad situation if their hard-drive should fail.

    ReplyDelete
  36. ITT: Everybody is so terrified of branch/merge operations that they never try them, and pass on their misconceptions to everyone they know, who all then never try them. DVCS allows people to do what they want and keep their secret terror intact.

    Just try it, people. Once you've done a couple, you'll wonder why you were ever so cripplingly terrified of something that's actually very easy and works well.

    ReplyDelete
  37. As a product manager in a previous life, I would add a couple of items to the test.

    - Do you have multiple release trains for your code. At least one that is bug fixes only and another train that is bug fixes and new features.

    This allows the company to be responsive to customer needs, win business and increases the confidence and respect towards the engineering team from other organizations in the company, especially sales and marketing.

    Multiple release trains is especially important for networking equipment and enterprise software. It is would be a problem for mass market software products like Microsoft Excel.

    ReplyDelete
  38. The main advantage of DVCS is that is makes branching and merging much less costly. Instead of long-lived branches, it encourages short-lived branches for single features. As someone pointed out, earlier, the true cost of branching is divergence from the main line of development causing long protracted merges. ANY additional weight you add to this process will encourage branches that hang around and get stale.

    With centralized VCS you suffer from apprehension about making branches because you are using a shared resource. I'm personally working on projects using SVN that have a *2 week* process involving a certain amount of bureaucracy before you are even allowed to make a branch. A similar process overhead for merging them. Why? The people in charge are used to SVN and are familiar with it's heavy overhead, so they have elected to make it much worse by adding so much overhead to the process that it guarantees that any branch will be long-lived and a difficult merge.

    One of the developers (the lead, actually) even screwed his source history up (by deleting all the files and re-adding them to make a "branch") so badly that merging it is all but impossible because most systems now consider the trees to be composed of different files. Well, except for using git to do it instead ; git doesn't "do" delete and add, just one tree and the next, and unraveled his mess nicely.

    The ideal? Take a look at the way Bazaar manages it's own sources, and how Kernel development proceeds. Bazaar has an enormous test suite (about 30,000 tests last I looked). Each and every merge to the mainline must merge cleanly AND pass developer review AND the tests.

    Branches are made for single features and merge outward from the mainline frequently. This ONLY happens when the effort involved is trivial. SVN used to be a nightmare because you had to manually log all your merges. Now it automates some of this, but you still can't merge a branch into the trunk more than once and it still doesn't cope with renames.

    I had to live through the age where management would mandate Visual SourceSafe because it was "free" with Visual Studio - this alone was an argument for allowing developers to choose their own tools, because no-one sane wanted to use VSS.

    I installed, used, and administered SVN repositories for 7 years and I wouldn't go back - I'd recommend Bazaar, git, or Mercurial now. Some of them are even usable in the traditional "central" style, but still offer all the advantages of lightning performance, superior merging and offline work.

    ReplyDelete
  39. DVC systems deal with large binary data very badly. Mercurial for example baulks at files over 10MB. What about companies that need to version large binary data, and want to keep it along with everything else that constitutes a build? Well for that use case, centralised systems are currently best. Things like Perforce and to a lesser degree other centralised systems can deal with multi gigabyte files effortlessly. DVC systems currently don't have an answer to this. And yes, I know about the "bigfile" extensions for Git, Mercurial etc. They're currently not robust enough to replace something like Perforce.

    I do actually believe that DVC systems are the future, but it really annoys me that people who use them for one use case that works (small source code files being versioned) just assume that theirs is the only use-case and that everyone who uses a centralised system is an idiot. Well we're not.

    What type of project needs to version large binary data like that you ask? Computer games. And almost all large games companies use Perforce for good reason.

    So the question should probably be, "Do you use source control intelligently?" :).

    ReplyDelete
  40. Apparently people didn't read fully what I said. Nowhere did I sasy that branching/merging isn't useful and then should be readily available, but that DVCS somehow invented this concept and are thus needed is absurd.

    Also people mentioning that you can't do experimental work in normal VCS is also absurd, it doesn't slow things down the least bit in any of the well known VCS, such as Clearcase e.g. It even has features that the allmighty Git doesn't have. However clearcase has been a system only used in larger corporation so very few people have actually used it, git is on the other hand readily available for anyone to use thus people know about it.

    I for one don't the least like to have people's work, even experimental, hanging on the thread of the sanity of their own hard drives. Proper setups include backups of even such branches, very much doable with a DVCS, but non the less doable with a traditional VCS.

    Of course faster check-in cycles are better and small incremental changes are better than living on branches for anymore than a day or two (note: I don't mean release train branches thate you of course need, but such branches even CVS can handle properly... any VCS will need a manual verification of code commited to a release branch, so even if the VCS you use handles it automatically, you will still need to verify it if it is merged from a branch based on anything later thant the release branch point, as the foundation you base that change on might not be valid in the release branch. Of course you can also hope your tests cover a broken corner case, I prefer a commit review to back it up).

    So DVCS essentials?

    No, better productivity, not neccessarily, I have actually observed the opposite as people have a hard time understanding where the root is. Also gits (or hg/bazaar) visualisation of branches is really quite bad, as compared to perhaps Clearcase.

    Branching/merging capabilities essential, definitely.

    The main missing and probably the largest productivity boost is

    Do you have automated unit/integration and acceptance tests? (as in, do you have a verifiable spec)

    Without that you won't need a VCS at all, because you won't be able to change your code faster than backing up the source code on floppies anyway =)

    ReplyDelete
  41. @Adrian

    You can't compare SVN and in the same sentence say that it is a strong and good traditional VCS, if you think so, then you haven't used a good Centralized VCS.

    And this is the large misconeption, people compare DVCSes to tools such as VCS and SVN or even Visual source.

    Replacing SVN with any comercial VCS will likely pay off in a couple of days, so free isn't always cheap. Perforce is excellent and so is Clearcase, and even so was Bitkeeper before Linus had to pay for it and thus made Git (when he instead could have put his efforts into monotone, hg or bzr..)

    And as people already have mentioned, handling large files in any sense is just madness in a DVCS (if you are even able to commit them more than once..).

    ReplyDelete
  42. My previous comment vanished, the web era with its fantasic user interfaces, I mean this box I enter text into is probably worse than what mainframes offered 35 years ago.. anyway (actually not much different than good VCS already to that DVCS suddenly made their trade mark).

    The main thing I think is missing is

    Do you have automated unit/integration and acceptance tests? (i.d. a verifiable running spec)

    And as someone else mentioned,

    Do you have an empowered team?

    That is a team that can make all decisions needed to complete a feature except functional decisions where pererably the end user is within reach in minutes

    Is your team as small as needed?

    (yes, just like designing code where you get less bugs the more code you manage to remove, the same happens if you remove as many developers as possible without significantly hurting productivity (magnitudes not percent..,

    this hit usually comes somewhere < 8 developers and more or less stays contant while adding more people above that for a sahred code base where people actually have to interact in any significant manor to get things done. Of course you can have a team write a kernel driver for some new HW and at the same time have another team design and build an application UI for your backend. But you might not get more productivity by having a second separate team writing the UI (or whatever uses the driver) for the HW the kernel driver talks too


    Without that you won't need a VCS at all because you will have all the time in the world to backup your source code on floppy disks if you want =)

    ReplyDelete
  43. Also I would like to mention that Joel's initial bullets where more about a mentality than about techniques for that reason alone any bullets that mentions how to do things should be striked (such as DVCS or not).

    To me only one bullet of yours makes that cut

    •Do you have automated build or deployment procedures?

    All the others try to say what is correct instead of making you think of what is correct for your particular team, setup and business.

    ReplyDelete
  44. @Anonymous

    "What the A-Teams have known for 30 years is that continuous pre-integration is where the big win occurs. When the developer can validate their change relative to the rest of the teams prior to integration, productivity goes way up. Impact and context switching to team members goes way down. If the developer cannot from their laptop run a private build, test, deploy cycle in parallel to their development effort on the next task, then the process is inefficient"

    I agree with this fully, I am not just sure if you are trying attributing DVCS to this? If so I don't agree, DVCSes have nothing to do with that. Just because you have a central repo does not mean you can only build your system on a central server, that would be insane (yet many systems are like this..).

    Perhaps it is more likely that a system in a DVCS can be built locally just because you try to empower the individual, but an A-team will do this in any VCS. They will rearchitect the code to run on a laptop, probably before they make a single change to the code, (maybe after setting up automated tests if they are not in place).

    It is absolutely essentials for each individual developer to be able to run as close as possible to 100% of a full system on his or her desktop/laptop. And for all the HW that you can’t run locally either you have test probes that can generate responses or you have abstract the HW into some kind of message passing so you can actually talk to the HW from anywhere.

    ReplyDelete
  45. I don't think the point of the test is to see if you pass it or if you fail it. I think it's a question of asking the right questions, and get your own conclusions.

    I'm sure there are several situations where you don't want to use a public bug database, or a DVCS, or when you can't afford a cool office or excellent paychecks.

    But I think it's pretty useful to ask those updated questions. Ask yourself if you're using a VCS, and you can end up being pretty happy for using SourceSafe. Ask if you're using a DVCS, and don't feel you're failing because you're not, if you have a good reason to do so...

    ReplyDelete
  46. you forgot:

    - Are you even relevant in today's software market?

    Add that in and Joel fails his own test.

    ReplyDelete
  47. @Anonymous

    Why wouldn't he be relevant? Fogbuz incorporates mecurial.

    Maybe if we are lucky they manages to build a useful integrated ALM one of these days, maybe even if a distributed task/bug database to accompany the source code. (veracity is trying this and also trying to fix the other shortcommings with todays DVCS:es, such as policies, ACLs and any other large corp "neccessity")

    ReplyDelete
  48. Here's why DVCS is a must...

    All other things being equal, would you start a new effort in F# or in Cobol? Well, F# of course, because it will attract the best developers. And having the best developers is the most important influence over success.

    Similarly, DVCS will attract the best developers. Compared to that, the importance of the branching ease and so forth is just noise.

    ReplyDelete
  49. @Morgan

    That was a funny way of twisting things, are you saying people using DVCS are better developers than those that don't? That has absolutely no reality correlation whatsoever..

    You might get the most "hip" developers that jump on the lastest trend, I might very well argue that you shouldn't have too many of those if you want to succeed...

    ReplyDelete
  50. 1. has your company outsourced to India?

    2. are your hardware contracts based on kickbacks and corruption?

    3. do you plan to go into bankruptcy immediately after shipping, and then send your executive committee off with a golden parachute?

    4. are there at least 8 levels of bureaucracy between your programmers and your users? 1 point per level.

    5. is the purpose of your software to dumb down or eliminate so many processes for the ultimate users that 80% of them will be layed off?

    if you score below 10, you are unfit for the corporate IT landscape.

    ReplyDelete
  51. 6. Are your contract based on man-hours instead of project milestones ? :P

    ReplyDelete
  52. AN unnecessary update. The original is still fine.

    ReplyDelete
  53. This seems a lot weaker than the original. The original had

    # Can you make a build in one step?
    # Do you make daily builds?

    while the new has just

    # Do you have automated build or deployment procedures?

    You can get into a lot of trouble with just automated build procedures if you don't run them daily.

    ReplyDelete
  54. one of the things some teams are doing is using git as a front-end to perforce and subversion....

    I only saw one post on here that made a good case for Perforce, and that was due to the nature of the gaming development community. Using CVS, Subversion and Perforce for most software efforts is like the person trying to write OO in bash or perl. Its just wrong. But who am I to tell the CVS, Subversion, Perforce user they are wrong. At the end of the day, the organization can respond to change, manage change, and hit schedule, or they can't. In my experience, every team I've seen not using DVCS has had one hell of a time managing multiple lines of development. I've seen teams in recent years that literally do everything they can to avoid branching... This is 2011 in a few days. Can a team live without DVCS, yes, sure why not. That same team is likely to be using bugzilla, cvs, tinderbox, bonzai, writing everything in perl.. and operate just like they did in 1999 and be successful on top of that. Or the team could look around and chose best of breed software tools, languages and laptops that are current. A team can choose to use trailing edge methods or leading edge methods or something in between. the sad thing is that the software industry is no better today at hitting schedule than it was 10 or 20 years ago. most software tools are still in the dark ages and have changed little in the past 25yrs. if you look at linux embedded systems and linux/unix operating system development environments, not much changed in 25 yrs. Oh... one thing has changed big time. 15 years ago, money was invested in the developers tools and systems. Today its common to spend a little a possible... so teams end up having subpar engineering environments.... you know things like cvs for source code control.

    ReplyDelete
  55. if you had 2 job offers to work on the same type of product.

    Company A had MacPro Laptops, Git, Jira, Hudson, Python, Ruby, Java, Intellij as the development stack.

    Company B had Dell Laptops, CVS, Bugzilla, Cronjobs, Bash, Perl, Java, Eclipse as the development stack... what does that tell you about the development culture?

    Company A spent twice a much on the developers laptop, purchased a commercial defect tracking system and commercial IDE

    Company B spent nothing on software and 1,000 dollars on the laptop.

    If you choose B, consider the message you are sending to your employer... Its ok to buy me the cheapest hardware and software.... and they will. Its ok to not invest in my productivity, and they will go along with that just fine.

    ReplyDelete
  56. So many of these comments are focused on just the single issue of the better approach to version control. As a job seeker, I also want to know more about the organization and what it's like to work there. I'd ask: "How am I going to fit into the culture of the company and the team?" In other words, can I work with this team? I would also change the question about the up-to-date schedule to be "How is the product roadmap and schedule determined?" so that I make sure not to walk into a Death March situation. Finally, I would ask about the management, funding, business model, and profitability of the company so that I wouldn't have to go looking for another job in six months (or less).

    ReplyDelete
  57. Distributed Source Control?

    much more important would be the inclusion of Continuous Integration Builds.

    Git has some neat tricks, but really if you need to split off a local branch that takes more than a day to work over, then you're sidestepping Continuous, to the detriment of the team workflow.

    even more to the point, if a developer is picking off multiple tasks and doing them bit by bit in multiple local branches, chances are they will never finish (m)any of them, further slowing down the whole team (by becomming a bottleneck)

    picking a source control tool that complements a steady team oriented workflow is always a better indicator of a good software process.

    and if you're not doing CI builds? you wont be building software for me!

    ReplyDelete
  58. If Joel Spolsky knew anything about software he would not write such horribly inefficient and bloated products.

    If he knew anything about building corporate value, he would not be explaining how to do it: he would have done it a long time ago.

    "He who can does. He who cannot teaches."
    -- George Bernard Shaw

    ReplyDelete
  59. Do you have a Secure Development Life Cycle (SDLC) in place?

    ReplyDelete
  60. It took me a while to get git into my org. During that time I heard all the same arguments here about SVN and centralized version control. Now the same people who said "what makes it so much better" can't live without it.

    Same can be said for scrum. In fact I refuse to work for any place that does not have a real scrum process. So I guess I would add.

    Do you use scrum?

    ReplyDelete
  61. Interesting for me to read that so many people assume that because a source control system is distributed or not then it has to have some sets of features that others can't.

    Branching patterns are not dictated by the model. The best example is Perforce that recently implemented the concept of 'shelving' where a developper can temporarily push back his modified files on the depot if he wants to work on some other task. This really permits to work on the same file for different tasks while keeping the changes distincts in the history. It also allows other developers to access the shelved files to test them on their own workspace to see if there are any integration issues for example.

    Same thing for the continuous builds, you can do them with a centralized or decentralized system.

    Where I work we have perforce, with a number of master branches for the main products, plus sub branches for the bugfix/versions, plus a mandatory weekly integrate/merge to a common trunk. Works well, specially when the continous build system builds every single committed revision, shows the status on a web page, and sends mails when things are broken.

    I'm not saying it's ideal, I'm just saying that if you try, you can make things work or fail with almost any kind of setup.

    For the rest, I agree, the update does not provide much improvement to the original test.

    ReplyDelete
  62. IMO, the distinction between a centralized and decentralized VCS comes down to this. At least compared to Subversion (and God forbid*, CVS), all the DVCS tools work better than the CVCS tools for some tasks. They have better merge tracking, they allow more local operations (though this does introduce the "I forgot to push" error, which I think you definitely shouldn't discount), support things like rebasing directly, etc. Overall, I think it's definitely a win to use Git over something like Subversion if the former's limitations (e.g. lack of partial checkouts) don't bother you. And you can usually make it so they don't.

    That said, for the sort of situation we're talking about here, they don't work better because they're distributed so much as... just by accident. The features they have aren't completely unrelated to the fact that they're distributed (because being distributed absolutely forces you to think about branching and merging) but I think there's no reason you couldn't make a CVCS do the same things.

    The main exception to this is being able to work from, say, a laptop, when you don't have repository access. I still think you technically could do it -- but by the time you do, you've got half of a distributed system anyway. If that sort of work situation is common, then there *is* a fundamental reason why you should very strongly consider a DVCS tool even over my hypothetical awesome CVCS, but I'm not convinced that's important for a lot of companies.

    * CVS is horrid. I would consider rejecting a place solely on the basis of them widely using CVS. I whine and complain about most software I use, but there are very few pieces of software I loathe with every fiber of my being the way I do CVS. If Git is twice as good as Subversion, then Subversion is 10x better that CVS.

    ReplyDelete
  63. lex parsimoniae: The more complex ya make the plumbing, the easier it is to clog up the drain.

    ReplyDelete
  64. "Do you fix bugs before implementing new features?"

    - In an ideal world, yes.
    When you're working on the same software base with several teams working on features, maintenance, fixes, r&d etc - - No can do.

    ReplyDelete
  65. Two of your new questions are too specific:

    Do you use a distributed source control system?
    Do you use a bug database where users can report bugs directly?

    Not everyone should have a distributed source control system and not every business should give users a DB to report bugs to directly. I agree that most businesses should have both, but not all businesses.

    The rest of your questions do have potential. The Joel Test is an excellent test and for the most part is standing up well to the test of time.

    Kudos for bringing it up to the attention of more people.

    ReplyDelete
  66. Thank you for reminding us this simple test and the value of simple tests in general.

    We should all realize that, when big companies of this world evaluate their software suppliers, they don't use the test of Joel, they use more thorough approaches. From my experience, CMMI and SPICE are the most frequently used. And then a maturity level of the software shop is computed for different process areas.

    But the test of Joel has the merit that it doesn't take days to perform and it doesn't take a team of certified assessors. I'm ready to defend that this kind of test is more cost effective than a CMMo or SPICE assessment in the end.

    Thanks,
    Adrian

    ReplyDelete