engineering perfection on rails

This post reverse engineers the reasoning behind what we do at C42 Engineering. We ask ourselves these questions every day, and these questions frame what we think it means to be a top notch provider of software engineering services. Much of it we're already doing. Some of it we're still putting in place. So yes, if you use this guide, C42 Engineering will likely show up among the finalists in a list of offshore Ruby service providers. We aren't the only ones out there though, and we hope that this guide will help you find a vendor who will help your business succeed.

The problem.

There are many small, new Ruby consultancies all over the world that do stellar work. There are many, many more that don't. As a startup looking to offshore work, how do you tell them apart? 

Is finding a local vendor that difficult?

In India, it is very hard to find consultancies or contractors who have experience in both Ruby and Agile; this is something we've heard from all our local clients. And we were prompted to write this article becasue we heard over and over again from friends and clients in the US that the same was true for them, and how risky (and hard) it was to identify the right offshore vendors prompting them to settle for less capable local contractors.

In our experience, you should value capability over the cost of managing timezone related communication overhead. If you compromise on quality to stay local, there's a high likelihood that when the time comes to start cranking out features or to change existing functionality, you won't be able to becasue your codebase is a mess and there aren't any automated tests. At that point hiring a top local firm will make little difference because they'll be hamstrung by the existing codebase and will likely recommend some form of rewrite anyway.

Is finding an offshore vendor that difficult?

It's quite likely that you've heard your share of horror stories about how offshore vendors work (or don't) - the false promises, the inflated billing, the missed deadlines, the poor communication and, to add insult to injury, all the defects that crop up. I'm also sure you realise that just like any marketplace, there are vendors that Do The Right Thing - it's just that they're in the minority and it's hard for them to stand out.

1. A Minimum Viable Product that allows you to go to market quickly

2. The ability to iteratively add features and pivot as feedback comes in

3. Making sure your code is well tested, so that point (2) does not break everything in point (1)

4. A transition plan, so that you can ramp up your own team as you get to be big enough

Most of our clients that are successfully outsourcing work have learned how to identify the right vendors over a period of years, often getting burned badly at the start. If you're a startup and are looking to offshore work, you don't have the luxury of learning by failing. This article distills what we've learned ourselves and from our clients about working offshore and working with offshore firms into a fairly simple set of guidelines. With some luck, this guide should allow you to zero in on the small offshore companies that are truly agile, passionate about good engineering and eager to see their clients succeed.

The bottom line is this - going offshore is inherently less effective than having a co-located team, especially in the context of agile projects. The only reason you'd offshore work is either because you have budgetary constraints, or you're unable to hire talent that meets your standards locally. What you want is a vendor who understands this and is actively working to mitigate the overhead introduced by having distributed teams while also having an engineering team that matches or exceeds anything that you'd find locally.

Is this guide relevant to you?

This guide should work for you if you:

• Want to execute your project using Agile principles, so frequent and fluent communication is crucial

• Have decided that your technology stack is based on Ruby (most likely Rails)

• Want top notch engineering so you get a clean, well specced, extensible codebase

• Have some budgetary constraints - after all, if you didn't you'd already have an in-house team, possibly augmented by a developer pair from a Pivotal Labs or a ThoughtWorks, right?

• On the other hand, you shouldn't be so cash strapped that you're offering equity as payment because most (but not all) services firms won't do equity deals. Most services firms, unlike VCs, can't spread their risk, are heavily dependent on cashflow month-on-month and consequently can't afford to gamble.

Cutting to the chase. Step #1 - Sourcing.

So the first question is 'How do you create a list of vendors that you can then filter?' The idea is to create a large list of broadly aligned vendors with a minimum of effort.

Most vendors do or say a common set of things to gain visibility. Here's how you look for them:

• Identify regions with a high density of offshore Ruby vendors. My list currently consists of Brazil, Uruguay, Chile, Argentina, Eastern Europe, India and Pakistan. Let me know in the comments if there are other regions/countries that I should also include here.

• Do a search for Ruby, Rails, Agile, Extreme Programming and Test Driven Development/Behavior Driven Development along with the name of a country or city from the list above. See which companies show up. Add them to your list.

• Scan local Ruby and Rails mailing lists in the areas above. Pick cities that you know contain offshore vendors, then find their Ruby users list and look for folks that post frequently and post sense. Find out where they work. Add them to your list.

• Look up local RubyConfs like RubyConf India, RubyConf Uruguay or RubyConf Argentina. Add the companies who sponsor these events to your list.

• At the same local RubyConf websites, go through all the presentations - typically, videos are available. Identify the speakers that made sense, then find out where they work. Add those companies to your list too.

• Explicitly ask for startups who've been working with offshore vendors for some time at your local Ruby or startup group and add those vendors to your list.

• Go to sites like elance.com and odesk.com and search for Ruby, Rails, Agile, XP and TDD/BDD. Add the vendors that show up and that have all of these keywords in their profile to your list.

Step #2 - filtering for the basics.

At this point, you should have at least a dozen firms on your list. We can prune the list fairly quickly based on some simple criteria.

• Open source - remove all the companies that aren't involved in open source in some form from your list.

• TDD/BDD - remove all the companies that don't mention TDD/BDD anywhere on their site or say "We do TDD/BDD for clients who want it."

• References - remove all the companies that do not have any references, either of clients that the company has worked with, or, in the case of a new firm, of clients that the founders have worked with.

• Pairing - does the company believe in pairing? While not essential, this can be valuable for two operational reasons beyond what XP covers. Redundancy, so that your productivity is not hit when, say, one of the developers falls ill. Knowledge transfer, because when the time comes to build out your in-house team, you can transition smoothly by simply flying in the offshore folks and having them pair with your engineers for a few weeks.

Step #3 - filtering out BS.

Next up is to lose the ones that either intentionally or unintentionally misrepresent their level of capability. An easy way to do this is to look for firms that claim to do TDD/BDD and open source. Go to their open source projects pages (most likely on github.com) and look under the test/spec folder - you'd be surprised at how many such projects from firms that loudly proclaim their love of testing have no tests at all.

Next, look at the open source projects themselves. Are there any popular projects in the list? Are they trying to improve existing Open Source tools? If the projects were started by the company, how complex are they and how many pull requests, forks and watchers do they have from people who are not employees?

Remove from the list all the companies that claim a trivial plugin or some such as 'open source.' The ones that seem to be doing something non-trivial in open source you should retain in the list, even if they don't have very many pull requests, forks and watchers.

GitHub also provides nice graphs about various projects. This will give you some meta data around the projects, such as, if the project was developed in a single day, and if the project has had continuous support and interest. The github punch card visualization will also give you some information about how passionate the developers are about their work and if they are more passionate about code on a particular day than on others :-).

Step #4 - filtering for credibility.

In the absence of an Accelerando style reputation marketplace, we'll have to fall back on a more direct approach to judge credibility. One of the easiest ways to do this is to use trusted external reference points like the proposal review committees of popular conferences, or the core committers of major open source projects.

• Start by checking how many employees of a firm have spoken at major conferences. Have they spoken at the local RubyConf or RailsConf? How about the annual international RubyConf and RailsConfs?
• How many employees of a firm have started open source projects? How many have made non-trivial contributions to existing popular open source projects?
• Are the employees of a firm active on mailing lists, including both the local Ruby/Rails lists as well as the lists of major open source projects?

We're now done with the easy stuff - this step and the rest of the filtration process will require more effort than simply looking through a few web pages. Now the next thing to do is to dig into the open source work of the firms still on your list and do some code reviews. At this point, all the vendors left on your list should have one or more non-trivial open source projects that they've been working on.

For this, you'll need to have experience working with test driven codebases yourself, or involve a trusted third party who does. Here are the obvious, simple things to look for:

• Complies with project structure conventions - whether it's a pure Ruby or a Rails project, it should follow the standard project structure.

• Coverage - the coverage should be no less than 85%. For rails projects, the coverage should be measured separately at the model, controller and cucumber level because with C0 coverage, a dozen Cukes can be used to produce (a completely misleading) coverage of 100%.

• Quality of tests/specs - it's usually fairly obvious when tests have been written after the fact. Remove any companies that obviously and consistently write tests/specs after the fact. Also keep an eye out for badly written tests, especially the kind that don't assert anything or are very hard to cause to fail.

• Cyclomatic complexity - toss Saikuro (and flog) into the build and see what numbers it spits out. Cyclomatic complexity numbers almost always (with the possible exception of parsers) maintain an inverse correlation with design quality - the higher the numbers, the worse the authors of the code were at OO design.

• Good old fashioned code review using the 'WTF per minute' scale. Just remember that all code causes WTFs, so be kind (but not too kind).

Step #6 - filtering for communication.

You should now be down to a handful of companies; it’s time to start talking to them. Drop them a note and schedule a Skype video call, so you can get a feel for how effectively they communicate. It doesn't mean much, though, because at this point you're likely speaking to a sales person (even if that sales person writes code).

Doing this will trim a few firms off your list because you'll find that they simply won't have capacity available in a timeframe that makes sense for your project so that will prune your list somewhat. For more information on why this is so and how small consultancies work, take a look at our talk on the subject at RubyConf New Orleans 2010.

Once you get on the call, your intent is to get a feel for the firm and how they operate. Ask how they mitigate timezone related communication gaps. Do they do daily stand-ups over Skype video? What tools do they use to support project execution? Why? How do they handle project management? Do they have dedicated project managers, or does the engineers also handle project management and client communication? If they have dedicated project managers, how big does a team need to be to require the attention of a manager full-time?

The next test is to ask for interviews with the developers that would be staffed on your project should you choose to hire that vendor. No, this isn't rude or offensive. Yes, we're used to being interviewed by clients - indeed, we prefer it. Just be aware that staffing cycles at small consultancies being what they are, you may have to interview several people even if you're only looking for a single pair because the vendor is unsure exactly who will be available when the project gets started.

Once you have the interviews with the engineers set up, we come to...

Step #7 - filtering for engineering, part 2.

Keep in mind that just because one trophy engineer in the company contributes to open source, speaks at conferences and generally kicks ass doesn't mean they all do, or that you'll get that one superstar for your project. Treat this interview pretty much like a hiring interview. Ask for information on the engineers you'll be speaking to before hand and take the time to go through their blogs and github profiles to get a feel for their capability ahead of the interview.

Here are some of the things to look for when interviewing the engineers aside from the usual CS interview questions that you'll probably want to run through anyway as a baseline safety net.

• Ruby Skills. If required, will they be able to open up rails, and understand how find_by_* gets overriden? Has the engineer ever worked on a non-Rails project? You don't want to hire someone who 'programs in Rails' and to whom Rails constructs are black boxes that are magic.

• Rails skills. How much of the framework does the engineer understand? Would they be able to make decisions about various ways to do things in rails?

• Breadth. How many programming languages are they familiar with? How many have they worked with professionally?

• The rest of the ecosystem. Does your engineer understand the rest of the ecosystem when your application starts to get complicated? Do they understand concepts like asynchronous messaging, background processing and evented systems? Are they comfortable doing TDD in javascript to build rich, single page applications?

• Engineering and design - does the engineer understand good object oriented design? Does he/she understand why TDD is valuable in a design context, or do they use it merely for regression cover?

• Justifying technical decisions - can the engineer take a position on any given subject, then defend it, or do they simply agree with everything you say?

• Business analysis - do the engineers make the effort to understand your business, or do they simply want requirements tossed at them over a wall?

• Try to get some objective numbers that indicate agility. What do they think is a good length for an iteration? How often do they commit and push? How often do they run builds? Do they use a CI server? Do they auto-deploy every green build to a staging server? Are they confident deploying to production from master/HEAD? What is their understanding of the term 'velocity'?

• How much time do they spend studying? What was the last technical book they'd read? What was the last technical book recommended to them by a colleague? What books do they have in the company library? Which of those are considered mandatory reading?

• How do they stay in sync with the latest in the technology world? Do they read blogs? Hacker News? Follow role models on Twitter?

• “Always two there are, no more, no less. A master and an apprentice ”. It is standard practice for services firms to pair a senior developer with one or more less experienced developer to help the less experienced developer grow their skills. Make sure that you are getting at least as many senior developers as you are less experienced developers. Ask what skills the less experienced developer will need to learn in order to ramp up. And finally, keep in mind that years of experience does not always translate to capability. Don’t be surprised if the most capable developers have the least experience in years.

At the end of these seven steps you should be down to three or four companies, sorted along the three axes of agility, engineering and price.

Step #8 - prioritizing by integrity.

Integrity is both the most important attribute of a vendor and the hardest to measure up-front. If you can’t trust your vendor, you spend a lot of time hedging against potential screw-ups instead of staying focused on churning out features. Do you really want to spend half your day doing code reviews because you can’t trust your vendor to Do The Right Thing and staff your project with competent engineers?

To get a Reflect on all your interactions with each of the companies - which seemed the most honest? How many times did they simply tell you that they lacked a particular skill-set, or make a similar statement indicating that they were more interested in making sure you had the right information to make decisions over closing the deal?

Putting it all together