The CodeSprint 2 Post-Mortem and Announcing More CodeSprints

From Friday, January 6th to Sunday, January 8th, 5000 programmers from around the world participated in the second edition of CodeSprint, a programming contest designed to discover talent and find programmers jobs at 86 companies. This post is a complete post-mortem of that event, and also announces the details and changes we’ll be making for CodeSprint 3, along with two completely new Sprints: SpecialtySprints and CompanySprints. [Skip ahead to future Sprint details]

CodeSprint 2 Statistics

The Participation Funnel

The numbers for the four main stages of participation:

Over 5000 people participated in CodeSprint 2, with 20% scoring well enough to apply to companies. Similar to CodeSprint 1, there is a noticeable gap between those who did well enough to be eligible to apply to a company, and those that actually did apply. Because we gathered additional profile data as compared to last CodeSprint, we’re able to further examine this gap. Here’s the breakdown of the 1003 eligible candidates, if they applied, and if they wished to stay hidden from outside companies.

We can say for sure that 5% of eligible programmers participated just for fun. The motivations behind the remaining 283 non-applied is slightly little more speculative, but a birds-eye view of the data tells us that most of these were in the 600-1000 ranking range, suggesting that it was a lack of confidence in their performance, more than anything else, that caused them to not bother applying. As we shall get to in the CodeSprint 3 details, we will try remedying the perceived non-competitiveness for these 600-1000, who, after all, still scored in the top 20% of all candidates.
One last note: Of the 18 who applied to companies, but did not want to be contacted by other companies, only 2 of them were non-students. The use case of “I’m secretly trying to escape my current employer” does not yet seem to be a popular one.

Participant Demographics

Demographic data is reliable only for participants that were eligible to apply, and, even then, there is some uncertainty. 6% of 1003 eligible programmers never filled in their country of origin, and 21% declined to share their current professional status.

Despite being now open to everyone, the participants of CodeSprint remained firmly university-centric. Over 65% of all eligible participants are currently attending university (or high school!). A quarter are looking for internships.

Geographically, CodeSprint 2 was nearly ⅓ US, ⅓ India (where InterviewStreet got started) and ⅓ everywhere else. We had eligible participants from 60 countries, slightly more than a quarter of all countries in the world, or alternately, the total number of countries in Africa.

The CodeSprint 2 Company Attractiveness Index

86 companies participated in CodeSprint 2. Below is a table of the 25 most attractive companies, sorted by number of weighted applications. (If programmer applied to 3 companies, each of those 3 companies receives 1/3 of a vote). Microsoft actually had the highest absolute number of applications by a very slim margin, but its applicants were slightly more promiscuous as compared to Facebook. On the whole, weighted applications followed absolute applications very closely, with the only other significantly selective companies being Groupon and Chachii (an Indian company). A histogram of all company applications can be found in the CodeSprint 2 Discussion section further below.

For CodeSprint 2, we limited applications to 10 companies per programmer. Roughly 30% of applicants maxed out, while the remaining 70% applied with a fairly even distribution with respect to the number of companies. In hindsight, a limit of 10 turned out to be a pretty good cap, and it doesn’t seem like it was particularly onerous.

Interview Invites

Compared to last CodeSprint, CodeSprint 2’s Invite Data has a greater possibility of systemic error: We explicitly exposed programmer email addresses to companies they applied to, and encouraged companies to reach out directly to these already-applied candidates. By reaching out directly, companies sometimes forgot to mark programmers in our system as being invited. (In fact, we here at InterviewStreet are guilty of this, we forgot to shortlist candidates, and instead reached out to them directly) Thus, this data is likely an underestimate of the true number of invites.

45 companies used our internal invite system to invite candidates to interview. 2 of them went overboard, inviting over 150+. We’ve made a note in our emails to programmers who they are. The company that sent out roughly 100 invites is large enough to make that quantity of invites reasonable.

In addition to this data being an underestimate, we’ve personally received word that 2 Very Large Companies were not able to look at candidates last week, and we’ve arranged for them to review their applied candidates late this week.

Removing the three biggest invite sources, candidates were invited in according to the following distribution. Roughly 300 people in this subset were invited to interview with at least one company.

Scoring and Contest Questions

Those that wish to view the contest questions, along with test cases and solutions, can do so at cs2.interviewstreet.com

The contest consisted of 15 questions: 3 Real World Problems, 4 Company Problems, and 8 Algorithmic Problems. Partial credit was given linearly to those who solved only a fraction of the test cases used to score a solution.

In terms of submission activity over the 24 hours, there was a noticeable uptick in the waking hours of the US and India, but overall, submission activity was more level than it wasn’t.

Once again, the C family was the clear plurality of language use at 43%, but did lose some ground as compared to last CodeSprint, with Python and Java taking up the majority of that loss. A graph of languages used per problem can be found here.

CodeSprint 2 Discussion

The Non-Effect of Company Profiles and Recruitment Videos on Applications

The inclusion of detailed company profiles for CodeSprint 2 was so that programmers would be better informed in choosing their applied companies. Even though, anecdotally, we have had great feedback on providing such great company information, it doesn’t seem to have had a substantial affect on aggregate application behavior:

There is no discernible trend between inclusion of additional company information and the number of applications the company received.

To be fair, this graph considers existence but not quality; a company may have filled out a profile, but badly. There were definitely some of these profiles, which I will tactfully call “unattractive”, but in the long tail we also had great, unknown startups; ones that I’ve personally talked to, who included a great company profile but still received <10 applicants. We here at InterviewStreet have a burden not only to find programmers great jobs, but also to help these more unknown-but-great companies find talent.

Clearly, profiles and video alone are not enough. Solving this problem was one of the main factors in us deciding to try out SpecialtySprints.

The CodeSprint Business Model

This was the first time we attempted to make money off CodeSprint. We used a pay-per-hire model, similar to technical recruiters: It’s completely free for companies to sign-up, look at candidates, and interview them. The only time they ever pay is if they actually hire someone. What’s more, unlike traditional technical recruiters, companies only pay a flat fee, not a percentage of salary.

We thought this business model was a good idea when we adopted it, and now that we’ve run it through one CodeSprint, we’re pretty convinced that it’s the best business model possible.

First off, getting companies to sign up has been an easy sell; there’s absolutely no risk whatsoever for companies to sign up and look at Codesprint Programmers. Secondly, by adopting a flat fee, there’s no incentive for us to place programmers at any particular company; all companies are monetarily the same. With pay-per-hire, CodeSprint is not optimizing for the number of companies, nor the number of programmers, nor the number of invites, nor the number of interviews. We’re designed to focus on one thing, and one thing only: effective hiring.

Needless to say, we’ll be keeping this model for the foreseeable future.

Test Cases and Problem Errata

In general, we’ve received nothing but great feedback on our problem statements. Everyone we’ve talked to has said that the problems, on the whole, are damn interesting. With that said, we’ve also received a lot of feedback, both in CodeSprint 1 and 2 about test case and problem statement quality. While we don’t want to give everything away with test cases and examples, the problems lacked polish. With the introduction of SpecialtySprints and CompanySprints, which have fewer problems, we’ll be able to give a lot more effort to each problem, and by CodeSprint 3 we will have hired enough additional problem creators to do the same.

The Development of Real World Problems

For CodeSprint 2, we had two main development goals: the Programmer Viewer (aka the Hackerboard) and Real World Problems. The Hackerboard was a decent sized undertaking; it took a fair amount of time to create and was released with a couple of rough edges, but it was still a very manageable project. Real World Problems, on the other hand, was a different beast entirely.

From its announcement up until CodeSprint 2, someone was working on Real World Problems every single day. Considering how much difficulty we have with just our regular CodeChecker, Real World Problems soaked up an insane amount of time and effort to get it to work correctly. Don’t get us wrong, we’re extremely proud to have created the Real World Problems infrastructure, but this project  was our equivalent of General Motors launching Saturn; a car with too many things to specially craft at one time. Real World Problems caused us to be underprepared for many other tasks, and is the main reason why our problem and UI polish was lacking.

Announcing CompanySprints and SpecialtySprints

In the past week, there’s been a lot of internal discussion at InterviewStreet on how often to hold CodeSprint, with possible frequencies from every month to once a year. It’s a decision with a ton of incomplete data and arguments from both sides. On one hand, we’d like to be able to cater to companies and programmers year round, and not have to make them wait a few months if they need a job / a job filled now. On the other hand, we don’t want to fatigue anyone, least of all ourselves: CodeSprint is crazy exhausting to host. As is all too common, the solution came to us in the eleventh hour, only after we took our minds off the problem and reframed it. The end result: Smaller Sprints, in the form of CompanySprints and SpecialtySprints. These will be more targeted, and most importantly for programmers, will be on a much smaller timescale: between 3 and 8 hours.

CompanySprints

CompanySprints are somewhat self-evident: They’re a CodeSprint for a single company. The 2-5 problems will be company-experienced problems, and exclusively appropriate to the positions that they’re hiring for.

The first CompanySprint will be CompanySprint: Quora, which will launch in roughly a month. It will likely include a relaunched, real-data version of the Quora Classifier problem. CompanySprint: Justin.tv will occur a month after, and we may be able to organize one or two additional CompanySprints even before that.

SpecialtySprints

For smaller companies, we’ll be holding SpecialtySprints, Codesprints designed for a single job type. The first one will be SystemSprint, which is most closely related to our area of expertise of algorithmic problems. It will contain many of the UI and Problem changes discussed below, and will likely take place on March 16th, the old date for CodeSprint 3. (We’re looking into if we can hold it even earlier, though) Following SystemSprint, and in preparation for its inclusion in CodeSprint 3 will be MobileSprint, focused on iOS and Android problems.

CodeSprint 3 Details and Changes

Less than 24 hours ago, CodeSprint 3 was originally schedule to take place March 16th, 2012. This has since been tentatively delayed to May 11th, though we may change this date depending on our experience with CompanySprints and SpecialtySprints. (To assure our loyal student user base: We will definitely have a CodeSprint take place in August / September, designed for the University Fall Recruiting Season) Whenever the next CodeSprint will be, however, here are some of the things that we’ve identified that will change.

More Company Problems

Our experiment with Company Problems was very well received by both programmers and companies. Programmers felt the problems were more interesting than algorithmic problems, and companies found Company Problems to be (somewhat self-evidently) better in assessing candidates. In marketingspeak: It’s a win-win situation for both sides of the equation, and so we’re going to ramp up Company Problems as hard as possible. We’re shooting for 25 problems for CodeSprint 3, more than CodeSprint 1 and 2 combined.

Introducing Specialty Weightings: AKA Problem Organizers

25 problems is a lot! It’s a lot not only for us to create, but also for you guys to do, and for companies to evaluate. In order to make sense of these problems, we will include Specialty Weightings. For example: a DBA Weighting, a Lead Developer Weighting, and an iOS Developer Weighting.

Weightings will determine the point values of a problem. The Quora Nearby problem, for example, is mainly a back-end programming problem, and so could be worth 25 points for the Back-End Programmer Weighting, but only 10 points for the iOS Developer Weighting. A Code Review problem would be useful for to evaluate both a Lead Developer and a QA Engineer, but with differing relevance.

Multiple Leaderboards

All problems will be evaluated on a simple 0-100% scale, and be multiplied by the weighting to obtain your total score and rank. We are eliminating bonus points. (which no one really understood in the first place) As you can imagine, it no longer makes sense to have just one Leaderboard, so each Specialty Weighting will have it’s own Leaderboard. For those of you that still want to compete for fun and bragging rights, we will still have a Generic Programmer Weighting which will become the canonical Leaderboard for CodeSprint Hall of Fame purposes.

We will release Specialty Weightings prior to the beginning of CodeSprint 3. Hopefully, programmers will enter the contest not planning to tackle a set of 25 problems, but only the dozen or so relevant to their interests.

Relaxing The All Languages Requirement

We’ve hesitated in including some kinds of problems in the past, because of their inability to be solved in all languages. Certain iOS problems, for example, can’t really be solved except in Objective-C. The same is true for many front-end issues, which is largely a JavaScript-only domain. Now that we have such a large problem space, we can include these languages as part of CodeSprint, and increase the breadth of our testing ability.

We be adding Problem-Creator-approved libraries for specific problems, such as Quora Classifier.

More International Companies

Our 86 companies were centered around our two major demographics, the US and India. Given that ⅓ of our participants are from elsewhere, CodeSprint 2 severely underserved some of our programmers. Of particular note is Europe, with 15% of our programmer base and only a handful of available positions. We’ll be making a more conscious effort to serve this region, along with the rest of the international programmer base.

General Sprint Improvements

More Interesting Algorithmic Problems

We thought our problems were really good, and our users overwhelmingly agree. But then someone showed us that we can do better. Not all of our problems will be that cool, but we’ll certainly try.

Better Test Case Feedback

Problem test cases will be baked into the design of the problem: Each test case will be included to test a specific part of your solution, whether it be an edge case or computational speed. Most importantly, test cases will have an associated message that, should your code fail, will let you know why the failed test case was included. It’ll be up to the discretion of the problem creator exactly how much information to reveal, but our goal is that you will always have an idea, at each stage of the problem, what you should be fixing in your solution. On rare occassions, when it doesn’t spoil the answer, we will also include more complicated sample cases (Quora Classifier would have benefited from this, Coin Tosses would not)

Small UI Fixes

Four things that we meant to include for CodeSprint 2, but didn’t get around to it:

• A countdown timer, so programmers don’t submit their final submission late
• A real-time, persistant information feed, so you are immediately notified of any announcements / errata, and can look back in it’s history to see if anything got changed.
• A very clear notification when you’re looking at the problems at interviewstreet.com, instead of codesprint.interviewstreet.com
• Country and Entry-Level filters for navigating company positions

Thanks and Congratulations

CodeSprint 2 was not quite as rushed as CodeSprint 1, but it was arguably more nerve wracking; this time, we had a reputation to keep.

We’d like to thank all of our companies for participating, but especially the brave ones that contributed a Company Problem: Groupon, Quora, and Newsle. We really appreciate the effort you went through to give us fully formed, completely drop-in problems.

On the student side, we’d like to thank the following people, who helped us immensely in IRC chat guiding people and helping them out: pedrosorio, aquatsr, as_df, sontek, Timothee, and TheCalled. If you see your name here (and your IRC handle was the same as your CodeSprint handle), expect an email from us sometime later today.

Finally, to the 665 applied candidates: Good luck.

Until next time,
The CodeSprint Team at InterviewStreet

If you are a company looking to participate in a Sprint, (Code, Specialty, or Company) please contact codesprint@interviewstreet.com

If you are a programmer and would like to receive Sprint Updates, please subscribe to our mailing list at codesprint.interviewstreet.com