We always want to build something that our users love and cherish the experience on the site. To make it helpful for programmers to test their code before they submit, we added a feature which would allow you to compile & test your code against sample test-cases before submitting. Since we were seeing a steep growth within a few hours of the launch, we decided to pump up our codechecker servers.
What is supposed to be naive & simple (adding more EC2 instances with the codechecker AMI) turned out tough and weird. When you launch EC2 instances, the configuration of the system is really not under your control. While the minor configuration changes might be okay for an instance running the web part of the system, it surely isn’t for codecheckers which rely heavily on time, processors, etc. Yes, we knew that but the assumption that beat us was all EC2 instances that were launched with the AMI have similar configurations – nope, they don’t!
So, the new instances that were launched had an older version of XEON with smaller CPU cycles as opposed to the existing ones. Hence the C++ code which was running at less than 3s, ran at ~3.3.s which exceeded the time limit. And hence submitting the “same” solution would cause different results because they were served to different instances. Phew! We had a long night finding the bug and now it’s resolved.
We kept launching EC2 instances till we got the configuration we wanted, everything looks fine and all the submissions the last day have been rejudged. Sorry for the hiccup, our goal has always been to make the experience much better than what you expect it to be.
