Some software bugs are harder to track down than others. One type that is particularly difficult to diagnose occurs when the same codebase behaves differently for software developers on the same team. For example, working code that a developer checks into a shared repository may not be able to run on another developer’s computer.
Other times, the opposite might happen. Kirk Docker, technical lead at HR software vendor myhrtoolkit, was once on a development team that discovered a bug on the production and testing environments but was unable to reproduce the issue on their local computers. The team tried various ways to track down the bug, including stepping through the code line by line, with little luck.
“This was a ‘It works on my machine, but it shouldn’t’ situation,” Docker said.
5 Tips for Tracking Down Runtime Discrepancies
- Check whether configurations are accurate when transitioning between environments.
- Use containers for cross-platform compatibility.
- Run scripts to weed out database differences.
- Use package managers to manage versioning differences.
- Keep clear documentation, especially for the onboarding process.
They eventually realized that the cause of the discrepancy was that the developers were using a different version of the team’s coding language, PHP, from the version running on the servers — normally a nonissue because language versions tend to be backwards compatible.
“In this case, however, there was a change in the behavior of accessing strings as an array which changed between the two versions, causing the bug on one but not the other,” Docker said.
Sometimes, like with Docker’s team, just getting a few more developers to take a look at the problem can go a long way to solving this type of puzzle. But there are also strategies teams can follow to minimize the likelihood of running into truly head-scratching “It works on my machine” moments.
Beware of Transitions Between Environments
One of the most common instances of the same code behaving differently occurs when code is run for the first time in a different type of environment. This can happen when local developers move their code to a test or staging environment, or when it’s moved further to a production environment. Applications usually have many configurations that need to be changed, so it can be difficult for developers to keep track of all the necessary changes despite that being the goal for a smooth development to production transition.
“That’s the ideal — you want your development environment to be as close to production as possible,” said Eric Elsken, co-founder and engineering lead at data orchestration platform Shipyard. “But you don’t necessarily start building the application thinking about how it’s going to run on production. If you do that, you’ve been in the game for a long time and you really know what you’re doing — but in my experience, it becomes kind of an afterthought.”
For some projects, moving from local to higher environments may include even bigger changes than a simple configuration difference. Teams may decide that it makes more sense to host the production code on a different type of operating system, resulting in developers needing to port the code over. Developers could find themselves needing to host their code on a cloud provider that doesn’t have the exact operating system developers were using on their local machines, for instance. Even machines that run different flavors of Linux like RedHat instead of Debian or Ubuntu instead of Fedora could manifest subtle problems, Elsken said.
“That’s the ideal — you want your development environment to be as close to production as possible.”
When different environments require different configurations in the code, it can lead to a lot of trial-and-error mistakes. Development teams have to root out the source of bugs and fix them. “A mismatch in the configuration of the machines,” according to Docker, is one of the most common causes of “It works on my machine.”
That’s where containerization comes in handy. After building working code on their local machines, developers can move the code to containers and test how the code works with tools like Docker (no relation to Kirk Docker). Once the code works in a container, it can be easily moved to any other environment.
“‘Container’ kind of means cross-platform compatibility,” Elsken said. “If you can get a container to work in Docker, then anywhere that can have Docker installed you can get that container to run.”
It also has the advantage of giving structure to the often haphazard practice of migrating code between environments. Tools that help with the containerization process like Docker Compose allow developers to streamline the configuration process and make sure nothing is accidentally left out.
Check for Database Differences
Another common problem that can cause unexpected behaviors on different machines is differences in the database. This can come up when developers or members of the QA team are testing and there are missing assets in the database that are needed for testing particular features.
“Maybe to invite a user you need specific permissions, or to use a blueprint feature you need to have a specific entity already installed in your database,” Elsken said.
It can be difficult for QA team members especially to test an unfamiliar codebase while trying to get the database in order to do it properly. They have to be able to see the same code execution as developers in order to greenlight features for release, but it can be a struggle to line up everything the same way, leading to a lot of “It works on my machine” problems.
But a couple of strategies can help. One way is to use database scripts to track and rebuild changes to databases. Instead of making one-off changes to databases during the development process, engineers can record each database change as a script, making it easy to spin up the same database when another person downloads the code for the first time. This method is especially helpful for QA teams that would no longer need to build out the same database manually before testing.
Developers can also write documentation that carefully keeps track of the entities in the database needed for testing specific functionality. This documentation could include information on what tables need to be populated and what types of permissions are necessary before testing can be successfully completed.
Developers Should Check for Consistent Versioning
Versioning discrepancies can also lead to unexpected differences in behavior. Versioning exists on several levels, both on a system level with the version of a programming language being used and also on the level of packages within a programming language. Elsken likened it to installing Office products on a user’s personal computer.
“It’s kind of like installing Microsoft Excel — do I want to install Microsoft Office 2013 or Office 2017? And there are going to be differences in those,” he said. “Within that, the package version would be kind of like a plug-in, like a theme for building a PowerPoint in 2017, updates [to] that theme, a new style or something.”
“Computer software and operating systems have become so complex that even if we think we have the same configuration, it’s very likely that that’s not the case in some subtle way.”
Differences in versions installed for either level can cause problems. At the programming language level, developers can simply use tools like Docker to ensure all environments are using the same version. It’s also fairly simple for developers to manually keep the framework version consistent because updates are infrequent.
But keeping track of versions for packages can be trickier. Package managers can help with versioning on this level, Elsken said. Developers can use package managers to lock specific package versions on their applications, forcing anyone who downloads the codebase to also download the same package versions as they have. Even with these precautions, however, developers should keep vigilant about differences in versioning on their systems.
“Computer software and operating systems have become so complex that even if we think we have the same configuration, it’s very likely that that’s not the case in some subtle way,” Docker said.
Greenfield Projects Can Take Advantage of Newer Technologies
Some of these discrepancies can be improved by adopting newer technologies that were created with these issues in mind, while other ones can be improved by organizational changes like creating clear processes and consistent documentation for developers.
“It works on my machine” problems are more likely to manifest when new hires are being onboarded into the company because there’s a lot of setup and configurations involved in the onboarding process, Elsken said.
Newer companies taking on greenfield, or brand-new, projects have an advantage when it comes to solving these types of issues over companies that have to deal with a lot of legacy code. They can adopt technologies like Docker from the beginning and sidestep many of the problems around configurations and versioning.
“It really comes down to the company or the ecosystem that you’re working with,” Elsken said. “There are best practices that you can follow nowadays where you would hypothetically not run into any of these issues. However, it’s always easier to start out and do that than it is to take something that was built years ago and upgrade it.”
