It feels like I’m “evangelizing” about that since forever, but people still keep coming and some of them really have problems with understanding that - so I’ve decided to write it all down for future reference - you know, the RTFM-like kind of thing.
Your test environment is a sacred ground, period.
You won’t be allowed to violate it with actions that won’t happen on the production environment, unless environment owner decides otherwise -> for test environments my team(s) do deployment on, I am the *owner*.
By hostile actions that will be responded with furious violence I mean (just examples, not a full list, or even close):
- write access to DB for any developer (!)
- debugging on the test environment (!!)
- manual change in any kind of configuration media (!!!)
- overwriting the binaries with something that wasn’t deployed in the official way from the CI artifact repo (!!!!!!!!!!!!!!)
Yes. And it’s all for your own good. Here’s why:
It’s not just your code that’s going to be running later on production. There has to be a proper configuration, everything has to be deployed to the correct destinations, proper rights are to be set, OSes to be configured, capacity granted, etc. All of that makes a pretty fragile construct of activities and artifacts that is not only complex and multi-dimensional - the most important thing: it has to reliable, so it requires a lot of testing. And in this particular case, testing means:
- compiling+deploying
- compiling+deploying
- compiling+deploying
- …
As much as possible. In the very same way on every environment starting with first tier of test environments and ending with production.
Automated (and as continuous as possible) deployment is the key to that. The more frequent - the better. The more automated - the better. The more intelligible to average developer - the better. The key word is “unequivocal”, because there has to be ONE, CLEAR WAY to perform every activity, so there will be no doubt about what has been done and what are the differences in output effect (direct or indirect)
The main mistake people do is the “fast path” aka “dark side”:
- "I don’t have time to wait for the deployment mechanism to go through full cycle, we have to start testing now! Do it the fast path, just upload the bin library I’ve send you via mail!"
Over my dead body. Not sooner.
If something’s too slow, if something’s too inconvenient, if something’s just too painful - do it more often! Don’t get tempted by “fast” (aka manual aka error-prone aka tribal knowledge-based) path as:
- you’ll lose all the incentive to continuously improve your processes and tools
- increasing number of mistakes, time spent on diagnosing the problems and lack of motivation of people who do “robot” work will do fatality to your team’s productivity (and morale)
What are the ground rules then?
- Compilation and deployment is running in an organized and transparent way - the same apparatus all the time: clear rules and traceable outputs.
- Configuration is an artifact as well. The elements of configuration that differs between environments are abstracted away as a list of “tokens”.
- Every persistent change to so called “configuration element” (distinguishable <versioned> element of target environment) is tracked in a simple and transparent way - that’s not only about code (that’s tracked on source control), but also about RDBMS DDL statements (alter tables, etc.), creating windows services, etc. - no SQL scripts as mail attachments!
- People test under conditions as similar to production environment as possible (following the common-sense, of course). If you have doubts, ask yourself: what would you do if this situation occured on production? For instance - if you can’t make this change in DB, because your UI prevents you, what would you do after system is live?
What about the poor developer?
Tester has found a bug on test environment, but it seems that it happens in this particular case that’s already present (and repeatable) on the test environment. Isn’t it cruel to prevent the developer from doing his work by denying him rights to troubleshoot with write access there?
Maybe it’s cruel - I don’t care. This ground is SACRED.
We can provide you all the stuff you’ll possible need:
- the binaries
- configuration
- DB backups
- deployment scripts, etc.
Hell, we can even aid you with setting a side environment for this one particular bug if it’s critical. But keep your hands off sacred ground - it’s production in a nutshell, it’s your guarantee of stability and predictability that will let you sleep well the night before GO LIVE - respect that.