The downfalls of environment branching patterns

Working for a software consultancy is fun, I get to see various client setups, coding standards, and branching strategies. Sometimes these are setup before I get there, I didn't get a chance to help shape them; sometimes they are fine, sometimes they have issues (every decision has a tradeoff). Either way, it's a fun challenge to see how I can help improve the process.

For real life (the scenario)

Not so long ago, I did some Azure DevOps work for a client (obviously won't name names) they had a setup similar to:

Azure DevOps
Coding with tools I hadn't used before (doesn't matter which)
Pipelines as code (YAML)
Git
Deploying Docker containers to Azure
Multiple environments (Dev, Test, Prod)

There were a few notable things that I discovered:

they were following an environment branch pattern e.g. a separate branch for dev, test, and prod (a path I'd usually not tread)
there was also a main branch
they were using pipelines as code (which is great) and the pipelines were triggered off of changes to any one of the branches
as a result of the above, they had to replicate the pipeline changes in each branch
also as a result of the above, they had to rebuild the code for each environment
to keep branches in sync they were using merge commits (I prefer rebasing)

Let's dig further

The first question that naturally came to mind was "If I am going to start to write some code, which branch do I start from?". I expected main.... the response from the client was a bit of a curveball.

Them: "Oh, we don't use main, at the moment we are currently working on test since we need to add some features there."

Ok, that's odd. From what I gathered, the repo was setup with main and a different vendor setup their pipelines with it, but the client wanted to have environment branches instead and just didn't remove main. No biggie, just an oversight.

I dug a little deeper and had a look at the commit messages, lots of merge commits in the different branches but it looked like one branch was well ahead of dev and it was hard to tell if they had been merged back.

Them: We are working in test at the moment, We'll merge it back afterwards, it works for us.

Optimistically, it sounded like they are just under the pump and will get to it later. I didn't want to push too hard, I was there to help them with their pipelines, not their branching strategy.

Given that I'd have to make changes in all 3 of their branches (and not squash any of their changes), I felt it was important to flag the potential issues in their process:

Any pipeline changes would have to be replicated in each branch. This would be a manual process and prone to human error.
Since the pipelines were triggered on any branch, it was possible to have different versions of the pipeline in each branch.
The merge commits made it hard to tell what had been merged back and what hadn't. This would make it hard to know what was deployed to each environment.

Let's get... stuff done

Done - Issue raised with the client, email sent with recommendations.

It was time to roll up my sleeves and dive into the actual task at hand — tweaking their deployment pipeline as initially requested. I got into it, made the necessary changes, and got the Pull Requests (PRs) merged into the respective branches. It felt like a well-earned victory, albeit a small one in the grand scheme of things.

Fast forward a few weeks, my phone buzzed with a call from the client. The news wasn't good. The changes I had made weren't working!

Surprised, I opened up the repo, only to discover that my changes weren't there. WTF! They had been reverted! Turns out instead of using PRs to merge changes between environments, the client had used the Git command line to do the merges and then pushed the changes to all the branches. Normally this wouldn't be an issue but in this case some changes were missed.

After re-implementing my changes, and informing the client what had happenned. This incident, though frustrating, served as a catalyst for the client to get me to get rid of environment branches (it helped that they were not live yet).

Since they were happy with always rolling forward my choice was to go for Trunk Based Development (TBD), It is a source-control branching model that emphasizes merging all developers' work to one shared "trunk" or mainline. They do this frequently and in small, manageable increments. It's a model that eliminates the complexities associated with managing numerous branches, thereby minimizing the chances of manual reversion mishaps like the one we had just encountered.

Other options I would've recommended were:

GitHub flow
Release flow

Whilst they were contenders, as I've had success with them before, the simplicity and efficiency resonated more with the client as it would be very clear what had and hadn't been deployed.

With the change to TBD, the client was able to enjoy the following benefits:

Fewer branches to manage
Using PRs to review changes
Using PRs to have a linear commit history
Triggering pipelines on main only
Build the container once then deploy to dev automatically
Manual promotion of a deployment to test and prod (for now) - super fast now that the container is already built
Same container image is used e.g. promotion to test uses the same image that was just on test aka binary promotion

This incident was a stark reminder of the critical role that clear and repeatable branching strategy plays in software development and deployment lifecycle. And sometimes, it takes a minor hiccup to propel a significant leap towards better practices.