Release process and communication about software changes

danielle · July 16, 2019, 11:14am

Hi again, so alongside of looking at great examples of software release/change comms, @sauloperez and @Rachel and I also spoke about how we do the release process and the launch/communication to users needs we have alongside of the technical release process we have.

Here’s what came out of that discussion:

WHO COMMUNICATES

When communicating about changes we use a cascading approach, where the software team communicates to the instance managers (currently via the release notes), who then communicate to their users (via whichever channel, social media or newsletter or whatever).

TYPES OF RELEASES

There are 3 “types” of releases that we have (or would like to have):

Small releases (small changes, bug fixes and tweaks) - version n.n.n
Feature releases (beta or full, the launch of a feature e.g. subs beta) - version n.n
New versions (like Spree 2.0 or a complete overhaul of something) - version n.0

HOW TO MANAGE COMMUNICATION OF THESE RELEASE TYPES

We believe that each of these types require a different level of communication. They all should have release notes (is the theory), however:

Small releases only require a notification in Slack (channel TBD) announcing that it has gone live and linking to release notes.
Feature releases are worthy of a global blog post, talking about what has been launched, how it works, what the benefit for the user is, and even how we want to evolve it next (if there is additional functionality already slated for delivery in the pipe).
New versions require a project management-style approach, where there is a comms plan in place that determines how and when instance managers are involved/communicated to, and how the changes will be communicated to users. Likely this will include blog posts, but in the case were there are staggered rollouts the comms should be designed around this.

IMPLEMENTING THIS APPROACH

Here’s how we think our processes need to change to allow for this release communication:

We start doing frequent, small releases. We don’t wait for things, we work like a release train were each week (as a suggestion of time) we will release the changes we’ve made regardless of size. If you miss the train this week you get on the one next week. <-- @sauloperez will look to implement this, and discuss with devs.
For any feature release there must be a dev (the release manager) and a comms person assigned to create the blog post about the feature. That way the comms person understands what is being released, how it works, and how it benefits the users through the representative of the team who built it, and the message is crafted in a way that is user-friendly and well written. The comms person role can be shared across the comms/product team so that it’s not always @Jen having to write it. And instance managers can then use their own communication channels to broadcast the blog post (or translate it) to their users.
We look at our project management process for large pieces of work like spree, ensuring that things like a comms plan and rollout plan are designed well in advance, that stakeholders are included in these plans and aware of their roles, and that it’s a simple thing to then implement the plan. The key point in here is that we do this at the beginning of the process so that implementation and launch is considered at the same time as we think about design of solution and build process.

OK, so that’s it. We’d like to implement this process, so please reply on here with your thoughts/worries/joy/inputs.

Ping @luisramos0 @maikel @lin_d_hop @Kirsten @Jen @Rachel @MyriamBoure @matt @kristinalim @EmilyRogers @lauriewayne1 @NickWeir @MSFutureFarm for your thoughts.

maikel · July 17, 2019, 12:26am

Happy train

Jen · July 23, 2019, 8:22am

Looks great to me team It would be excellent to work towards incorporating videos along with blogs, but that’s a nice-to-have/translation option for now.

I like that we’re just highlighting feature releases in this way - I think communicating bug fixes to the layperson is dangerous as they just see lots of bugs

sauloperez · July 23, 2019, 11:10am

Love it!! just to add onto what @danielle perfectly explained is that in order to improve we will need to review our assumptions and processes around the delivery of software. key point: everything small!

Quite related: Brainstorming release notes to be better

tschumilas · July 23, 2019, 8:22pm

Love it all. As an instance manager where there is no dev I have an additional request. It would be good to know when the train is coming. Even if its just a day ahead via a slack post? I regularly have users asking, “is it fixed yet?” with regards to bugs, and I’d like to be able to tell them without continually trying the thing out. And - I don’t know the solution to this but - one of the problems I have is that I don’t understand the release notes. They are often very technical (or at least more technical that me or other users), and I don’t understand the implications of the release. Is there a way to put a users perspective to them? ie - ‘this means that you will now be able to delete products without implications for creating new products’ or ‘this means you will be able to edit shopping carts again with errors’ … LOVE this plan.

sauloperez · July 25, 2019, 2:01pm

That’s very useful feedback @tschumilas. Right there is a success criteria for this: you spent less time dealing with repetitive testing tasks and you can better communicate improvements to your users

maikel · August 1, 2019, 8:35am

@tschumilas We struggle with a similar problem of communicating the right bits to the right people. I believe that Sally hat some kind of spreadsheet where she tracked which user had which issue. And if you reference the issue number there, then you can check on Github if the issue is closed and in which release it’s included. The last step is a bit tricky. I recorded a little video how you find the release to a given issue.

visit the issue
go to the associated pull request
choose the last commit of that pull request
and find that little box telling you the version number if it’s included in a release

Rachel · August 1, 2019, 10:10am

@maikel @tschumilas in France we deal with it as part of our support tool (which is how I’ve mostly done it in my previous companies). Here is how it works:

A user send an email about e.g. a bug
I answer to him and label the email “Bug in progress” thanks to the tool we use
Once the bug is fixed I transfer the status to “waiting next release” (but that is optional sometimes I’m lazy to do it it is a good practice however if you want the tool to calculate an SLA you want to communicate. But IMO we are not there yet with OFN, and I’m not even sure this is something we want)
Then, once our server has been upgraded, I just have to review all the request that have been tag in the tool with these two tags and I can send an email to the users to tell them it is fixed.

The rest of the user will only have the general communication Danni’s presented.

So this to me is not related to how to communicate “broadly” but how we deal with user requests on a 1to1 relationship. This is support So we might need another topic for that.

Jen · January 29, 2020, 10:50pm

Hi folks, just checking in on this process with some reflections following yesterday’s release (v2.7.1)

What happened: We released a complete change to shopper experience, without warning shop owners or customers beforehand. Instance managers had been alerted that this change was coming in the next few weeks (probably early Feb) but not that it was coming in this release. Here in Aus we scrambled to inform shop owners and wrote a crap newsletter in a hurry that didn’t explain it well, and it was just luck that I was able to shuffle tasks to write it there and then, and luck that @danielle was online to review the newsletter. I didn’t really have adequate documentation of the logic of the change to explain why it was being made. Shop owners had zero time to inform customers of the change, and some shop owners couldn’t see how to open order cycles - the shop tab wasn’t intuitive, they were just freaking out. There is still no documentation in the user guide to point them to.

Some alternative ways we could approach this:

Where there’s a dramatic change to shopper experience like this, it would be great to be able to alert shop owners at least 1 week beforehand so that they can pass this info on to customers. I think a change like yesterday’s fits somewhere between a usual weekly release and a feature release, and in a case like that it needs to be flagged to not release until it has been communicated out to shop owners and shoppers. (Or in reality, to not be released until a certain amount of time has passed since the instance managers have been informed specifically of the date of a release that needs this type of communication - whether they act or not is up to them)

I would also really like to be communicating changes that are coming to users, rather than changes that have been made - when a significant bug was discovered we didn’t roll the release back because we had just told all of our platform users about the changes an hour or two earlier. If we had told them that it was coming, we could have also rolled it back quickly and more of them would just assume it hadn’t arrived yet.

I’d love to see us including something in our templates so that the logic of a change was more visible to people communicating this on to users who are outside the product /dev team. Yes, I can see the what/why in https://github.com/openfoodfoundation/openfoodnetwork/pull/4655 but it doesn’t speak to user experience - why is this change being made from that perspective?

What’s our process for flagging which things require a user guide change, and allocating someone to them? To me it would seem ideal that the user guide update is ready to hit publish at the same time as we hit publish on the release.

In my dream version of this release, shoppers and shop owners would get a little pop-up that says something like ‘You may notice some changes! We’ve done xyz (insert logic of why) which means that you’ll find all of the products on the ‘Shop’ tab, and all of the info about how to order on the ‘Home’ tab. Check out this video to see a quick demo’

It also feels like we’re falling down on this step as a global team. The fact that I’ve written a software newsletter for Australia - rather than there being a global version that means we don’t all have to reinvent the wheel - seems like it’s not ideal. Sure, I can just share it with the rest of you (it’s here https://mailchi.mp/900662b472f1/new-open-food-network-shop-display-and-more-latest-platform-release ) but that’s too late for a release that’s already out in the world.

What do others think would work? Maybe it just boils down to this tranche of work needing a comms plan and not having it - I’m not sure. So maybe the main change needed is just identifying non-features that need a comms plan?

maikel · January 30, 2020, 3:16am

Yes, that was a big fail where all the people involved of the change didn’t realise that it needed communicating. How could we have done it better? We have a lot of good processes already but we need to improve them here.

Every Thursday we draft a new release. From then on, everybody can review the release notes and communicate changes. It seems like we don’t have someone in Australia to pick this up or translate the release notes. @Jen If you had seen the release notes at the time, would you have known that this change is coming or would it have been too cryptic?
Following the release draft, we ping the testers. I think we should also ping the “release communicators” (new role?).
The release draft contains a version number which should indicate how big the changes are. In this case, we increased only the patch release number indicating that there is not much to communicate. I had a discussion around this last week when I didn’t know how to increase the number for the previous release. We don’t have a document for this approach. The current releasing manual is out of date. Action -> We need to talk about it again and someone needs to write it up.
If a release candidate passed the tests, we publish it on Tuesdays. We then also deploy it to all servers unless someone requested a delay. This delay process is in place but we didn’t use it because we didn’t know that the communication should have blocked the deployment.
The fact that an important change is going into the next release is known once it’s merged into our master branch. We don’t have any communication here. It could be the merging developer who tells people, or the project manager who keeps an eye on the important changes. To be discussed.

I think that our processes are close but one gap in the communication can cause this kind of fail. Let’s close the gaps. And do we need redundancy? We always have two people reviewing pull requests but nobody reviews the release notes?

In addition, I now also noticed that the time to fix an issue is really short. We deployed on Tuesday and two days later the next release is drafted. That is only two days to discover a bug, find the cause, write a solution, review the pull request, test the change and merge it into master.

danielle · January 30, 2020, 5:59am

This was my fault, and I apologise. The only reason I can give for it is the difficulty I am having in managing this work with all the other work I have on, and how crazy non-routine January has been. This is one of the downsides to fitting product owner stuff around all the things. There was actually a couple of stuff ups on this, all learning opportunities, and it’s great that you’ve initiated this @Jen.

I also think it’s because the pipe doesn’t mesh well with things outside of the pipe, so we need to start focusing more on the way we “go to market” with these changes and being more strategic about the bigger items we are doing.

So, ways that I am looking to change this for the mobile ux work:

All changes go out as a feature releases (based on the sub-epics that have been created in github), complete and whole, updated in the user guide and communicated in advance to users. I don’t feel that the current weekly release process that is great for the delivery pipe but not so easy to manage for everyone outside of that, and so I’ve slated a conversation in the next delivery pipe meeting to discuss this and come to an agreement on how changes will be released moving forward.
I will be organise a short weekly catch up about this project, which I will use to manage comms about the work as it goes through the pipe. It would be great to have a user guide representative at these as well as the devs working on the project. I’ll post the notes from this meeting into the mobile ux channel each week, and I’ll share that to the instance managers channel so that you are all aware of what’s going on (if you’re not in the mobile channel). That way there’s one place where we all gather to have a synchronous conversation at least once a week as work progresses. We’ll check in each week to see if there’s been progress, if there hasn’t then we can cancel the session.

In the longer term, I propose that we bring feature flags into the way we develop things. Frequent releases are great, but we need to make allowances for the fact that often there’ll be something merged to the master code base that we don’t want live to users yet.

The flag will give the people who communicate with users and write user guides, etc, the time to do this and will remove the likelihood of this problem ever happening again.

It’s also an opportunity for instances to choose to release access to particular features discretely. We did it with subscriptions, to me this should be default behaviour for all larger projects rather than the exception.

Finally, I agree with @maikel that there are gaps in the process and these won’t be solved by feature flags (ie. when something is little but has a change that needs to be reflected in the user guide or that changes the translations file). There are some excellent ideas in that list, but perhaps we need to have a gathering of communicators and delivery train people and talk through the best way to do this? Topic for the global hangout perhaps?

Rachel · January 30, 2020, 10:30am

Just a small comment around that: I’m really afraid of the quality of having big releases again. The fact that we merged and went live on small steps made us find very quickly 2 bugs and solve them. One of them (the filter on big shop) is something very difficult to catch on staging server. Yes we need to improve our staging data, but this case will always happen. If we decided mobile was going to be released at once we might have missed those and pilling up other small bugs…

Small live releases means also something easier to roll back. So I have the feeling that we have learnt from our previous mistakes and improved the tech side of it by having smaller releases, with one PR = 1 issue.

What we need now is to improve the communication of this process to instance manager, and find a way to nudge them without forcing them to be in a position close to mine (my users where notified in time, but I’m also heavily involved in the pipe).

In short: I would be in favor of putting this up in global hangout rather than delivery train. We need instance managers for this discussion and delivery train is not the right place to invite them.

tschumilas · January 30, 2020, 2:09pm

I love how this community debriefs things!!! Thanks. My perspective is it wasn’t that bad and our processes are doing OK - not perfect - but OK. Maybe for me there was an advantage because I take all the relevant non-pipe roles here: super admin, user support, communications, translations, … So - I knew it was coming, I knew there would be changes to the tabs. I just didn’t know when (as @Jen said) so I didn’t give our users a heads up. I saw the change, and I thought it was intuitive, and I assumed our users would think it intuitive too (so that was my mistake). I agree that having non-pipe people review and interpret release note, update the user guide, and prepare user communication materials is ideal. BUT that is a budget issue. The downside of allocating most of our global budget to pipe (devs, design, testing) is that all the other functions — the functions that we all agree are key at times like this – are being done on sparse volunteer hours. At several points @danielle posted about updates to the user guide - but none of us had the time to address this. If we wait until the non-pipe aspects (coms, user guide) can be done, releases like this will be significantly delayed - and I don’t want that either. Do we need to revisit budget - and re-allocate more time to non-pipe roles when we are doing larger releases?

I"m not whining - just debriefing. Our bottleneck - when it comes to larger releases - seems to be with coms, user guide, super admin stuff.

A suggestion for going forward with the mobile changes… can we take the time now to do a forecast for our users - step by step - what to expect and APPROX when they’ll see the changes? (I have hubs asking if it’ll be ready by…?) And can we pay someone a stipend (does anyone have the time?) to update the user guide as we go.

danielle · January 30, 2020, 11:42pm

To clarify, I wasn’t wanting to do big releases so much as break things into chunks. For example, in mobile I have set up the epics to be smaller, releasable chunks that provide value. The individual PRs alone mean that we’ll have a frankenstein approach to development, but I can’t make the stories the size of the epics because the PR would be too big. Thus the compromise of trying to release a stone (epic), not the pebble (story) or the boulder (feature).

I believe that releasing the epic as a complete thing still allows us to find the bugs easily. Case in point, the tabs - we could have finished the epic of changing to tabs/restyling them and the static content inside of each and still easily found the same bugs. The next epic is the product list, which has a few stories but I firmly believe that all these stories should be done before the new product list is released. Same with the order cycle display changes, same with the search and filter changes.

So my proposal is not a big bang feature release, it’s an iterative release of sub-parts.

I would love to do this @tschumilas and can definitely take everyone through the proposed chunks of changes that are coming up. The problem is that I can’t tell you when they’re likely to happen because the work is being done ad-hoc by @Matt-Yorkley mostly, @maikel is also supposed to be working on it but has so far been on bugs and other things that have meant we haven’t got as much done as I was hoping. That’s why it’s a week-by-week proposition, and part of the reason why I’m finding it hard to juggle (if I knew the whens I could better plan my time around them).

The next global gathering I’ll do an overview of the mobile scope and the chunks to be released. And we can also chat about what to do to solve this problem. We’ll need all the instance managers to be there, as @Rachel says, and also user guide representation as this is a core part of the release process.