As part of global strategy we have an objective to be a Single, Cohesive OFN Unit. One of the priority goals in this objective is “any system administrator can support any instance”
- Instances being set-up in a standard way so easy for any sysadmin to hop in and work with it - see sys admin standardisation
- Access - sys admins can access instances when needed [ad hoc, but gradually happening]
- Process - people knowing what to work on and when and what is ‘funded’ e.g.
- what has to be done
- how urgent - how is it prioritised alongside / within global delivery pipe
- if/how this work will be paid for
This post outlines a possible way forward for 3 (NB. ‘straw man’ for discussion!). Summary first with reasoning / detail below. @MyriamBoure @danielle @enricostn @maikel @lin_d_hop @tschumilas - thoughts?
- Github global sys admin pipe - @danielle to advise how she wants me to set this up e.g. project?
- Global sys admins can count in their ‘global hours’ tasks that are:
- minor support for new non-funded instance with own devs (Slack #deployment)
- pick from top of Github sys admin github pipe
- a) new instance or b) maintain/upgrade, where ‘funded’
- c) call from any ‘established and contributing instance’ to emergency support. Issue in GH pipe and then call to Slack deployment for immediate help
- The sys admin github pipe may also contain requests from non-funded/non-contributing instances e.g. ‘upgrade USA’. These tasks can be done at individual discretion (e.g. volunteer) or if discussed and specifically agreed by global community for strategic reasons.
- Established instances like Aus, UK etc maintain own sys admin resources for now
Sys admin tasks can be tagged to instance in Toggl where relevant so we can keep an eye on costs vs contributions
REASONING / DETAIL
What has to be done
Types of tasks / issues that arise:
a) Establish an instance
b) Maintenance and upgrades
c) Help! Emergency: someone’s server is down, emails aren’t working - includes troubleshooting after release if issues seen on one instance first but will likely affect everyone
As they are mostly clearly specified (at least the problem is), time critical and non-negotiable, I think the discourse icebox process is too onerous.
Required tasks in a) and b) like ‘set up instance’, re-provision XX, upgrade yy are easily captured in a github issue. @danielle as the github master - can you recommend whether this should be a project, milestone or how would you like me to set it up?
When a c) occurs the reality is that Slack is the most likely place to deal with it. Would be best if the instance with the problem raises a github issue with as much info as they have, whacks a P1 / S1 (or whatever we come up with) label on it but then hits Slack #deployment to call for urgent attention / response / help
How urgent - how is it prioritised alongside / within global delivery pipe
We could likely develop a severity/priority labelling system like for bugs. Let’s get this set-up and I can put the things I know of in there and then propose a prioritisation process
If/how this work will be paid for
Reliable and experienced sys admins are frequently called upon to support deployment and other instances. Some of these requests have budget and some don’t.
Maybe best to think this through with some examples.
Case 1: e.g. Indonesia
- new instance, own devs, no money
- support required: responses to snags / questions on slack
- core funded? allow for core and occasional devs to support
Case 2: e.g. USA
- new/established instance, some dev but limited and need help, no money
- support required: hands on assistance with set-up, trouble-shooting, upgrades
- core funded? Global decision as to whether strategic to extend support in absence of funding. Likely very limited
Case 3: e.g. Canada - ‘established and contributing instance’
- established instance, no dev, has money
- support required: all sys admin
- core funded: yes. $$E into global pool and tasks taken from github sys admin pipe by whichever dev is best placed to do them. May maintain a relationship with key sys admin contact e.g. @maikel to know who to contact directly for c) cases, but likely this will fade over time once global process working
@tschumilas once we agree and adopt this process, and @danielle sets up github (or tells me how she wants it) I will create issues for the things you want done
Case 4: e.g. Aus, UK, Katuma - ‘established and contributing instances’
- established instance, has own sys admin, has money/resources, likely has custom ‘surrounding’ sys admin needs e.g. additional wordpress connections, forum etc
- support required: none / all
- funded: yes
- Option 1: retain own sys admin resources e.g. Aus keeps some money and time back in Aus, out of global pipe to ensure our sys admin tasks are dealt with when and by whom we want
- Option 2: put our ‘allocated’ sys admin resources (2 hours per week of Maikel) into the pool. Then put our needs/tasks into the sys admin pipe. Again probably assigned/done by Maikel in first instance but perhaps this could change over time
- if our Aus allocation is not needed for Aus it would just flow into surplus for commons sys admin needs. I think this is most likely, our sys admin need here has been total 12 hours this year and that’s because we just completely redid our production server (upgraded and onto cheaper hosting ) and was a total of 20 hours all last year
- if is not enough?
I feel like Option 2 makes more sense in the long term, but with the different surrounding sys admin needs it likely still makes sense to keep it separated. At least until everything is clear and working well for Cases 1, 2 and 3? Open to discussion.
Continuing the discussion from Single global dev team: integration process, rules to be certified (and thus paid), and rates: