Yeah, I don’t think hiring someone new for this is a great solution. Firstly, the amount of actual hours of sysadmin work that needs doing in any given month could be maybe 0-5? Weeks can go by where there really isn’t anything to do. How tempting is that as a job offer?
When a real problem does come up, it’s fixed a lot faster if the person investigating is very acquainted with our specific setup. It would be very slow otherwise.
Also it would require giving full access to all our servers to someone that’s not really in the core team, which I’m not sure is a great idea…
If I’m doing some sysadmin-specific tasks I usually track them under the “operations” tag in Toggl. Looking at this year’s entries, I have around 23 hours over the last 10 months. So on average it’s ~2.3 hours per month, and that’s for various issues, investigations, tasks, migrations, and emergencies across all instances. 4 of those months had no hours at all. It doesn’t include deployment which is generally under the “releasing” tag, but it does include work on two server upgrades we’ve done in that time (which look to be around 3 hours each). Task descriptions since January 1st include:
Aus server migration
French email errors
Aus Production down
FR and BE DNS issues
UK IS ON FIRE
Provisioning
UK Zapier issue
Zapier meltdown
French Prod Downtime!
Katuma Staging is broken
Katuma Server Upgrade
Staging servers broken…?
Staging issues
…and there’s various small bits of tracked time with no description. I think some of these bits related to Zapier were billed separately to UK.
I wonder if we could have something like a jobs board (or channel), where instance managers can post small jobs they need doing and devs could potentially pick them up but invoice the instance separately? Eg: setting up a Zapier integration, which might take less than an hour.
Didn’t we talk about this before? (Maybe I’m imagining that.) I think I was in a discussion like this before - and we liked the idea of the ‘jobs board’ but wanted to do it in a way that we ensured the dev was keeping their OFN Core hours stable, and that the job board hours were in addition. Job board tasks cannot siphon off hours from the pipe. I like the idea a lot if we think we can manage this.
Instances that feel under-served by sysadmin skills want lots of things included in the global sysadmin pool.
From people involved in the tech of this a resistance and concern that including all instances in this will create an endless stream of requests
There must be a middle ground.
I feel like we are moving toward at least 3 levels of instance tech support:
Instances that do not at all rely on global sysadmin skills
Instances that rely on global sysadmin for provisioning and deployment
Instances that rely on global sysadmin for all their sysadmin needs
I think 3 levels is important because I would be terrified if we found ourselves managing a world of sysadmin for a new instance that has 1 enterprise and other than signing the pledge no evidence of commitment to the global commons. Equally for instances that clearly are committing extensively the fact that they fall over due to a lack of basic sysadmin support.
Proposal
How’s about we define 3 levels of instance tech support?
No Global Support: You provision, deploy, manage all of your own tech. Probably we don’t recommend this unless people want to white label or are associates
Basic Global Support: We’ll provision your OFN server and manage the deployment of new releases. You need to manage your DNS, email server and other tech support type roles.
Full Global Support: Resolving operational problems, major upgrades, tech support eg with DNS issues.
This proposal raises a tonne of questions…
How do you get to be Basic Global Support and Full Global Support?
Is there some way in which new small instances get Basic support once they demonstrate some basic capacity to run an instance?
Is there a progression based on funds/commitment/turnover/something else that elevates an instance to Full Support?
I feel like there are some interesting things we can do with this as part of an instance on-boarding programme and I think it could be really interesting. Interested in thoughts
So we have three levels. Then to apply for either Basic or Full Global support, an instance makes a proposal according to things outlined above here i.e. they either pay their own way or demonstrate strong contribution to the Commons in other ways
Yeah, I think this is a good direction @Kirsten.
So expanding on the proposals so far…
Whitelabel and associate instances can start just by downloading the code and getting going. Affiliates can too but we tend not to recommend it (because they end up taking up so much time in Slack with questions)
After signing the Pledge and filling in some kind of survey+canvas+?? that demonstrates their commitment and plan - and paying for a server - an affiliate instance can join the Basic Support and have provisioning and deployments as part of the global pool. We don’t fix critical issues at this stage. But they have a server so that they can try and drum up usage and funds.
To shift to Full Support the instance needs to demonstrate that they are committed to the commons, through fundraising efforts, contributions to any commons resources and evidence that they are growing and able to cover the immediate costs (eg servers, tools etc) as they do.
How do we implement and document such a plan?
Well, I guess first we need to decide:
What is included in Basic Support?
Access to all our resources
Responses on Slack
Inclusion in circles
Option to (but we won’t wait for) contribute to community feedback on features and processes
Provision and deployment of servers
Other things that didn’t just pop into my head…
What is included in Full Support?
Everything in Basic Support plus
Sysadmin help with issues such as DNS, emails, tech support
Inclusion in analytics
Responses to critical issues such as S0s (server down), server/OS/DB upgrades
Submitting issues, papercuts and S3s
Active inclusion in community processes including feedback on features and processes. We will wait for your feedback (though not indefinitely… )
Other things that didn’t just pop into my head…
How do people apply for Basic and Full Support?
Basic:
Fill in an online form that asks for evidence of networks, team, business model canvas.
Complete some kind of ‘How to set up an Instance’ training?
Sign the pledge as an Associate
Outline their plan for contributing to the commons - eg fundraising plan etc. This means we can review efforts against the plan
Full:
A review every 6-12 months of progress in terms of users, turnover, fundraising, contributions.
Maybe the Instance-Managers circle can be the circle that decides this?
I clearly went a little overboard with the proposals here. Hope you enjoyed the ride.
I know that @jen has put thought into this in the past so I suspect you’ll see obvious things I have missed. As will many others.
I have to admit, I’d love to see this
This is shaping up - I also love a plan (although realize they are always more nuanced that they appear as written - damn it!)
And just to clarify - some of the OFN-CAN issues, and issues that @Rachel I think has raised also - re: a dev/sys admin who can work with an instance on additional things that require access to the database, or server access are not really addressed in the above.
Thoughts on @Matt-Yorkley’s suggestion of a ‘job board’ for these things ?
I particularly like the idea that an instance managers circle might exist - and might be a home for building bi-lateral kinds of relationships across instances generally - not some kind of ‘approval’ body. I think @Jen was wondering if our current global monthy zoom might be this. I had a different thought on that - but after yesterday’s global meeting, I think I now concur. Its a good forum for bi-lateral relationship building.
@tschumilas There is a difference between prioritising tasks and actually identifying who can ask for tasks to be done.
By all means, let’s create a process for prioritising sysadmin tasks aka a jobs board.
I hear your fears about the sysadmin work you feel cannot be done for CA. Please just read ‘Full Support’ as ‘All the sysadmin things that fully supported instances need’. That is literally what I mean by
Sysadmin help with issues such as DNS, emails, tech support
The FR instance would happily apply to the Full Support so is aligned with the general idea and willing to pay for it.
Now with the OFN global hat
I wonder if we need to start analyzing how we can cap the work needed on the full support so it does not impact too much our pipe.
Should we cap the number of hours an instance can ask for, in a given amount of time? Or say e.g. an instance can only ask for XS/S and M sysadmin tasks…per month/quarter…? It’s a bit annoying because it would mean to start doing estimates, but I feel we need to be careful with the full offer.
That being said we can start trying without any scope to see what kind of load we are dealing with. I don’t have a strong opinion on this, I just see that on the two other project I have aside, people can get carried away in asking things every other day
my mind went here too @Rachel . And there is a risk - that something we ‘make do’ without now - will quickly overwhelm resources if we defined it as something a fully supported instance had a ‘right’ to get. (I don’t like how that language sounds - but I think you know what I mean.) I wouldn’t want to be irresponsible in making requests - but could be, if I didn’t understand fully the parameters.
For sure there might be too much work, but let’s try out something in which FR, CA and US will not have to feel nervous about asking for sysadmin support. We can create priorities and assume that s3s probably won’t be done for a while just like with other issues. We can create a sysadmin pipe and priority list and connect things up and etc. I’m sure we can find a good process.
In all of OFN there is too much work for the people that can do it. It doesn’t stop us trying things out
What are the next steps?
Agree on the three levels of support. Perhaps we need to take a proposal to the Global Hangout? How do we gain agreement for this? I suppose the proposal will need clarity over rights and responsibilities within each level.
Develop the process for the sysadmin pipe (aka task board above). This is a decision for the dev and devops circle.
I like the shape this took since I checked last! I’m up for trying this out.
Re the sysadmin pipe, the only precedent we have for it is the sysadmin column but it’s only related to ofn-install related issues so I guess we’ll need to set a clear boundary between these and instance-specific sysadmin tasks.
I’m hoping that the huge efforts on sysadmin standardization we did in the past poy off here and not many instance-specific things pop up. But for that, we’ll also need to be clear that we only support hosting on Ubuntu machines like the core instances. There might other variables we might need to consider enforcing though.
Lastly, would this include things like managing instance-specific services such as Discourse, local websites, and similar? I think it’ll be safer to start with product-related things only: the app, Zapiers, Data replication, etc. but that wouldn’t fit your needs @Rachel. Perhaps a monthly quota of those instance things is what you meant?
I consider that product-related as it’s necessary for the app to work. To me, this and a local discourse or a local website are slightly different things. While all the devs in the core team are familiar with the former, they might not with the latter. But, well, I want to know what’s your opinion on this.
Instance hat: yes this could be a good first step.
OFN global hat: I would be in favor of mentioning this on the local website template in super admin guide. Currently OFN global gives out templates for wordpress. So with providers that offer one-click solution to install wordpress and OFN global that offers a theme, any non-tech person can start a local website.
But maintaining it… So it be good to write somewhere that we don’t offer support on this.
Hi, I really likes this idea, as @Rachel said is not easy to find a person to help on sysadmin task, and is one of our big challenges to growth.
It would be helpful to our funding efforts to have a clear idea of the full support cost to include it in our projects.
I’m totally agree with all the propose requirements.
Thanks
Hi again everyone,
Breathing life back into this thread.
I’d like to move forward some of the discussions above into something more concrete. I’ve created this GDoc to describe the proposals. Note it is currently a GDoc so that we can collaborate on it. Please comment and suggest things
Updates to the Instance Manager Guide to describe the process
A new process for the Instance Manager circle to review and decide support levels for instances
Clearer Rights and Responsibilities for Instances towards the commons.
Please just request access if you don’t have it via the doc.
In particular this impacts #instance-managers circle as it creates some responsibilities for you.
It impacts sysadmins, which is the dev circle.
It also impacts business and governance circles, though I’m not sure I can place this circle yet and kinda just made it up.
And of course global gardeners - @NickWeir@Jen
Sounds great. I’m not sure if everyone wants the global team to admin their server but I guess that people can opt out if they want to. It’s great to include people to start with because it makes everything easier.
Hi all,
I’d like to register OFN Ireland’s interest in getting Basic or Full support help at some point if possible. We are currently managing our own instance.
I’m up for trying this out.
So it be good to write somewhere that we don’t offer support on this.