Overview
This is a technical overview of the subscriptions feature. The intention is to describe the architecture in detail, and to provide an understanding of how subscriptions integrate with the rest of the OFN platform. This is not a user guide.
Background
Discussion of a subscriptions feature (also called “Standing Orders” at one point) for the OFN started at least as far back as February 2015. The core concept of a subscription seemed to be fairly universal, in that it was just an order that was to be repeated at a particular frequency for a particular customer. We needed to flesh out the requirements by understanding the kinds of business models that the feature would be used to support, so we asked the community for ideas as to what components a subscriptions feature should have in an ideal world. We quickly realised that there were several variations on the kind of business models that would be supported this feature and that what we had been thinking of as a single feature was perhaps actually better described as several related features.
I attempted to write up a full list of user stories from the feedback that we got from the community, and that was treated as the full spec for what would be considered a complete feature. From there started pulling together the major components that would be required to pull it off.
Major components
The major components that we identified would be required were as follows:
The Order
The details of the order to be placed needed to be recorded somewhere. Which products were to be included in the order? Which customer was the order for? Which shop was the order to be placed with? What shipping and payment methods were to be used? We toyed with the idea of just using an actual Spree::Order as a source, but the main reason we decided against this was that it would have meant that an order needed to be placed before a subscription could be set up. In hindsight that doesn’t seem link such a big deal, but I remember thinking it seemed icky at the time.
The Timing
We needed a way of modelling the timing information for a subscription. How often should the order be placed? When should the subscription start, and when should it end (if ever)? Was the subscription currently on hold (for eg. while the customer is away)?
Interface with Order Cycles
Order Cycles already existed at the time, and so we identified a requirement for subscriptions to work with order cycles to achieve the timing/repeating component. We needed allow the timing component above to understand which order cycles the subscription was applicable to. Order Cycles have no understanding of their relationship to one another, and so the simplest way that we could think of to enable the repeating function was to create named sets of Order Cycles that we ultimately ended up calling Schedules.
Payment
The consultation process made it clear that we needed to have a universal solution to the problem of automatically charging customers for orders. Implementing different payment methods for different countries was not considered a viable option, at least for Version 1. We were tossing up between Mangopay, PayPal and Stripe, and ultimately landed on Stripe due to it’s wide availability, and platform oriented ‘Stripe Connect’ product that fit our requirements. Thus the Stripe integration project was born. This kind of exploded into a fairly large project in itself, but I won’t go into it in any more detail is necessary to explain subscriptions.
Another important thing to add is that we were originally thinking about the idea of having a built in customer credit system for the OFN, but decided to go down the route of a credit card gateway as a first step because for most business models this is going to be a prerequisite for a functional customer account system anyway.
Admin interface
There are several parts to this. Enterprise users needed to be able to manage subscriptions for their customers. This includes tasks like creating, altering, pausing, resuming or cancelling subscriptions. They also need to be able to set up and manage relationships between subscriptions and order cycles (ie. schedules). We also wanted to hide the complexity of the subscription feature from people who weren’t interested in using it, so decided on the need for a toggle that would allow enterprise users to opt-in to seeing the UI elements.
Front-end interface
The full spec articulated a requirement for customers to ultimately be able to completely manage their own subscriptions, in the same way as enterprise users. An additional layer on top of this was the need for customers to alter/cancel individual orders that had been generated via subscriptions up until the close date of the relevant order cycle. An interface was also required for setting up credit cards for use with automatic billing.
Automation
To pull it all together there was a need to automatically run some business logic whenever an order cycle opened or closed. Most of this logic was going through the motions of processing orders or payments without the customer being present, but there were also a few additional considerations about how to best trigger this work.
What to build first?
The decision that we made was to try to build subscriptions in a way that would only allow enterprise users to create and edit them (on behalf of their own customers).
We definitely made a mistake here in attempting to build far too complex a system for the first cut. I put it down to lack of experience, but I now know that I should have worked much harder to find much smaller pieces to build and validate before tackling the complexity of a larger system.
We ended up with this spec for Version 1. At the time, it seemed like a fairly big compromise relative to the total spec, but that was probably the wrong way to look at it. That is more or less what we have ended up building, so I’ll now describe the technical architecture of what we have in greater detail.
Technical Architecture
I’ll start by going over the data model in more detail, and then I’ll dive into the business logic that pulls it all together.
Data Model
The data model for all of this is actually fairly straight forward. In order to meet the needs of the original spec, there are only four components required to model the main subscriptions functionality in addition to what we already have.
Subscription
As mentioned above, the subscription model has two responsibilities, and might actually be modelled better as two separate objects (FavouriteOrder and Subscription?). More discussion of this below under ‘Bloated Subscription Model’.
The first responsibility is to model the order that the customer wants to repeat (FavouriteOrder?), which means recording all of the information that will be required to construct a new order from scratch.
shop_id identifies the enterprise (shop) with which orders for this subscription will be placed. Also used for validation and building interfaces to limit the set of available customers, schedules, shipping and payment methods, and to look up which products are available to add to a subscription.
customer_id identifies the customer for whom orders will be placed. Also used to find the user and default credit card that should be charged if this is relevant.
shipping_method_id the shipping method to use for orders.
payment_method_id the payment method to use for orders. Restricted to Cheque and StripeConnect type payments methods only. PayPal and other payment gateways are not supported by subscriptions at this stage.
ship_address_id the shipping address to use for orders. The admin user interface for creating a subscription will prefill this with the help of SubscriptionCustomerSerializer and AddressFinder if possible.
bill_address_id the billing address to user for orders. The admin user interface for creating a subscription will prefill this with the help of SubscriptionCustomerSerializer and AddressFinder if possible.
subscription_line_items a has_many association that models the products and quantities required to fulfil the order (see below).
proxy_orders a has_many association which contains a set of order placeholders which allow for lazy instantiation of actual orders (see ProxyOrder below).
shipping_fee_estimate used to cache an estimated shipping fee for orders generated from the subscription. The value is calculated by a SubscriptionEstimator. See ‘Fee and Price Estimation’ in the discussion below for more information.
payment_fee_estimate used to cache an estimated payment fee for orders generated from the subscription. The value is calculated by a SubscriptionEstimator. See ‘Fee and Price Estimation’ in the discussion below for more information.
The second responsibility of the Subscription model (at the time of writing) is to describe the timing and state of the repetition component of the subscription. State is required because some users wanted to be able to pause subscriptions and then to resume them later. It is also possible to cancel a subscription, and to alter the day on which the subscription is scheduled to end.
schedule_id: identifies the schedule which allows mapping of the subscription onto a set of order cycles. This is the mechanism by which we can understand which order cycles a subscription applies to.
begins_at: a start date for the subscription. No orders will be placed in order cycles that close before this date.
ends_at: an end date for the subscription. No orders will be placed in order cycles that close after this date.
paused_at: used as a flag to determine whether a subscription is active or not. If the subscription has been paused, no orders will be generated until it is unpaused, regardless of the values of begins_at and ends_at.
‘canceled_at’: when a subscription has been cancelled, no orders will be generated from it, regardless of the state of all other attributes. This also hides the subscription from the index interface.
SubscriptionLineItem
The equivalent of a Spree::LineItem for subscriptions. Used directly to build instances of Spree::LineItem. It holds information about which products are being subscribed to and how many of each should be included in the order. A very simple model.
subscription_id identifies the Subscription to which the line item applies.
variant_id identifies the Spree::Product that the line item refers to.
quantity describes how many of the product the customer would like to order.
price_estimate a cached estimation of the total amount that the customer will ultimately end up paying for the item, including fees. The value is calculated by a SubscriptionEstimator. See ‘Fee and Price Estimation’ in the discussion below for more information.
ProxyOrder
Facilitates lazy initialisation of actual instances Spree::Order. Used as a placeholder, to indicate that the system anticipates that an order will need to be placed for the specified order cycle, but that the order itself has not been created yet. Under normal conditions, an actual Spree::Order will not be instantiated until the relevant OrderCycle opens (see SubscriptionPlacementJob below).
The only exception to this norm is when a customer wants to make a change to a future order before the order cycle opens (ie. for a future order cycle). In this instance, the subscription is used to instantiate an order that the shop manager can edit (note: at this stage customers cannot edit their own future orders). When the order cycle opens, the already instantiated (and edited) order is kept, rather than a new one being created.
subscription_id identifies the Subscription from which the proxy order was generated.
order_cycle_id identifies the OrderCycle to which the actual order will apply, note that there is a db-level uniqueness constraint on the combination of order_cycle_id and subscription_id so that one one proxy order represents the order for a given order cycle - subscription combination.
order_id identifies the actual order that has been placed for the customer (if it exists), can be nil.
canceled_at: allows a future order to be cancelled before it is initialised. If this field is not null, no order will be instantiated for the relevant subscription when an order cycle opens.
Schedule
Provides the interface between the existing OFN platform and the subscriptions feature by allowing a subscription to be mapped onto a named set of order cycles. Each order cycle can belong to any number of schedules, meaning that subscriptions can mapped onto order cycles in any number of ways.
name: the name of the schedule, usually something like ‘Weekly’ or ‘First Saturday of the Month’.
order_cycles a many to many association that uses a join table to identify the set of order cycles that belong to a particular schedule.
subscriptions a has_many association that describes the set of subscriptions that repeat according to a particular schedule.
Business Logic
There are several classes that are pretty much pure business logic since I’ve tried to keep as much business logic out of the ActiveModel classes as possible. There are two main clusters of business logic. This first deals with the rules around creating and updating a subscription, or making changes to associated instances of Schedule or OrderCycle. For the purposes of this documentation I’ve called this the ‘Subscription Updating Cluster’. The second cluster of business logic deals with the process of automatically processing orders and payments when an order cycle open or closes. I’ve called this the ‘Order Processing Cluster’.
Subscription Updating Cluster
SubscriptionForm
This class is an encapsulation of all of the logic required to use submitted form data for the purpose of creating or updating a subscription. It is used by Admin::SubscriptionsController, and is fairly procedural in nature. It uses a bunch of utility classes to perform the bulk of the actual work. These are SubscriptionValidator, SubscriptionEstimator, OrderSyncer, LineItemSyncer and ProxyOrderSyncer.
SubscriptionValidator
The validation logic became so complex that I thought it was best to pull it out of the model and build a class around it. This class basically codifies all of the rules around what constitutes a valid subscription. It is used by each instance of SubscriptionForm to validate the data it receives from the form before it attempts to perform more complex update tasks.
SubscriptionEstimator
Used by SubscriptionForm to generate an estimate of the cost of the products specified by a subscription and of the fees associated with an order. To achieve this the estimator looks at the any currently open order cycle for the subscription, if none are found, it uses the next order cycle, and if no future order cycles exist it uses the most recently closed order cycle. For a more detailed justification for the existence of this component, see ‘Fee and Price Estimation’ below.
OrderSyncer
This class is responsible for applying any changes made to a Subscription to any instances of Spree::Order which exist for future order cycles. Remember, orders are only instantiated before the order cycle opens when a customer wants to make a change to the order ahead of time.
For example of the use case of OrderSyncer: if a change was made to the address that a Subscription was to be delivered to, the OrderSyncer class would be used to find any instances of Spree::Order for that Subscription, and to update the shipping_address on such orders in line with the change to the underlying Subscription.
This logic works in the simplest case, where the value being updated is the same on both the instantiated Spree::Order and on the Subscription. However, it is possible that the situation will arise where the value that is being update on the Subscription has already been changed to something else on the Spree::Order, so there is no way to be sure which value is correct. In this scenario, a notice is provided to the user (with the help of the OrderUpdateIssues utility class), but the update to the subscription is allowed to proceed.
LineItemSyncer
Used by OrderSyncer to apply any product-related changes to any instance of Spree::Order (or rather SpreeLineItem) which already exist for future order cycles. For example, if a subscription originally specified 1 x 1kg of Apples and a user attempteda to change the subscription to instead request 2 x 1kg of Apples, the LineItemSyncer class would be used to update any corresponding Spree::LineItems on any Spree::Orders that had already been initialised for the subscription.
In the same way as occurs for other conflicting changes, the user will be notified if a conflict occurs, but this will not prevent the change to subscription itself from being made.
ProxyOrderSyncer
At the time of writing, the system is set up to take a very active role in maintaining the list of proxy orders pertaining to each subscription. See ‘Active Management of Proxy Orders’ below for a more detailed discussion of why this may not be the best approach.
To facilitate the active maintenance of the set of proxy orders for each subscription, I built the ProxyOrderSyncer class which is capable of making a determination as to whether proxy orders have become orphaned by a particular change or whether new proxy orders need to be created. It also creates or removes proxy orders as required.
For example, when an OrderCycle is added to a Schedule, the ProxyOrderSyncer is run and it will add a new ProxyOrder for the new OrderCycle for each Subscription that applies to the Schedule. In this way, we can be assured that whenever one of the order processing jobs is run (see below), we have a complete list of proxy orders that can be used to determine which subscriptions apply to whichever order cycles have just opened or closed.
Order Processing Cluster
Cron Jobs
These are not really part of the business logic but they are relevant to understanding subsequent parts of the business logic.
We use the whenever gem to install a set of cron jobs onto the server. Two of these are particularly relevant to subscriptions. Each is run every five minutes, and each is designed to set up a new instance of Delayed::Job to be run whenever there is capacity.
SubscriptionPlacementJob
This is the first of two classes that are designed to be run by delayed job every five minutes. The essence of what it does is to determine whether any order cycles have recently opened and if so, to create orders based on any subscriptions that apply to that order cycle. In reality the logic is a little more complicated, but that is basically what is happening.
For each order that is generated as part of this process, the customer will receive a special email that tells them that an order has been automatically generated for them. They will also be notified of any issues that were encountered .
A summary is email is also sent to the owner of each shop that had subscription orders to process once the job has finished running. The summary email documents any issues were encountered while automatically processing orders and summarizes how many order were sucessfully processed and how many had issues. To achieve this reporting function, the job uses the SubscriptionSummarizer and SubscriptionSummary utility classes.
SubscriptionConfirmJob
This is the second of two classes that are designed to be run by delayed job every five minutes. This job instead looks for order cycles that have recently closed (rather that opened), and processes any payment that need to be automatically processed (using Stripe Connect and the SubscriptionPaymentUpdater class). After payment has been processed the customer will receive an email confirming the payment and the order.
Again, any issues are reported to the shop owner with the help of the SubscriptionSummarizer and SubscriptionSummary utility classes.
SubscriptionPaymentUpdater
Used by SubscriptionConfirmJob to ensure that a chargeable credit card is present (if required). If none can be found it adds a processing error to the job, which in turn notifies both the customer and the shop owner.
Interfaces
Enterprise user have the option to enable the subscription feature on an enterprise-by-enterprise basis. This will be an option under Shop Preferences, but at the time of writing the field itself is only visible to Super Admin users, effectively meaning that it is force toggled off for regular users. Once the feature is fully released we can just remove the line that makes this form element only visible to Super Admin.
Admin: Subscriptions
This interface consists of an index page (located under the top level Orders menu item), and a form for updating/altering individual subscriptions. Some basic operations such as adding/removing products, and cancelling future orders can be performed from the index page. Everything is written in AngularJS, using asynchronous requests to save an load data where necessary.
Admin: Schedules/Order Cycles
The UI for creating and editing Schedules and populating them with Order Cycles has been bundled into the existing Order Cycles interface. The elements will only appear when a enterprise had subscriptions enabled so as to avoid confusion for users who are not using this feature. I have some concerns that the UX is not particularly great and is a bit confusion even for users who want to use subscriptions, so I have discussed this further below under ‘Separate Interface for Schedules’.
Frontend: Viewing/Editing ‘Open’ order
From the Account page in the frontend, a customer can see any ‘open’ orders. That is, orders that were automatically generated by a subscription, and for which the order cycle is still open. Depending on the Shopfront preferences of the shop, customers may be able to edit their open orders by clicking on the order number. At the time of writing, it is only possible to alter the quantity of existing line items, to cancel individual items or to cancel the order completely from this interface. It is not currently possible to add new items, and instead a new order must be placed.
There are various other elements that appear throughout the checkout process which remind the customer that they have an automatically generated subscription order that can be reviewed.
Frontend: Account/CreditCard
The main parts of this interface that are relevant are the components that allow the customer to set a default card, and to grant permission to specific enterprises to charge their default card. Once this is done, the default card can be automatically assigned as the source for a Stripe payment by SubscriptionPaymentUpdater, meaning that new payments can be made against the customers account for each new subscription order without their intervention.
Outstanding Issues
Earlier this year (2018), we drew a line under the list of things that were to be considered part of Version 1. Below are a list of concerns or issues that have appeared or increased in prominence on my radar since then, and some suggestions about future directions.
Bloated Subscription Model
My feeling is that the Subscription model has too many responsibilities. As touched on briefly above, I can see a fairly clear case for splitting it into at least to separate objects: FavouriteOrder for holding information about how to build new orders, and Subscription for holding information about how the order repeats (schedule) and its state (paused, cancelled, etc). Under this scenario a Subscription would need to have a favourite_order_id reference.
An alternative would be to remove the FavouriteOrder model altogether, and simply use an actual instance of Spree::Order as the basis for creating other orders. The obvious issue with this is that an order would be need to exist before a subscription could be created, so we would need to come up with a way of dealing with this. It would also be difficult to edit a subscription, and we would probably need to just cancel the old one and create a new one whenever a change needed to be made. There are also some advantages however, as we could probably dispense with a lot of the validation and estimation logic…?
Fee and Price Estimation
The challenged I faced in building some of the admin interfaces, and will likely face in building some of the frontend interfaces too, is that it seems necessary to show order totals and prices for items. This seems obvious but actually getting it to work has been a lot more difficult than it may at first appear. The trouble arises because we only determine the final price that a product is to be sold for via an order cycle, and each order cycle can have different fees applied to different products. The final price of the order is only really determined when the user hits ‘Process Order’ because the shipping method and payment method can have different fees attached too.
Since the subscription model has no real concept of price built in, and I don’t want to replicate all of the complex logic of how to apply fees to products, I have attempted to build a minimalistic mechanism for computing the fees for the next order cycle that a subscription applies to, and then caching these values on the subscription itself. This kind of works, but since the price and fees are not actually finalised until the order is placed, I can’t make it 100% accurate without placing an actual order. I think the short term solution is just to make it more clear in the interface that any prices are an estimate.
Timing of Placement of Orders
At the moment, subscription orders are placed (that is processed automatically to the complete state) as soon as an order cycle opens. We leverage the fact that Spree allows changes to be made to orders after they have been completed to allow subsequent changes by the customer up until the order cycle closes. The main advantage in doing this is that subscription orders are given first access to available stock, and this minimises the risk that products will become unavailable to subscribers during the course of the order cycle.
Since building and playing around with this way of doing things, I have pretty much convinced myself that a better approach might be to NOT confirm the order at the time that the order cycle opens, and to instead extend Spree’s stock allocation system to allow stock to be ‘reserved’ for a particular order without that order being processed. I haven’t given it a great deal of thought more than that, but a simple table ReservedStock table that recorded an order_id, order_cycle_id, variant_id and quantity seems like a good start. In addition to this we would need some business logic that worked with SubscriptionPlacementJob that knew how to decrement the relevant stock on behalf of each of the subscription orders being instantiated. Seems workable?
Frontend UX For Editing An Order
What we have works, and it is ok for a first pass, but I am concerned that the flow and interface are not particularly intuitive. The inability to add new products without creating a new order in the same order cycle is also quite problematic IMHO. I would like to move to a situation where we can allow customers to load their automatically generated subscription orders into the main shopfront (which they are probably more used to) and allow them to make whatever changes they like from there.
I have made a start on a prototype for this feature, but I have not had time to finish it. Since Spree has no concept of a ‘cart’ as distinct from the contents of an order, there is not really any way of allowing alterations to an order that can subsequently be discarded to revert to the original order. I have solved with in the prototype by creating a duplicated order which can be loaded into the cart, and then, instead of progressing through the checkout, the customer can just ‘Save Changes’ to the order, which results in the state of the duplicated order being copied back across to the original order. It probably needs more thought, but it seems promising.
Time Taken To Generate Orders
Based on my limited testing, it seems to take a significant amount of time to run the SubscriptionPlacementJob when there are a lot of subscriptions to process. As far as I can tell, this is mostly just an underlying issue with Spree::Order. There seems to be an awful lot of processing going on in the step to process an order to completion. This is perhaps something that will be improved by the Spree upgrade, as I know a lot of work went into improving the flow for updating an order.
At a minimum it would be good to find a way to get some metrics about how much time this is taking. If it going to take 5 seconds to process each order and we need to process several hundred orders when a particular order cycle opens, I think we may eventually run into problems.
Active Management of Proxy Orders
As touched on above, the current regime whereby proxy orders need to be actively managed in order for the whole system to work was never really designed to work this way. It just kind of fell out while I was building other components. I think the two cases cited above (editing and cancelling ‘future’ subscription orders) necessitate the use of proxy orders, but I think that as it currently stands, there may be a lot of unnecessary complexity created by maintaining a list which can probably be reconstructed lazily as required. I am thinking that we can probably remove all of the ProxyOrderSyncer logic, and instead only create proxy orders in the specific instance where a future order needs to be edited or cancelled.
The main blocker that I can see to this is that things like the SubscriptionPlacementJob currently rely completely on the presence of an up-to-date list of proxy orders to function properly. We would need to find a new way of understanding which subscriptions apply to the current order cycle before we can think about switching to a more passive use of proxy orders.
Separate Interface for Schedules
I am interested in hearing from beta testers about this, but since I incepted and built the initial version of the interface for creating/editing schedules, the whole concept of a Schedule has developed a much closer affinity with the concept of a Subscription that it has with OrderCycle in my mind, despite a schedule being a collection of order cycles. I think that it may be more intuitive to put the schedules interface under a top-level Subscriptions menu item, and remove all references to Schedule from the admin OrderCycle interfaces. The schedules index could be more full-featured, and could allow the enterprise user to see which subscriptions apply to a particular schedule, in addition to the existing functionality of adding and removing order cycles from schedules.