In my previous posting here I talked about CAVCON and what it means. Following some noteworthy points that were raised in the CAVCON Meter thread on the MystOnline.com forums, I thought I’d use this posting to discuss a very closely related subject, that of “donations.”
Donations is a topic that can become a touchy one: That’s understandable because we’re talking about asking people to part with hard earned money, so it’s worth making sure that people know what it’s all about.
In the sections lower down, I’ll go into some detail of why MO:ULa needs donations and what they’re used for. You only need to read those if you want all the gory background; the next section is the “short form” for those that want a quicker read.
Donations are a “gift”
There has been some comment, even complaint, that there is a constant demand that “fans must donate.” Now, I’m not sure that I’ve ever personally seen anyone being quite as insistent as that, although I believe there is a fairly consistent level of activity around the cavern and forums to remind people about donating to MO:ULa. But we need to remember that by definition a “donation” is a gift, freely given. Later, I will explain why donations are essential to MO:ULa, but the principle must be that no-one feels coerced into donating, nor should they ever need to explain why they don’t wish to donate.
Personally, I think that things like “CAVCON Awareness Parties” or someone going round the city calling “Hey, it’s one week ’til the end of the month and CAVCON is only at 2.5 – Please consider donating if haven’t already done so” aren’t a bad thing. It’s quite easy in the hustle a bustle of daily life to forget about donating if there’s nothing to draw your attention to it. Even the CAVCON message on the MO:ULa login screen becomes a bit “invisible” after you’ve been seeing it for a few months.
So, reminders are good. Badgering people is not so good. Not everyone is able to make a donation, even if they wanted to, so you shouldn’t ever put someone in the embarassing position of needing to explain why they don’t donate. For similar reasons I get a bit uncomfortable when people talk about exactly how much or how often they donate – it places needless expectations on others. Donations are voluntary and people should only be donating what they can afford, when they can afford it.
A few months back, Marten made a very good observation to me, that is worth repeating here. Big donation drives can actually turn out to be counter productive, as you can tap your resource so heavily that you drain it dry. Someone could organise an event with the aim of reaching the elusive “CAVCON 5″ status, and it might even be successful. But at what cost? Once people donate their hearts and souls into reaching that target isn’t there a real danger that they’ll simply be unable or unwilling to donate again for some time? Possibly long enough to cause CAVCON some real harm.
I think it’s a real concern and one that I hadn’t really considered before Marten mentioned it. So strong encouragement to donate makes sense when there’s a need to recover from a month or two of sitting at CAVCON 2. Otherwise, gentle reminders to make sure that we keep just above the CAVCON 3 break even point.
Now, the boring detail bits:
The need for donations
If MO:ULa is being offered as “free” why are donations needed and what are the donations used for? Well, I partly covered this in the CAVCON article, but it’s worth expanding on this. Firstly, what we have to recognise is that running a service like MO:ULa has unavoidable operating costs that have to be met somehow, and Cyan has chosen to ask for voluntary donations to meet these costs.
But if there are these costs, can it really be considered “free?” Well, it is maybe worth noting that what Cyan is doing here is far from unique: A lot of “free” software relies on donation based funding to keep the development project alive. Many of the Linux distros operate that way – if you download Ubuntu Linux, for example, you’ll be presented with a page asking you to donate a sum based on the “value” (in $) you perceive for various features. But that’s maybe a slightly geeky example, so let’s use an example that’s more mainstream.
There can hardly be a reader of this post who hasn’t used Wikipedia at some point, and probably some people use it on a near daily basis. It describes itself as “the free encyclopedia that anyone can edit.” So how does a website like that with over 4 million articles and 18 million registered users manage to operate “for free?” The answer is “donations” and typically you’ll see a banner on Wikipedia for a few weeks each year carrying an appeal for donations. MO:ULa is in a very similar position, just on a much smaller scale.
How donations are used
We can’t be 100% sure of what the cost of running MO:ULa is; Cyan will not release specifics of how the costs are made up, and I’d guess that one reason for this that it’d probably expose something of its internal costs and that could be advantageous to a competitor. Cyan may not be running MO:ULa as a business, but it still needs to be competitive in the areas it is trading in.
However we can make some pretty safe, educated guesses at what sort of things the donation money gets used for.
This is probably the most obvious expenditure there is. Although the server code is “closed source” we’ve been able to deduce from various little snippets of information quite a bit about the server setup for MO:ULa: We know that MO:ULa uses Amazon’s EC2 (Elastic Cloud) servers and run Microsoft Windows Server as the Operating System rather than Linux as typically used by shards. For the database, which forms the basis of the “vault”, MO:ULa uses Oracle, a commercial product. In contrast, MOSS, the open source server used on the Minkata shard uses PostgreSQL. We don’t know exactly what drove those choices as it’s different from the server setup that was used for the original Uru Live, but it seems reasonable to assume it may simply have been the platform that was available for the GameTap service. Since MO:ULa was essentially just a re-incarnation of the state of MO:UL at the point it closed on GameTap it has inherited everything from that era.
Even armed with that knowledge there are several possible configurations of the EC2 servers, so we can still only guess at exactly what Cyan is using, but we can see that there is a charge per hour plus a charge for the amount of data passing in or out of the server. Since the number of hours in a month is known, that cost ought to be predictable. But the data charge is dependant on the amount and type of usage and that’s not very predictable: A new user downloading all the game files will probably use more data for that than they will in several days of normal gameplay.
Incidentally, Cyan has stated that it plans to move from “on-demand” server instances to “reserved” instances. What this means is that instead of paying for the servers on what is essentially a “pay-as-you-go” scheme, Cyan will instead commit to leasing the servers on an annual contract, with a reduction in cost of maybe 30-40%. The down-side of this, if there is one, is that Cyan is committing to paying for a year (there’s an up-front annual charge in exchange for substantially reduced hourly rates) and gambling that there will be sufficient ongoing donations through the year to meet the costs.
Maintenance and Support
All servers require maintenance and support. The good thing about out-sourcing servers through a service like Amazon’s is that you don’t have to worry about hardware failures as Amazon takes care of all that (and very efficiently too). But, beyond the provided operating system, the software you run on the servers is your problem. The MO:ULa server is really a collection of distinct elements and many people will be aware of terms like “game server” or “auth server” even if they don’t fully understand what those mean. These are things that Cyan is responsible for keeping in good order.
I guess a lot of folks are only too aware of the KI oddities that occur in MO:ULa – they’re the prime reason (or at least a symptom of the prime reason) for the server re-starts that occur every three weeks or so. And occasionally, one of the servers will crash for no apparent reason, possibly the most noticeable case is when either the auth server or backend server fails and users can’t log in, so the relevant server needs to be manually re-started. Those things need Cyan staff to get involved and their time needs to be paid for out of donations.
Now, no-one is going to reveal how much a Cyan employee gets paid, not even as an average – that’d be too much of an intrusion. But we are able to estimate how much time is spent on a re-start, partly because we can “see” how long the logins are restricted; typically it’s about an hour and a half and longer if there’s an update involved.
Why does a re-start take so long – surely it can be done in a few minutes? For
the Build 912 update on May 29 this year, Christian Walther and I were invited to assist the support team with the update verification. And this is probably the key thing: After the re-start, the support team, Tor’i, Dexter and ResEng M.Dogherra, run through a series of checks “in-game” to ensure that all the server elements are operating and that everything is working as expected before allowing the public back in. Some of these are things that some players might not even be particularly aware of such as the “bad word” filter on KI chat but it all takes time. I expect it’s all part of Cyan’s “quality control” that they’ve used for Magiquest Online and during GameTap MO:UL, so it’s second nature to work that way. Once you throw in some of Chogon’s time youre looking at maybe 5 man-hours of effort for a re-start.
By the way, during that 912 update Christian and I concentrated on checking that the new features had made it in OK while the Cyan folks did their routine checks: By parallelling those tasks we kept the downtime to a minimum but it was still getting close to three hours. So updates will certainly cost more than a regular re-start.