This is a question for any KI employee reading this.
Every week or so you do a scheduled server maintenance where you shut down all servers for 3 hours. You do this in the early am, eastern time when only night owls and people in other time zones are playing.
However, for some of us this is the best time to play and your maintenance method seems to leave us with unnecessary downtime.
My question is why dont you take half or most of the servers down at first, but leave some servers up and the game running. At that time in the am, none of the servers are full and very few servers would handle the wizard101 load.
When the new code is installed or maintenance performed on the servers you have taken down, you can have a 5 minute shutdown while you return those servers to service and take the other servers off line to install with new code or perform maintenance functions.
This doesnt apply to emergency shutdowns if there are major game issues. However, the method of general maintenance seems totally unnecessary.
I am very sorry to hear that our server maintenance disrupts your game play. I also appreciate the suggestion that you made.
Unfortunately, it is not as simple as it seems. Our maintenance involves multiple steps and includes a variety of game configuration components not just individual servers: patch and gameplay servers, database and operation monitoring tools, etc.
Usually during maintenance we are changing shared components and it needs to be done at once. We cannot, for example, change a database structure and have half of the game servers shutdown and the other half online. The half online will not know about the new changes and could lead to data loss. If that happens we are going to have much longer down times and painful data restoration. Neither you nor us want that to happen.
So, as unpleasant as it is, there are no other easy ways to perform a maintenance other than turn off the services completely. To better mitigate the down time we picked just one day in a week and the time when there are least players playing. I know this is not what you wanted to hear, but I hope you would understand our situation better now.
That only partly explains the inability. If you look at my post, I understand that you would need to shut the game down at some point. I was thinking of data base structure when I thought of that.
I can understand that it maybe complicated by shared components although not sure how extensive that is. However, data loss seems irrelevant to the discssion. Even if you save all data to one shared server, there would be no chance of corruption if you simply continue running the game under the old code while you do maintenance on non shared components. When you have performed whatever maintenance that you can perform, you then force a shutdown saving character data at that point. At that point the whole game is offline but at least some portion of the maintenance has been done and the downtime shorter. There is no reason, you couldnt do all the non-shared components, then take the game down for any shared components and any database conversions you needed. The servers that were running while you performed maintenance on non-shared components could then be maintained off line while the game is again running on the new code. Data loss doesnt come into the picture.
Again, this is a suggestion to minimize downtime. I am sure that it would require some thought on how to do it. I have both played and maintained online fantasy games in the past and havent run into the amount of downtime maintenance I have seen in your game before.
I have both played and maintained online fantasy games in the past and havent run into the amount of downtime maintenance I have seen in your game before.
Hmm, I'm interested in what successful titles those were. Every MMO I've every played has had weekly maintainence schedules.
I was a coder and wizard on one of the first and biggest text based Muds that was a direct ancestor of this game called SneezyMUd. I also was a player at Sojourn. Coincidently, I was a friend of and played there with Brad McQuaid who went on to produce Everquest. I played there for a few years although I was just a player and not involved in maintenance.
Once again you have slightly misread my post. I have not said in any form you do not need maintenance. Regular maintenance is of course required. I have said in 2 posts, that at some point the game has to go offline. It was not the "weekly" I asked about, it was the length of the periodic maintenance.
I am sure that this game is more complex than the game I maintained mostly in the number of players and the use of graphics. I do not underestimate the challenges of saving millions of players.
However, much of the game is the same and the overall tasks are the same. Our code needed to keep status of players, world, and items and save changes at all times. We had about as many attributes in our characters as in your character screen and our items had all sorts of data including the ability to be changes. We took it offline to change code and once and awhile we had to change the data structures. However the procedure was pretty simple. Kill the server, convert data structures, copy the new code over then reboot.
I had posed this as a question. KI clearly runs a test site that has independant status. It runs on new code and has independant data saving. Changes to it do not cause problems with the main game. I also see time when servers are added to the game. Thus, my question of whether you could take only half the servers off line to add non-shared code. The remaining servers could continue to run while the coders changed the servers they took down. Then a downtime to save/convert player data. Then bring the game back up with the maintained servers and new data structures while the coders maintain the other half of the servers that are now offline, adding them when done.
It may even be that KI does some of this already. There are some one hour shutdowns as opposed to three. I just get and got the feeling that KI runs its maintenance to be more convenient to coders rather than to reduce downtime. They just take the whole game down and put it all back up when everything is done rather than just everything needed for the players to continue playing. Your response about shared systems and data loss seems a little pat and reinforces that. I am again asking if there is a way to reduce downtime by first getting the game going with a reduced number of servers and then adding more as they are upgraded.
Coincidently, I was a friend of and played there with Brad McQuaid who went on to produce Everquest.
Wow, neat!!! Before his time with Verant or while he was with MicroGenesis?
My experience with Everquest was their routine maintainence was approx. two hours versus Wizard101's three hours. (Not being sarcastic....) How long are you expecting the downtime to last? Those other companies also change up the times periodically (as to not affect the same people continously). Would that make a difference?
Also, Wizard101 servers are not fixed like Everquest. In those games, which ever server you create your character on, they will stay until an employee manual moves them. In Wizard101, characters flow freely between servers. I assume that has an impact on all servers coming down at the same time. Since a player that logged off on one server yesterday can end up on another the next day, possibly due to overpopulation (and that has happened to me before).
I was a coder and administrator on Sneezy but preferred not to play on the game I was coding. When I wanted to play, I played on Sojourn where Brad was a groupmate of mine. This was before Everquest went into production. I still remember sitting in the main city chatting with him a few hours when he told me he had just landed a big budget contract with Sony to produce a graphic online game. I had no idea how big it would be. I played EQ for a few years starting with the .5 beta. I only stopped because life didnt permit the huge mandatory time sink that EQ had become. I never went to WOW for same reason. Wizard101 is a time sink but it is more voluntary :).
I dont remember the exact duration of EQ downtimes but my memory is that it was somewhat shorter than Wizard101. Moreover, who knows if they did it right as an administrative matter.
My question for KI was, admittedly, not based on knowledge of the KI systems. However, logically, what I am suggesting might work to reduce downtime. It is clearly more convenient for a coder/administer to just take down a whole system and put back up a whole system regardless of impact or whether parts of it can run. However, if the goal is to reduce downtime, it might be possible to find better upgrade procedures. They may add some work to the people doing upgrades but would be better for customers.
Last to your direct point, my memory is that EQ also saved the server you played on. Regardless, if you did partial upgrades to reduce downtime, it might require server changes for some characters logging in during the upgrade process but that is not a huge deal. Their character data has to be secure since in all cases it will be saved under the old code before he plays on a new server running new code.
That only partly explains the inability. If you look at my post, I understand that you would need to shut the game down at some point. I was thinking of data base structure when I thought of that.
I can understand that it maybe complicated by shared components although not sure how extensive that is. However, data loss seems irrelevant to the discssion. Even if you save all data to one shared server, there would be no chance of corruption if you simply continue running the game under the old code while you do maintenance on non shared components.
The lead technician gives you an answer, and it is not good enough for you? Just because you "worked on or with" other MMOs does not mean that they are all structured the same way. Do you know exactly what is involved with KI performing maintenence?
Probably not. I am sure that if they could reduce the amount of downtime, they would. After all, more downtime=less revenue per unit time.
My friends play a different game. Their game does maintenance most every Tuesday for 7 hours. I know we have been chatting in the morning and they are waiting for the game to come back online. I remember one time it was down most of the day. When I say morning I don't mean at 3 A.M.. It is prime time morning 9 A.M. MST.
No matter when maintenance takes place someone will be inconvenienced. I also think KI does it the best way for their system. Megan
Not sure why I am bothering to respond to the last posts but will make it quick.
I have no idea if the "lead developer" responded to my initial post but neither do you. He maybe a coder, manager, or just someone who is tasked with responding to posts. Or he may actually be a developer. Neither of us has any idea. The other game mentioned sounds like it is really badly coded. I think Wizard101 is extremely well made and I enjoy playing it.
With regard to the game which I helped code. It had every piece that is in the current game. We had a dedicated client. We had upgrades and downtimes. We had a very sophisticated set of saved character traits ( more in fact than are visible to players of this game), clases, races, accounts, etc etc. All of the current games are not that different in structure and content than the game I worked on although the current games are more sophisticated. I am sure there maybe different code approachs and system approachs but those are not all that significat.
The most salient points of Bartleby's response was something about continuing to save data under older code could create data loss and the statement that there is no "easy" (emphasis added) way to save time.
I think that is exactly what is going on. It is easier to just take a game down regardless if you could design a less easy (on the coders) method of performing an upgrade that would reduce downtime but cause more work for the developers. Using database conversion as the excuse reinforces my belief that the current method is not the quickest, just the easiest. Database conversion whether data is saved on game servers or centralized can be converted during the full game shutdown not while the developers take down and maintain non-shared equipment. It might even require extra steps at that point to seperate shared systems from non-shared (i.e. temporary systems that store data change untill right before the final conversion).
Again, I like and respect this game. It should not be offensive to ask developers to examine if their maintenance routine can be shortened even if not as "easy" as the current method.
