State of S2KI Update
03-23-2011, 09:33 AM
#1
Administrator
Thread Starter
State of S2KI Update
Dear members, sponsors and friends of S2KI,
As you no doubt know we recently undertook the task of "upgrading" the software we use to power S2KI.com from the heavily modified, massively optimized but aging and increasing unmaintainable software we started using in 2003 to a more modern, off-the-shelf, current version of the software available today. There are a large number of compelling reasons to make this change and after giving the new software a thorough functional review I decided to begin the process back in November 2010.
In the four months leading up to the switch-over I ran the upgrade process dozens of times, refined it, modified it, massaged it and tested it as best as could be done with the hardware that we had available for job. We developed an extensive checklist and except for a problem with private messages being imported from an old backup it went fairly well in terms of meeting the goals. It was intact after the upgrade and it was completed within the 3 day window I allotted to get it done.
I think it is important to stress many of the reasons behind the change. There are some who say "if it ain't broke, don't fix it". What they fail to realize is that the old software was in fact broken in many respects they may not be completely aware of.
- Memberships couldn't renew and were sometimes lost
- search was so bad we needed two search features plus google just to get by
- Photo gallery couldn't delete albums or do much of anything beyond simple upload and post
- PMs were poor copy of 1970's email, no status other than read/unread, no search
- no classifieds
- no seller reputations/feedback
- very few moderator tools, awkward to manage, impossible to control spam and spammers
- no profiles, forcing everything into signatures
- etc.
- these problems couldn't be fixed without breaking something else
- no viable way to add anything to it or modernize the site with changes in how people use the Internet made since 2003 (eons in Internet time).
The software was unmaintainable, stuck in time with few options for adding value for members and maybe most importantly we had no viable way of fixing the problems that we knew existed with the resources we have to work with. There is no team of developers, system admins, support staff or any of that. There is HondaGal taking care of the members, part-time, ludedude taking care of the sponsors, part-time, and me doing pretty much everything else full-time(ish).
The only realistic way for us to continue to improve the site, fix the problems and offer more value going forward was to put ourselves in a place where we can take advantage of the plugins, hooks and add-ons available in the new software and to attempt to stay on track so we could take advantage of newer features as they came out over the long term.
Unfortunately using new software meant tossing all of the hand-carved precision tuned code we had worked hard over 8 years to polish. It wasn't always a smooth process getting there but over the past year or two it got fairly slick at what it did while consuming very little resources. The old s2ki was a lean, mean machine more than anyone probably realized. It could phone-in 10 million pages a month and take the rest of the day off. I miss that and from many comments I've seen I'm not the only one.
There were also a few little big things we had put into the old site that would be missing from the new software like club badges and the MyS2K landing page for regulars who could see what was going on with the conversations they were participating in at a glance. It was my browser home page so I miss that too as much as anyone. Most of these can be redone using the API and other tools provided with the software without compromising our ability to stay on the upgrade path and receive fixes and new features as they come out.
However, all things considered, the benefits of moving to the new software, not just now, today, but long term, next month, next year and beyond far outweigh the immediate pain and suffering and inconvenience imposed by making the change to the new software. Even now, after 3 weeks of high-stress, high-frustration hell and abuse I still feel that to be true.
I realize that people are suffering and the old site is sorely missed right about now as we go into the 4th week of laggy performance, meh design and many on-again-off-agin broken features like the gallery and some missing sent PMs. I am aware of these issues and I will address all of them as soon as possible. First I want to deal with the 800lb gorilla: speed.
The site is slow. Even when it is fast, it is slow. A page should show signs of loading within half a second of clicking a link or pushing a button and right now we are running at 4 to 8 times that long with a lag of 2 to 4 seconds for every single page, even worse in some cases. There are a number of reasons for this and I am working on fixing it from a range of angles:
- Increased features = more code to run per page = more RAM and more CPU and, most importantly, time needed to respond to each page request. Time is the essential element here. There are a limited number of "slots" available for handling your requests for pages. The faster those requests can be processed the sooner those "slots" can be reused to handle other requests. If requests start taking say 10% longer to process then the turnover rate falls astronomically and creates a traffic jam just like reducing the lanes from say 3 to 2 will do on a freeway when it gets busy. The slots available can very quickly become zero and lead to a pileup. Each slot takes CPU power and memory so the number of slots available is a function of that and it's not something that can be immediately fixed as it requires additional hardware to increase the capacity for slots to handle requests.
- Poorly optimized code runs slower and consumes more resources. More resources per slot = fewer available slotsThe highly optimized old s2ki software used very little resources and was tuned to execute very quickly. The new software isn't optimized this way. Add this to the increase features problem and it makes it much worse. Honestly, it is something of a pig which works well if you have lower traffic and lots of hardware to run it. To resolve this lack of optimization and make the software less expensive and faster to function I have all of the developers who wrote it looking at it, monitoring and profiling it and working to fix the inefficiencies within the core product so that future upgrades will include much of what was done. That is an ongoing process and they are logged in every day actively working on it but there is no way to judge how much they are going to accomplish.
The only way I can see to quickly resolve many of the performance issues is to put more hardware to work on it. To that end I have ordered and expect to receive this week a new 12 core (2CPUx6cores) web server designed to supplement the 1CPUx4cores server we currently have, a 3 fold increase in available resources to process requests. I addition, today I am working on testing other web server software (we use Apache 2.2 + mod_php + APC), which is designed to be way more efficient in how it uses resources which should make more slots available for handling page requests. Taken together, new web server software + more hardware, I fairly confident we can get this speed and performance issue tamed and under control. If it becomes necessary to add additional servers to the pool to get the "lag" time down to under 600ms then we can and will do that. Since I don't know the impact adding one server will have to performance I want to evaluate that first before deciding how many are needed to make the site as fast as we want and expect it to be.
I vastly under estimated the impact the new software would have on our hardware and OS configuration. I was spoiled by the old software and figured that even if the new stuff was half as fast we wouldn't have any problems. I did not realize the extent of our optimization effort of the years until you get to compare it head to head. That is my mistake and I'm working on fixing it as quickly as possible by getting people in to optimize the software (as much as they can), by tuning the servers for a higher yield we didn't need before and putting as much additional hardware in the rack as is necessary to make it all go at the speed we all expect from it.
Aside from the performance issue, which is pretty all-consuming, I want to touch on a few other issues people have "mentioned" which are important but have been overshadowed by the performance concerns; speed affects everyone but not all other issues do.
MyS2K - MyS2K was a "hack" we made to the old software's User Control Panel (UCP) to serve as a landing page for regular users and offer a summary of which forums they frequented (favorites list if you like) and the active and recent topics they have participated in so as to make it easy to watch and continue to participate in on-going discussions. It was a good feature. The developers of the core software took a different approach more along the lines of social networks with a "profile". It is different but we can't get on the bus and still end up in a different place than everyone else so your new "Profile" will replace MyS2K going forward.
Gallery - The gallery is a bit of a mess right now. There are a number of problems. I need to rebuild the gallery images and albums which is going to take a huge amount of time and I simply can't with resources already way over stretched. Once I can get some more resources onboard I can begin to rebuild the gallery images and albums. Until then it can be a little dicey. This should be underway as soon as tuesday but may take a bit longer depending on how the performance work goes. Another issue we have is that the new gallery software dumped everyone's albums into a single master album named "Member Albums". That's causing a lot of problems I haven't figured out how to resolve yet. I'd like to say a fix is imminent but unfortunately the other issues force me to defer a full scale solution in favor of just getting the basics working which I think they sort of do now.
Sent PMs - Some people have reported that their previously "sent" private messages are no longer accessible. I haven't done a detailed study but at this point the only solution to that issue is to bundle sent PMs from the backup archives and send or somehow make them available as archives for people to download or have delivered via email.
Design and layout - The design of the site, the "skin", is a temporary design adapted from a boilerplate. It is not designed to be a long-term design for the site but only a temporary stopgap until there was time available to customize. The issues we've had with performance have set this item on the back burner until I can get a handle on the other things.
I realize this change has been disruptive. I expected it to be but perhaps not anywhere near as much as it has been. I knew there would be some people who hated it and there would be some who loved it and unfortunately I'm stuck somewhere in the middle of those right now. I'm sorry this has been painful. I am working very hard to make it right and I will do so as soon as I can.
To those who hate the new site because the performance sucks I say that help is on the way in the form of additional hardware which has something of a lead-time and by my commitment to getting the performance to at least above average levels.
To those who hate the new site design I say it will change soon rather than later. Not today, not next week, but within weeks the design will once again be pure s2ki.
To those who hate the site because they just hate it I say sorry, I can't help how you feel and either you'll get used to it or you wont. Either way I have to continue to do what makes sense down the road in the long term and doing nothing was a course with no future. We can agree to disagree but is about all. There is no going back. I/we are going to need to tough out the last mile of this marathon, or not, but I need to finish this and it doesn't involve quitting and starting over on March 1st like the last 23 days never happened. This isn't the Tonight Show.
To those who like the changes or are at least interested to see what comes of them I say stay tuned because once these immediate distractions and annoyances are past there are going to get major enhancements to profiles, classified ads, article/library content to name just a few. Obviously the current situation clouds that but this will pass and I will be able to start work on the good stuff very soon.
Thanks for your patience. I know it has been tough and I know it continues to be so but I can assure you progress is being made and it will noticeably improve fairly soon. I also apologize for this very long post. There was a lot to cover and I can be wordy. Thanks for reading.
- cthree
As you no doubt know we recently undertook the task of "upgrading" the software we use to power S2KI.com from the heavily modified, massively optimized but aging and increasing unmaintainable software we started using in 2003 to a more modern, off-the-shelf, current version of the software available today. There are a large number of compelling reasons to make this change and after giving the new software a thorough functional review I decided to begin the process back in November 2010.
In the four months leading up to the switch-over I ran the upgrade process dozens of times, refined it, modified it, massaged it and tested it as best as could be done with the hardware that we had available for job. We developed an extensive checklist and except for a problem with private messages being imported from an old backup it went fairly well in terms of meeting the goals. It was intact after the upgrade and it was completed within the 3 day window I allotted to get it done.
I think it is important to stress many of the reasons behind the change. There are some who say "if it ain't broke, don't fix it". What they fail to realize is that the old software was in fact broken in many respects they may not be completely aware of.
- Memberships couldn't renew and were sometimes lost
- search was so bad we needed two search features plus google just to get by
- Photo gallery couldn't delete albums or do much of anything beyond simple upload and post
- PMs were poor copy of 1970's email, no status other than read/unread, no search
- no classifieds
- no seller reputations/feedback
- very few moderator tools, awkward to manage, impossible to control spam and spammers
- no profiles, forcing everything into signatures
- etc.
- these problems couldn't be fixed without breaking something else
- no viable way to add anything to it or modernize the site with changes in how people use the Internet made since 2003 (eons in Internet time).
The software was unmaintainable, stuck in time with few options for adding value for members and maybe most importantly we had no viable way of fixing the problems that we knew existed with the resources we have to work with. There is no team of developers, system admins, support staff or any of that. There is HondaGal taking care of the members, part-time, ludedude taking care of the sponsors, part-time, and me doing pretty much everything else full-time(ish).
The only realistic way for us to continue to improve the site, fix the problems and offer more value going forward was to put ourselves in a place where we can take advantage of the plugins, hooks and add-ons available in the new software and to attempt to stay on track so we could take advantage of newer features as they came out over the long term.
Unfortunately using new software meant tossing all of the hand-carved precision tuned code we had worked hard over 8 years to polish. It wasn't always a smooth process getting there but over the past year or two it got fairly slick at what it did while consuming very little resources. The old s2ki was a lean, mean machine more than anyone probably realized. It could phone-in 10 million pages a month and take the rest of the day off. I miss that and from many comments I've seen I'm not the only one.
There were also a few little big things we had put into the old site that would be missing from the new software like club badges and the MyS2K landing page for regulars who could see what was going on with the conversations they were participating in at a glance. It was my browser home page so I miss that too as much as anyone. Most of these can be redone using the API and other tools provided with the software without compromising our ability to stay on the upgrade path and receive fixes and new features as they come out.
However, all things considered, the benefits of moving to the new software, not just now, today, but long term, next month, next year and beyond far outweigh the immediate pain and suffering and inconvenience imposed by making the change to the new software. Even now, after 3 weeks of high-stress, high-frustration hell and abuse I still feel that to be true.
I realize that people are suffering and the old site is sorely missed right about now as we go into the 4th week of laggy performance, meh design and many on-again-off-agin broken features like the gallery and some missing sent PMs. I am aware of these issues and I will address all of them as soon as possible. First I want to deal with the 800lb gorilla: speed.
The site is slow. Even when it is fast, it is slow. A page should show signs of loading within half a second of clicking a link or pushing a button and right now we are running at 4 to 8 times that long with a lag of 2 to 4 seconds for every single page, even worse in some cases. There are a number of reasons for this and I am working on fixing it from a range of angles:
- Increased features = more code to run per page = more RAM and more CPU and, most importantly, time needed to respond to each page request. Time is the essential element here. There are a limited number of "slots" available for handling your requests for pages. The faster those requests can be processed the sooner those "slots" can be reused to handle other requests. If requests start taking say 10% longer to process then the turnover rate falls astronomically and creates a traffic jam just like reducing the lanes from say 3 to 2 will do on a freeway when it gets busy. The slots available can very quickly become zero and lead to a pileup. Each slot takes CPU power and memory so the number of slots available is a function of that and it's not something that can be immediately fixed as it requires additional hardware to increase the capacity for slots to handle requests.
- Poorly optimized code runs slower and consumes more resources. More resources per slot = fewer available slotsThe highly optimized old s2ki software used very little resources and was tuned to execute very quickly. The new software isn't optimized this way. Add this to the increase features problem and it makes it much worse. Honestly, it is something of a pig which works well if you have lower traffic and lots of hardware to run it. To resolve this lack of optimization and make the software less expensive and faster to function I have all of the developers who wrote it looking at it, monitoring and profiling it and working to fix the inefficiencies within the core product so that future upgrades will include much of what was done. That is an ongoing process and they are logged in every day actively working on it but there is no way to judge how much they are going to accomplish.
The only way I can see to quickly resolve many of the performance issues is to put more hardware to work on it. To that end I have ordered and expect to receive this week a new 12 core (2CPUx6cores) web server designed to supplement the 1CPUx4cores server we currently have, a 3 fold increase in available resources to process requests. I addition, today I am working on testing other web server software (we use Apache 2.2 + mod_php + APC), which is designed to be way more efficient in how it uses resources which should make more slots available for handling page requests. Taken together, new web server software + more hardware, I fairly confident we can get this speed and performance issue tamed and under control. If it becomes necessary to add additional servers to the pool to get the "lag" time down to under 600ms then we can and will do that. Since I don't know the impact adding one server will have to performance I want to evaluate that first before deciding how many are needed to make the site as fast as we want and expect it to be.
I vastly under estimated the impact the new software would have on our hardware and OS configuration. I was spoiled by the old software and figured that even if the new stuff was half as fast we wouldn't have any problems. I did not realize the extent of our optimization effort of the years until you get to compare it head to head. That is my mistake and I'm working on fixing it as quickly as possible by getting people in to optimize the software (as much as they can), by tuning the servers for a higher yield we didn't need before and putting as much additional hardware in the rack as is necessary to make it all go at the speed we all expect from it.
Aside from the performance issue, which is pretty all-consuming, I want to touch on a few other issues people have "mentioned" which are important but have been overshadowed by the performance concerns; speed affects everyone but not all other issues do.
MyS2K - MyS2K was a "hack" we made to the old software's User Control Panel (UCP) to serve as a landing page for regular users and offer a summary of which forums they frequented (favorites list if you like) and the active and recent topics they have participated in so as to make it easy to watch and continue to participate in on-going discussions. It was a good feature. The developers of the core software took a different approach more along the lines of social networks with a "profile". It is different but we can't get on the bus and still end up in a different place than everyone else so your new "Profile" will replace MyS2K going forward.
Gallery - The gallery is a bit of a mess right now. There are a number of problems. I need to rebuild the gallery images and albums which is going to take a huge amount of time and I simply can't with resources already way over stretched. Once I can get some more resources onboard I can begin to rebuild the gallery images and albums. Until then it can be a little dicey. This should be underway as soon as tuesday but may take a bit longer depending on how the performance work goes. Another issue we have is that the new gallery software dumped everyone's albums into a single master album named "Member Albums". That's causing a lot of problems I haven't figured out how to resolve yet. I'd like to say a fix is imminent but unfortunately the other issues force me to defer a full scale solution in favor of just getting the basics working which I think they sort of do now.
Sent PMs - Some people have reported that their previously "sent" private messages are no longer accessible. I haven't done a detailed study but at this point the only solution to that issue is to bundle sent PMs from the backup archives and send or somehow make them available as archives for people to download or have delivered via email.
Design and layout - The design of the site, the "skin", is a temporary design adapted from a boilerplate. It is not designed to be a long-term design for the site but only a temporary stopgap until there was time available to customize. The issues we've had with performance have set this item on the back burner until I can get a handle on the other things.
I realize this change has been disruptive. I expected it to be but perhaps not anywhere near as much as it has been. I knew there would be some people who hated it and there would be some who loved it and unfortunately I'm stuck somewhere in the middle of those right now. I'm sorry this has been painful. I am working very hard to make it right and I will do so as soon as I can.
To those who hate the new site because the performance sucks I say that help is on the way in the form of additional hardware which has something of a lead-time and by my commitment to getting the performance to at least above average levels.
To those who hate the new site design I say it will change soon rather than later. Not today, not next week, but within weeks the design will once again be pure s2ki.
To those who hate the site because they just hate it I say sorry, I can't help how you feel and either you'll get used to it or you wont. Either way I have to continue to do what makes sense down the road in the long term and doing nothing was a course with no future. We can agree to disagree but is about all. There is no going back. I/we are going to need to tough out the last mile of this marathon, or not, but I need to finish this and it doesn't involve quitting and starting over on March 1st like the last 23 days never happened. This isn't the Tonight Show.
To those who like the changes or are at least interested to see what comes of them I say stay tuned because once these immediate distractions and annoyances are past there are going to get major enhancements to profiles, classified ads, article/library content to name just a few. Obviously the current situation clouds that but this will pass and I will be able to start work on the good stuff very soon.
Thanks for your patience. I know it has been tough and I know it continues to be so but I can assure you progress is being made and it will noticeably improve fairly soon. I also apologize for this very long post. There was a lot to cover and I can be wordy. Thanks for reading.
- cthree
03-23-2011, 10:00 AM
#2
Community Organizer
Am I the only one who has read this?
The first to reply?
The first to reply?
03-23-2011, 10:13 AM
#3
Guess so...now I have. Thanks for the update!
03-23-2011, 10:50 AM
#5
Administrator
Thanks for the update cthree. There are people here who can and will help, willingly, for free
Having been through previous upgrades with you and S2KI, I am confident you will do all is required to get it sorted, and running like a mean machine again.
PS: I think this should be an announcement, not a topic
03-23-2011, 11:12 AM
#6
Nice long informative post. In my opinion, we (Erik) spoiled everyone with the old lean machine site. Maybe in hindsight, the site should have been a standard software package with no "fun" things, but it wasn't. The owners have acknowledged many requests from the participants of s2ki.com over the years and that is how the community was developed. Life changes and this is a change that appears to have affected many people, including all who work on the site and volunteer their precious time to help out. When I see comments and complaints I realize we have some great friends, moderators, site mods, CO's that truly care. S2ki has affected many many people over the last 10 years and this has been an eye opener for all of us. Family and friends are super important, just like a marriage, in sickness and in health and I would like to hope that our friends are willing to help everyone through this change.
It is important to fight through some crucial issues at this time and when things settle down we will look at ways to ensure your friendship continues to be valued. Erik is only one person, and the one who needs to fix the problems, so any ideas or tasks that need to take place for the membership will need to have some time cycles to figure out what is best for everyone.
Trending Topics
03-23-2011, 01:01 PM
#9
Registered User
Join Date: Oct 2009
Posts: 2,220
Likes: 0
Received 0 Likes
on
0 Posts
Newer, more modern forum software that is slower and hogs more resources while still not delivering what the old, out-dated version did. Another example of what passes for "progress" in modern times 
03-23-2011, 01:16 PM
#10
Administrator
Nice long informative post. In my opinion, we (Erik) spoiled everyone with the old lean machine site. Maybe in hindsight, the site should have been a standard software package with no "fun" things, but it wasn't. The owners have acknowledged many requests from the participants of s2ki.com over the years and that is how the community was developed. Life changes and this is a change that appears to have affected many people, including all who work on the site and volunteer their precious time to help out. When I see comments and complaints I realize we have some great friends, moderators, site mods, CO's that truly care. S2ki has affected many many people over the last 10 years and this has been an eye opener for all of us. Family and friends are super important, just like a marriage, in sickness and in health and I would like to hope that our friends are willing to help everyone through this change.
It is important to fight through some crucial issues at this time and when things settle down we will look at ways to ensure your friendship continues to be valued. Erik is only one person, and the one who needs to fix the problems, so any ideas or tasks that need to take place for the membership will need to have some time cycles to figure out what is best for everyone.
