A rough couple hours

[Mike]

First off, apologies to anyone who was on the site, around 10:15 PM EST, this evening.

We went away and it wasn't pretty.

What with the additional traffic we've been seeing, I decided it was time to get busy with the server, trying to milk some more performance out of it. Most of you won't understand, but I upgraded PHP from version 5.2.17 to 5.3.24. My next plan was to upgrade MySQL, so I could then start doing some logging and tweaking on both MySQL and Apache, to better optimize the site.

But after the PHP upgrade, the site was down and I could not discover what was wrong. I was scouring the local error_log and the Apache error_log, but there were no errors being logged. The site was borked, big style, but not as far as the server was concerned.

A tech in the datacenter and I have been chewing at this for over 2 1/2 hours. I've had a grand total of 3 hours sleep in the last 33 hours and my brain is melting away.

Finally, we noticed an installed opcode caching script was removed with the PHP upgrade. After reinstalling it, still no hope. So, we decided to go back to square one and recompile PHP 5.2.17, to see what would shake out.

And we were still having problems, after the roll back. Finally, I removed the caching rules from the forum's configuration file and bada-bing! We're back.

Once I got in here, I checked the XenForo error logs and have found thousands (yes, thousands) of error messages in those logs. In addition to the opcode cache system, I also run a distributed memory object cachiong system, on the forums. And all of these errors are showing that systems extensions are not being loaded. I just ran through the logs and there were 2104 errors generated.

I just ran ps aux | grep memcached and it looks like it is being called, but XenForo says it is not. So we are verifying memcached is still with us. And then, like a couple of caffeine-crazed coders, we are going to make another stab at upgrading PHP and MySQL. Could someone brew another pot of coffee, this looks like it is going to be a l-o-n-g night. He says as he turns up the volume on the Grand Funk Live album...

OK, buckle your seat belts, because here we go. See you on the other side, if we make it that far.

 

[Mike]

PHP has been upgraded and the opcode cache is running on it. So far, so good.

Keeping my fingers crossed...

 

[Mike]

And, five sleepy hours later, MISSION ACCOMPLISHED!!

The opcode cache is running, memcached is running, PHP is upgraded and page load times are faster than ever before.

After I get some sleep, then it is on to upgrading MySQL and sticking a tuning logger on it for a couple days, so we can tune it up. Then we'll see just how fast this old place can really run.

Say g'nite, Gracie.

 

[Mike]

OK, after a refreshing two hours sleep, a trip to the grocery and a load of laundry in the washing machine, it is back to the desk and tweaking some more settings.

(Note to self - Never define thread_cache_size more than once in /etc/my.cnf as it will most certainly prevent MySQL from restarting. )

A script is doing some background logging on the database server, so I can sort out the bottlenecks and tweak around them. I got some rough, start-up configuration suggestions on the first run-through and have applied them. In another 48 - 72 hours, it will be time for another look, to see what else can be tuned up. Some MySQL DBA experience would sure be helpful, but I'll soldier through it. I'm learning a bit as I go, so it's not all bad.

And at the end of this process, all of the effort should result in a leaner and meaner server configuration, that will help keep server costs down for the foreseeable future.

 

[Tfitch03]

Thanks for your hard work Mike!

 

[Duke]

Huh Yeah, What he said !

 

[Mike]

It's just a matter of letting the page visits add up, so the opcode caching will be more effective. We were typically seeing hits in the cache up in the 97% range, and as of right now, hits are at 15.7%. A page is not cached until it is served the first time, so it takes time. But memory fragmentation is down at 00.00%, so I am smilin'. And in another 45 hours or so, I'll have some real data to help me tune up the database server.

And you think dialing in a new carburetor is difficult?

 

[2old2fast]

My complete lack of knowledge of all things computer leaves me not knowing if I'm being "dazzled" or "baffled"..

dave

 

[Mike]

It's real easy to get lost in all this. I often see forum admins asking other admins for copies of their database configuration files, but that shows they don't have a clue as to what they are doing, because the configurations need to be tuned to the specific server, the amount of RAM it has available, the size if the database and the average traffic the site sees. What works on a server with 16 GB of RAM is not going to work on a server with 256 MB. And the buffer pool size on a 2 GB database is completely different than the pool size on a 200 MB database. So you can't just start changing things, without knowing what you are changing.

And make no mistake, I am not a MySQL database guru. Which means I have been learning a lot, in the last 36 hours, or so. So I don't end up bedazzled or bamboozled, myself.

But that's OK, because the docs now say the brain needs to be exercised on a regular basis, so I like to keep learning. Actually, I wouldn't mind trying to get my own Linux admin certification, one of these days. I'm at the age most guys are looking to retire and I still want to get better at things I enjoy doing. It's a little late in life to go to work in a datacenter, but I sure would like to know I am qualified to do the job.

 

[Keeper]

What tuning script are you using? I have been meaning to tweak our MySQL server at work but I keep putting it off for fear of what just happened to you! I would be lynched if I lost 8 years worth of client files!

 

[Mike]

I am using mysqltuner.pl

tuning_primer.sh is also out there, but I don't think it is quite as popular.