A Day Late

We’ve been having some trouble with our MySQL server running on the HPUX machine. It’s been crashing multiple times for the last few days. When it crashes it becomes completely unresponsive. No clients can connect and any clients that were connected don’t respond. I tried keeping a few windows open to monitor the processlist and extended-status but they didn’t help a whole lot when the server crashed (all values seemed normal when the crashes occurred). I was even unable to shutdown the server. I tried running the script to stop the server but the server wouldn’t respond to any kill commands. Basically the only thing I could do was kill the process with a “kill -9″ (NEVER A GOOD THING).

I’ve been watching the logs and I haven’t found anything out of the ordinary. Nothing significant in the error log. Nothing in the binary log and nothing in the query log file. I assumed it was one particular query that was killing the server but the last queries in the log files never matched. I checked the system log files and couldn’t find anything suspicious except for one thing. The server was restarted a few days ago for unknown reasons. I put in a ticket to engineering inquiring why the server had been restarted (assuming they’d made some changes). I got a response back indicating the server had crashed. I suspected the database might have some corruption problems due to the kill -9’s but it wasn’t until I learned the server had crashed that I thought corruptions could be the cause of the problem.

I decided to take the server down and run a myisamchk. It found quite a few errors. I ran it again with the –recover option. Just to be safe I then ran a CHECK TABLE [table name] EXTENDED on all the tables (MyISAM and Innodb) to verify they all had an “OK” status. They did. Hopefully this solved the problem. We’ll have to see how things go tomorrow.

When I reconfigured my server using Debain I wanted the same harddrive raid setup I had with Redhat. Basically a completely redundant file system that would boot off either drive in the event of a failure. I found a number of howto’s (this one, and this one). I followed both instructions but still could not get the system to boot. I was 95% sure it was a problem with my initrd.img file since I was getting cramfs errors and a kernel panic with “kill init”. So, for the benefit of those who have the same problem, here’s what I did.

(more…)

If you’re in the root folder of your filesystem, don’t type grep -r “some stuff to search for” *. Why? Because you’ll get a message like “grep: memory exhausted” and your system won’t respond (at least it won’t respond to ssh, http, smtp, etc). Actually, it appears it just crashes the network port. Executing a simple ifdown eth0 followed by an ifup eth0 from the console appeared to solve the problem. Then again, if you’re like me and your server is housed 20 minutes away and this happens at midnight it’s a little inconvenient to go access the server.

Why does this happen? Well, apparently grep keeps appending to its buffer until it comes across a newline character ‘\n’, at which point it clears the buffer and starts reading from the next line. This is a problem when it tries to search a large binary file (like something in /dev or /proc). I should have known better and excluded /dev and /proc to begin with, or even better run the command inside a subfolder such as /etc (which is probably all I needed to search). I’ll remember that next time. Still, after this, I’m nervous about executing a grep command. What if it comes across some large binary file I didn’t anticipate. I’m thinking of writing a simple shell script and add it to cron to run every fifteen minutes to cycle the network interface if it can’t ping the gateway. I think that should take care of it, but that solution seems a little hackish. Anyway, I’m open to suggestions for a better one.

I finally got around to making some upgrades and changes I’ve been planning for a while. Since I reached the end of life cycle for Redhat 9.0 and I wasn’t interested in moving to Fedora, I decided this was a good time to move to Debian. Initially I planned to bring up a temporary system while I rebuilt the main server but I decided I might as well just get it over with. So, after I backed up the server I wiped it clean and started from scratch.

The installation was a little tricky since I was using a multiprocessor system with a RAID 1 setup (software). Getting both processors working was simple. I used apt-get to install the SMP kernel and that was it. The RAID was quite a bit more involved and troublesome but I’ll save that “Howto” for another time.

For the most part I’ve built the system using the debian packages. I wanted to use Exim as my SMTP server but I also wanted it to use MySQL to store the configurations for virtual domains and email usernames. Since that wasn’t possible with the Debian Exim package I built it from source by following these instructions. I also built Qpopper from source using the same instructions. The instructions are ok but seemed a little incomplete at parts. Basically I just looked at the Exim and Qpopper configuration files (included at the end of the instructions), commented out the Amavis sections and that was about it. It’s working perfectly.

I also made some changes to my Url’s. I moved my weblog from http://www.daylate.com/blog/ to http://www.daylate.com/. So basically that URL is for my blog. I also have another URL, http://www.jwholmes.com where I moved my picture gallery and such. The best thing is, all the old links work, thanks to mod_rewrite.

So, if you’ve noticed the site has been down a little bit in the past week, this is why. Now that I’ve worked through all the kernel issues it should be up and running for a long time (I hope, knock on wood). Debian really is cool, by the way.

The holiday season is nearly half over, I can hardly believe it. Christmas was good and its been nice to have a break from work and things. Although by Christmas day I was starting to get a little bored. I finally decided to upgrade the server to Redhat 9.0 since its end of life cycle is good until the end of April. This gives me a little more time to decide what I want to do with the server and of course actually doing it.

The upgrade was straight forward, didn’t run into any snags, but it didn’t take very long either. I decided I would eventually like to migrate the system to Debian, and since I haven’t worked with it at all I’ve spent a few days playing around with it. Overall I’m impressed. It’s a solid distribution and has some very nice features (apt-get and dselect to name a few). I’ve gone through the install process a couple of times just to get accustomed to it. I’ve been doing the testing on an old HP desktop machine that was sitting around. I named the system “kenny” since my other production system is named “cartman.” One might wonder why I chose Kenny and not Stan or Kyle…well I figure this is an old slow machine and eventually I’ll take it down, or kill it. Those familiar with Southpark will understand why the name is appropriate for a temporary system.

Anyway, I hope to have all my services up and running on kenny that I have on cartman. Then I plan to take cartman down and let kenny handle all the services for the domains while I rebuild cartman with debian. There’s a couple of things I want to do with the installation. I want to use exim as my smtp with qpopper as the pop server. I’d also like to have both use MySQL for authentication and mail handling. I’ve read a little about this and it seems like an excellent way to handle mail for virtual domains. Unfortunately the Debian packages don’t appear to have mysql versions for either, so I’ll have to compile them from source. I found a little bit of information on this and it doesn’t appear to be too difficult.

As I was thinking about the mail setup I was a little curious how ISP’s handle the authentication setup. Obviously they have to handle virtual domains but they also seem to use one authentication system (username and password for email and ftp, etc is the same). If anyone has tips on how to do this I’d be interested.

I also need to research using software RAID on Debain (I’m planning to use the “Woody” distribution, so I’ll probably install the bf24 flavor since the 2.4+ kernel supports RAID). Setting RAID up on Redhat was cake but I’m not as confident with Debian.

The mail setup is really the only major difference I plan to have between the current setup and the Debian setup. At least that I can think of. I’m sure when it comes time to do it I’ll think of a few more. Oh, one thing I need to find out, is how to export a list of debian packages installed on a system and use that list to build another system. I’m sure there’s a fairly simple way to do it I just haven’t come across it. That should make the upgrade go fairly smooth. If I can setup the kenny system very close to how I want cartman setup I should be able to export the package list on kenny and use the list on cartman to have it install all the packages I need. The only thing left is migrating the configuration settings and data.

I’m not in any rush to get the upgrade done. I think I’ll use the rest of the holiday break to catch up on some weblog entries I’ve been meaning to write. I’ve read a couple books over the last few months that I’ve been meaning to write about and there’s a couple other things I’ve been meaning to mention. With that said, there should be a few new entries coming over the next week.

P.S. Remember those glasses I lost over the Thanksgiving holiday? Yeah, I found them two weeks ago.

When I got my powerbook I switched from using Internet Explorer to Safari solely because Safari had tabs and IE didn’t. I was happy with Safari but then heard rumors it suffers from a memory leak. I decided to continue using Safari, once again, solely because of the tabs. At about this time I got my new x86 laptop at work with Windows XP which means I was back to IE (at work anyway). Honestly it drove me nuts opening a new window for every page I wanted to view. Not too mention when you open 3 or 4 windows of the same program in XP they scrunch together as one on the task bar, making it cumbersome to navigate between them.

A guy I work with recommended Mozilla and when I realized it had tabs I was sold. I installed it on both my Powerbook and Dell D600 and have never gone back or felt the need to try anything else. The only thing that’s been annoying me is when I click on mailto links, it tries to open the Mozilla mail program (composer). I’ve searched the “Preferences” window repeatedly for a setting to “Use the system default mail program”. Tonight I finally decided to search Mozilla’s site, figuring at least a few other people have found this annoying. Sure enough I found the solution. Now my web browsing world is perfect.

Mozilla also comes with a built-in Javascript Debugger, very nice for a web developer. Also, since I’ve started doing full time web development I’ve gained a greater loathing for IE. The behaviour amazes and disgust me at times.

For example, today I was tweaking a page for a new site I was working on. Basically I copied a template from another of our sites and proceeded to change the graphics, menus, etc. On the top of the page there was a logo graphic 90 pixels tall, to the right was another graphic that was 70 pixels tall and another below it 20 pixels tall (all with borders=”0″). It looked fine in Mozilla, but when I checked it in IE I was stunned when I loaded the page and noticed the logo graphic had a gap on the top and bottom. I checked the other two graphics closely and they appeared to stretch from top to bottom with no gaps. I double and triple checked the graphic dimensions and img tags with no luck. I loaded the site I took the code from and everything appeared fine. I switched the graphics with those from the site that was working and the gap was still there. The code was almost exactly the same except for one thing. The site that was working used javascript for the top right image (70 pixel one) to automatically select a random picture when the page was loaded. The code looked something like this:

<script language=”JavaScript” type=”text/javascript”>

var random = Math.round(Math.random()*(4-1))+1;
document.write(’<img src=”/images/top_images/image’ + random + ‘.jpg” width=”469″ height=”70″ border=”0″/>’);

</script>

However on the new site I only had a single graphic so there was no need to use the javascript I just replaced it with:

<img src=”/images/top_image.jpg” width=”469″ height=”70″ border=”0″/>

No problem right? Not with IE. Since that was the only code difference I could see, I replaced the img tag with the following javascript:

<script language=”JavaScript” type=”text/javascript”>

document.write(’<img src=”/images/top_image.jpg” width=”469″ height=”70″ border=”0″/>’);

</script>

Reloaded the page, and suddenly the gap disappears, all is well with IE again. Strange and utterly annoying. How much time did I waste trying to figure out what was going on? TOO MUCH!!

My server mysteriously rebooted a few days ago. I’ve checked the logs and I can’t figure out what caused it. It’s possible someone pressed the power button and put the system in a shutdown mode, I did have some nieces and nephews over that night. Still…the system wasn’t that accessible, the power button had tape blocking it which didn’t appear to be disturbed. Plus if the power button was pressed why did the system start back up. I dunno…I’m confused. A few days before the power went off for about 45 minutes and the UPS kept the server and DSL running fine. Then the next day my uptime gets put all the way back to 0 from this, oh well.

I’m finally rid of the kdebase package I couldn’t seem to get off my server. At first I thought the kernel panic was possibly caused by the multiprocessor kernel. From the searching I did on kernel panics, a lot seem to deal with the smp kernel. I booted into the kernel without smp support and tried to remove the package again, still ended up with a kernel panic. Next I wondered if it could be caused by some file system corruption so I booted to a prompt and ran fsck which uncovered a bunch of errors, one of which dealt with a kde file. I initiated a repair, rebooted and tried the remove again only this time everything seemed to work.

Oh yeah, I also upgraded the kernel. Now I just need to get the UPS system hooked up and I’ll be in good shape.