Music-crush: Tim Minchin

I've been watching this fellow's videos on YouTube, and yeah, music-crush.

He's like a vaudevillian pianist performing incredibly intelligent songs showcasing dark humor and morbidity.  I've found it rather enjoyable to play a YouTube Playlist of his music in the background while internetting or fucking around the house.  Some key songs that I enjoy:

If You Really Loved Me -- one of my faves, it's quite sweet and.. terribly twisted
Storm -- a nine-minute beat poem that debunks all new age BS
If I Didn't Have You (I'd Have Someone Else) -- a statistical analysis of love
Taboo -- probably my fave song, it's about a taboo word in modern society
Donnie Darko -- I'll ashamedly admit that I didn't fully understand the movie until I heard Tim's explanation

Tim's got a great stage-presence and in all his videos, he appears to really enjoy performing in front of a live audience, feeding from their laughs.  He's also a brilliant pianist whose instrumentals could stand up on their own, though coupled with his lyrics and voice, it's totally awesome.  If he ever tours the US, I'll be there in a heartbeat.

The 10 Golden Rules for Troubleshooting Linux

I'm doing a brain vomit, and you're the lucky recipient of my geek bile!

1. Man pages exist and should be used.  Seriously, everything's there, from application docs to syscall docs to syntax and formatting of log files.

2. Don't reinvent the wheel.  99% of problems you're experiencing or ever will experience, somebody's already gone through it and figured it out.  Google is your friend.

3. If you don't know what something's doing, or why it's not working, strace it!  Nobody ever uses strace, yet I find it invaluable.  Especially for Apache issues.  Is your site or PHP code or whatever not working?  
  3a. Make sure your timeout is set to something long enough for manual human interaction, and fire up two terminal windows.  
  3b. SSH to server on one and gain root access.
  3c. In the other window, telnet to your server on port 80.  Make a GET request for the page causing you issues, such as GET /page.php HTTP/1.1
  3d. Switch to SSH session.  Do `netstat -plant |grep your.ip.add.ress' and find the ESTABLISHED one with an apache process attached to it.
  3e. Run `strace -p (pid of apache process from above) -vvv -Ff -s 256'
  3f. Back in telnet session, type "Host: domain.com" and hit enter twice.
  3g. Switch back to SSH session and watch the syscalls go!  The answer is held within.  Always.
  (note -- you can also launch Apache in debug mode (`httpd -X'), but this requires taking down the service.  Debug mode sets MaxClients to 1 and it doesn't fork child processes, making it easier to strace (you don't have to switch back and forth to find the pid of the child you're connected to), but it's not feasible on a live server.)

4. Logs exist for a reason.  Read them.

5. Applications crash, servers don't.  If your server crashes, it's either bad hardware or a kernel bug (fairly rare on popular distros).

6. Always make backups.  Always.

7. Always mount NFS mounts with the 'intr' option.  Having to reboot because of a network blip is uncool.  (Humorous aside: Macs mount bonjour-introduced mounts via AFP, which appears to have all the awesome negatives of NFS.  If the mount goes away (other server going down or whatever), Finder will freak the fuck out and your programs will start having bizarre issues.  My Finder was hung, and attempting to restart it failed.  Then iTunes got stuck in a loop.  Then Quicksilver crashed.  The only thing I could do, literally, was run `shutdown -r now' in the Terminal window I had open.  Lesson learned -- unmount share when doing software updates on the other server.)

8. Learn to use `grep', `sed' and `awk'.  Learning to manipulate text is surprisingly important for a text-based interface.

9. Load average does not mean CPU usage.  100% memory usage does not mean you don't have any more available for new applications.  You can run out of inodes before you run out of disk space.

10. TCP wrappers suck.  If you've been hacking at an issue for over 3 hours, look to your TCP wrappers.  /etc/hosts, /etc/hosts.allow and /etc/hosts.deny will hold the answer.


I set up a Shoutcast server on my racker box so I can listen to my music from home when I'm at work, so I don't need to waste gibbies and gibbies on mp3s on my laptop.  An interesting exercise in RPM building (and rebuilding) and hacking makefiles.  You can enjoy the stream at http://www.superhappykittymeow.com/stream.m3u .  Note: this is music that *I* like, so suck it up if it doesn't tickle your fancy.

While setting this up, I figured the easiest way to stream the music would be to use the source on my fileserver, so I wouldn't have to shuffle files around.  I got it working, but my overall connection was suffering from it -- my poor Time Warner connection was being anally raped and I couldn't do anything else while streaming.  I used Time Warner's online chat to upgrade to their 15mbps package (special online upgrade price of only $5 more than what I'm paying now!), and one modem reset later, nice and zippy.  I hate Time Warner with a burning passion, but I gotta say, this upgrade was pretty painless and I'm pleased with the results.

I have exquisite taste in music, FYI. 

Linux tip-of-the-day

For some reason, I always thought that `perror' was solely for MySQL errors, since it comes with MySQL. 


From the man page:

       For most system errors, MySQL displays, in addition to an internal text message, the system error code in one of the following styles:
          message ... (errno: #)
          message ... (Errcode: #)
       You can find out what the error code means by examining the documentation for your system or by using the perror utility.
       perror prints a description for a system error code or for a storage engine (table handler) error code.

Very handy for interpretting strace output.  Also of note -- you can man syscalls like getsockopt and whatnot -- also useful for strace.  In fact, here's a handy chart about the man page sections, which always made me wonder what they are, so I looked them up and found a pretty chart:

Section #Topic
1Commands available to users
2Unix and C system calls
3C library routines for C programs
4Special file names
5File formats and conventions for files used by Unix
7Word processing packages
8System administration commands and procedures


(no subject)

 I've decided to try to record my dreams.  I've found that I have a hard time being creative when I want to be -- ie, no "drawing time.. now!".  When I wake up, however, I tend to have fading memories of crazy dreams, and I reckon that my brain's a lot more creative when I'm not forcing it to be.

I put a notepad and a pen by my bed this morning before I fell asleep.  I have a habit of tossing and turning about twice a night, and tend to wake up very slightly during this period -- I'll still be immersed in whatever dream it may be, but I'll gain a bit of consciousness, such as noting the time on my alarm clock, or realizing a cat had been sleeping on my face and wiping the cathair off my nose.  I'd use these moments of consciousness to record my dreams.

I had mixed results when I woke up this afternoon.  First off, my notepad only had two phrases scrawled on it:

"It is a tale of Intergalactic Rejection"

"so it was."

And so, apparently, it was.  These have no relation to the dream that I woke up remembering, though, which involved myself and my family flying from Russia to the US in a Concorde jet.  It was a rather luxurious jet, and a smooth and quick flight.  Arriving in the US, it seems my passport had disappeared, and I was detained by Customs and held in a small room with a 2-way mirror.  I was interrogated by a lovely young woman who kept showing her cleavage.  She was joined later by another young woman and they got into a fight over my situation.  I woke up before I was released.  The holding room was cold.

I'll keep the notepad by my bed for future sleeps.  Perhaps I'll get enough random strings that I can create a Hockney-esque song of surreality.

(no subject)

 I just did some work on some guy's servers who were named after Warcraft bosses.  That was pretty cool.  He also had a server named "uranus", which I made certain to use wherever I could ("I'm still investigating the issues with uranus." "I've just updated the packages on uranus.").  hahaha

Geek rant

 I deal with a lot of the compromises that we see on the linux side.  These are always simple web app exploits (Joomla, phpBB, random PHP script) that are easily tracked down and cleaned up, though of course, the problem isn't solved until the hole is fixed.  The security hole that was exploited is always the hardest part of the job, but I can almost always find it -- down to the specific line of code -- and make suggestions to the developers to fix it.  

I can't fix it, of course, because I'm terrible at coding.  I'm good at pointing out terrible code, though ;)

However, over the past few months, I've seen a HUGE rise in iframe exploits.  Johnny Q's customer's start complaining to him that their Norton popped up when they visited the site, or they get the Google "This page may harm your computer" page that requires two extra confirmation clicks to visit the site (this is a feature in the latest version of Safari and Firefox, and a number of AV programs use Google's database to prevent site accesses).  

Digging into Johnny Q's site, there's no script exploit, there's no weirdness on the server itself.  It's all in the HTML, and it's all a single added iframe line that redirects to a page off-server that downloads 2304890234 trojans, viruses, malware, etc.

I started researching this when I first saw it.  I couldn't figure out how these iframe lines were added to the code.  Everything checked out, the code just mysteriously changed.  Then I looked at the FTP logs.. oh my.

A user from China, Russia, Romania, or any gaggle of countries had logged in and downloaded every HTML and PHP page that matched index*, default*, home* or main*, one at a time, appended about 100 bytes of code, then re-uploaded the file.  Checking the secure log, there were no password failures.  They knew the password.  

Johnny Q had visited a page that had one of these iframe exploits, and, without his knowledge (or his antivirus's knowledge), downloaded malware.  This malware sniffed his FTP account info and phoned it home to some hacker, who then ran a script to login and append the iframe exploit.  Some other Johnny will visit this site (or 50,000 Johnnys), get infected, and in turn, get their own sites infected.  

The hackers are always one step ahead of the AV companies, and will modify their code (and auto-push updates to all the workstations they've infected) to sidestep attempts to kill the malware.  They keep spreading infectiously, and at an alarming pace -- a few months ago, I would see one server a week with these iframe exploits.  Now I'm seeing about 3 servers a night.

Google's Security Lab is constantly studying this, and wrote up an excellent whitepaper on the subject, All Your iFRAMEs Point to Us.  It's a bit dated, but the methods remain the same.

Every time this happens and I clean it up off a server, I change the passwords and tell the Johnny that he needs to clean his client computer -- and no, antivirus won't fix it.  It will try, but it's never enough.  Reformat and reinstall your Windows OS, run Firefox with the NoScript extension... that will keep you a bit safer.  However, you really won't be fully safe unless you run a non-Windows OS.  The malware attacks you and installs itself without user notification, using privilege escalation exploits in a bevy of applications -- Flash, MSIE, random application A, other application B.  New vulnerabilities in any Windows app are found and exploited daily.  Hint: this doesn't occur on Linux or OSX.

I'd rather deal with web app exploits  and c99 shells, Apache-user ownage, than these iframe exploits any day.  The iframe exploits are easy to find and clean up -- a simple `sed' will do the trick -- the hard part is explaining it to Johnny.  And then explaining it again when his sites are exploited next week.  And again the week after.  

Conclusion: Stop using Windows.  That'll make my job a lot easier :P

(no subject)

 I bought "Space Invaders Extreme" on a whim for $10 for my PSP that I never play.  Ho-lee crap, that's an ADD-ridden arcade masher.  Talk about feverish gameplay.  Highly recommended for quick fun.

I also recently purchased "Chrono Trigger" for my (also) neglected DS.  I'd never finished the game all the way through as a child, as I never had a SNES, so I figured, hey, I've got nothing better to do right now.  I'm currently killing robots in the future trying to find some food for the dirty people that live in the domes.

Yesterday was cooking day.

 I slow-cooked a pig butt in my crock pot; 12 hours later, it falls apart at the touch of a fork and is perfect for kailua pork.

While that was cooking (and making my house smell like pig), I poached a salmon fillet and steamed some asparagus.  Both of these actions can be done easily in the microwave without adversely affecting the flavor.  Unfortunately, I haven't yet figured out how to make hollandaise sauce in the microwave, so I had to dirty a pot for that.  

I'm very happy I discovered the Horizon Organic milkboxes -- there's a little over 1 cup of milk per juicebox, and because they're individually packaged, they have an extremely long shelf life.  I've got a 6-pack in the fridge for whenever I want a bowl of cereal or need a cup of milk in a recipe.  Saves a trip to the grocer's!

Kikkoman makes a tofu miso soup packet that is absolutely delicious.  Miso is an acquired taste, I reckon, but if you've discovered that you like it, I highly recommend picking up some of this powdered soup.  Surprisingly filling for powdered fish paste, soy, dehydrated seaweed and tofu.  Beats the hell out of ramen, IMO (which feels blasphemous to say).