Andrew Boring

I enjoy reading post-apocalyptic stories. Some of my favorites over the years have been Folk of The Fringe by Orson Scott Card, Alas, Babylon by Pat Frank, The Postman by David Brin (and not the Kevin Costner movie by the same name), Nightfall by Isaac Asimov, as well as young adult novels I read in grammar school like Children of the Dust and The Tripods.

After the recent dispensational premillenialist rapture predictions, the economic collapse that has wiped the American middle class of their wealth, and a number of natural disasters that demonstrate how precarious our society has become, it's high time to consider what happens when our society can no longer sustain itself.

This is one such thought experiment. This is a list of career opportunities you can pursue when the shopping mall is closed for good, the supermarket stocks no longer stocks food, and you no longer have to worry about casual Friday.

1. Warriors

With the absence of commerce and technology, we'll no longer be an "information-based" society. If things get really bad, we'll probably lose no manufacturing, too. The displacement of manufacturing and information-based jobs will leave a lot of people unemployed and listless. There will be no food at the grocery store and no one would have money to purchase it anyway. Even in our "civilized" nation, many people know only one rule: satisfy their desires at all costs. Without the threat of police, justice, and other trappings of civilization that keeps us honest, many will turn to marauding, looting, stealing, and banditry, either solo or in groups. This will be a strong-against-weak world, so anyone who can provide protection to their local communities of survivors will be in high-demand. This will be a status opportunity for men (and women too), and these high-status warriors will enjoy extra food, companionship and other benefits that a community can offer. The strongest warriors will probably become local warlords.

2. Farmers

Local neighborhoods with land and homes they can protect (see warriors above) will revert to small agrarian communities. This allows each person with land to begin growing food - at first to feed themselves and later to barter with neighbors. Many small urban communities are already doing this (think urban hipsters with their "square foot gardens", chicken coops, and other return-to-the-land initiatives). This will be a no-brainer opportunity for anyone with a house and a plot of land.

3. Scavengers

Initially, most of us won't immediately revert to the ultimate retro lifestyle. How many people know how to sew? How many know how to make more needles to sew with? Remember, manufacturing has probably stopped. Who among my readers know how to make a pair of shoes or a coat from animal skin or wool? We've grown indolent and forgetful in our lazy society with the automation that brought with mass-produced goods. We no longer have skilled craftsmen who can make things such as boots, utensils or even gardening tools. So those that can scavenge existing items will be able to immediately barter for food, protection, and other basic needs. However, these items will eventually run out. As they become scarce, there exists a risk of fighting over a small box of screws found in an abandoned hardware store with other scavengers or even the local thugs/warlords.

4. Engineers

It won't take long for stable communities to being looking for improvements. Engineers were the ones who built aqueduct systems to carry water over long distances to irrigate crops or keep domestic animals hydrated. Engineers are the ones who designed buildings as large as the pyramids and as small as a Engineers will be the ones to help design and build new things out of raw materials, whether it be from scavenged bricks and lumber or small trees cut from cleared field. And I'm not talking about software engineers either. We're talking about engineers who can design and build things, as well as direct groups of people when building larger projects (corn mills, dams, community fortifications, etc).

5. Priests

Many churches, mosques, and synagogues already have built-in communities near their places of worship, as religion has always played a large part of many people's lives. These communities will prosper by providing a system of community government, headed by the high priest (pastor, bishop, rabbi, etc). The Priest provides psychological validation and reassurance, functions as a leader of a community, and can quickly organize a community around a shared belief. The high priest will always need assistance, so the hierarchy of supporting priests will prove increasingly important. This will be an opportunity for those who have a natural talent for people skills over the physical labor involved farming or warriorship.

6. Medicine Man / Healer

It goes without saying, people (and animals) get sick. In many communities, this profession will be absorbed into the priesthood ("God heals those who pray"). However, some secular communities might retain a vestige of science or knowledge to provide relief to those in pain or in need of bodily repair. There will no longer be any pharmaceutical companies manufacturing pain relievers, cold medicine, itch creme, lib balm and the like, so the ability to locate herbs used in healing and ointments will be necessary. Additionally, someone who is willing to splint broken bones, sew up gaping wounds from war with neighboring communities or marauders, and assist women in childbirth while maintaining an iron stomach in such circumstances will be highly regarded.

7. Teacher

Teaching as a profession will be nearly lost, or at least so transformed as to be unrecognizable. Most trades, like farming or any craft will be taught in an apprenticeship. Parents will return to being the primary instructors of their children, passing on their family and community values along with the specific life skills their progeny might need. In many communities, the priesthood will absorb the responsibility of teaching the community at large, but there will be an opportunity for itinerant teachers to move among communities and peddle knowledge in exchange for food. A skilled teacher might convince a community to allow him entrance and give him food and shelter in exchange for the knowledge he brings from other communities. Many of those communities might be less eager to share their own proprietary knowledge, fearing it could be used against them. The skilled teacher will need to be adept at building trust and providing a balance of usable knowledge (specific skills, craft knowledge) and abstract knowledge (history, philosophy).

8. Merchants

As communities organize themselves, gain internal stability, and successfully defend their territories, they will eventually rediscover trade and commerce. A community with a surplus of corn may learn that the community nearby has a local healer who can teach their healer new techniques, or provide additional scavenged materials for engineers to complete local projects. This will lead to a merchant profession, as those who have mobility and can gain trust with strangers will be able to move goods and services between communities and gain status and eventually leadership.

So as we move toward economic chaos and the breakdown of society, think through what you know now. Think what function you might serve when your civilized society is limited to the small community in your geographic area. Think how you might survive, and even prosper in our new world. And good luck.

I'm a big fan of Ambrose Bierce and his Devil's Dictionary. Every once in a while, someone comes along and publishes a piece of satire worthy of Bierce himself.

I present to you, Crispian Jago's Periodic Table of Irrational Nonsense.

Thanks, Crispian. You've made my day.

When you're running HA-Proxy or Nginx in front of Apache, you lose client IP address information. The TCP connection to your Apache server come from Nginx, so all your logs reflect one single client IP address. This is a problem if you run a forum, as it will calculate the number of site visitors by IP address. Worse, this may flag the flood controls on some forum software. At minimum, it throws off log analysis.

X-Forwarded-For is an HTTP header that allows allows Layer 7 (HTTP) Proxies to pass along the original, external client IP to the next destination. To use this, your reverse proxy, caching server, or load-balancer must be configure to add that header to HTTP requests, and the destination point must be configured to look for it.

Everyone has a tutorial for configuring X-Fowarded-For across two servers (say, Nginx+Apache, or HAProxy+Apache). But what happens when you have an HAProxy load balancer balancing between three Nginx caches, which forwards to Apache for PHP/MySQL? That client IP address needs to passed across three different HTTP servers.

Assuming that HAProxy is has address 1.2.3.4, Nginx is running on another server with 1.2.3.5, and Apache is at 1.2.3.6:

HAProxy will need to be configured with this option to pass the X-Forwarded-For header of the connecting IP.
option forwardfor

Nginx will need the following to both *receive* the X-Forwarded-For header from the HAProxy server, and then *add* the X-Fowarded-For header to the new connection to Apache.

In your main Nginx.conf file:

set_real_ip_from 1.2.3.4; # this is the HAProxy connecting IP address
real_ip_header X-Forwarded-for; # The specific header to be read


In your proxy configuration, either in nginx.conf or in a separate include file, you'll need something similar to this:

location / {
proxy_pass http://1.2.3.6;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}


Finally, Apache can use the Rpaf module to read the X-Forwarded-For header from Nginx:


RPAFproxy_ips 1.2.3.5 # The Nginx connecting IP address
RPAFheader X-Forwarded-For # Apache is looking for this header to use as the "client IP"


References:
http://haproxy.1wt.eu/download/1.5/doc/configuration.txt
http://stderr.net/apache/rpaf/
http://wiki.nginx.org/HttpRealIpModule#set_real_ip_from
http://wiki.nginx.org/HttpProxyModule#Variables

Disclaimer: The links in the write-up below are for reference only. I receive no affiliate commission from traffic driven to those sites, nor am I endorsing them personally in any way. Some are vendors or partners of A Small Orange, some are not. Any opinions expressed here are not necessarily those of my employer.

My employer sent me to the Parallels Summit this week. I don't get many opportunities to go to Florida, so I was quite happy to enjoy some 80 degree weather in February.

This was my first time flying that I used the electronic boarding pass option. The electronic boarding pass is an SMS text message sent to your mobile device (iPhone, Blackberry, etc) with a link to a QR code. There is a QR scanner at the security check area, so you pull up the link in your browser, dip your phone into it and it displays your boarding pass info. The TSA agent checks your ID, and waves you through. Brilliant! No more worrying about leaving papers at home. Electronic Boarding Pass on my iPhone, with QR code.

When Delta first implemented the "self serve" ticketing check-in kiosks at the Atlanta-Hartsfield Airport, I jumped on it. It was a great way to bypass the long ticketing lines and the kiosks were never in use. Later, as airlines began directing people to the self-serve kiosks, I found myself stuck in long lines behind people struggling to figure out how to use the touchscreen check-in systems. Shortly afterward, I discovered online check-in and that I could print my boarding pass at home, bypassing almost everything in the airport and go directly to my flight. So this QR boarding pass was new to me, but I used it without hesitation on both the flight to Orlando and the flight back. I'm pretty paranoid about leaving all my papers and documents at home, so the QR code on my phone is pretty handy.

At the summit, I took copious notes during the sessions. Most of it boils down to "use Parallels software to put the word 'cloud' in your advertising". That's okay. It was their summit, so they can tailor the sessions however they want.

Parallels was pretty good about entertainment. There were social events/parties on Tuesday and Wednesday, both with open bars, and on Thursday morning they provided bloody marys and mimosas for the morning keynote (hair of the dog, I suppose). My Wednesday night party attendance was mired by having to work on a client emergency...several rather. Here's some free advice to any sysadmin attending a conference: don't answer your phone, don't look at email on your phone, and for Deity's sake, NEVER take a laptop to a party. You'll just end up working instead.

While the general sessions and presentations were filled with valuable information and not a few buzzwords ("enabling me to leverage my core competencies while partnering with vendors to provide cloud services thereby increasing revenue, reducing churn, and producing profitability in a new economy"), I found equally or more value in networking with other people. The vendor booths were chock-full of people ready to hand me a business card in exchange for mine, eager to sell to me valuable services and perhaps not realizing my ulterior motives were equally present. At some point, I discovered I was naturally gravitating toward booths with attractive sales women, so I had to make a special point to visit as many booths as possible and give them equal representation.

One of the booths staffed with attractive women who were also knowledgeable about their products and how they work, was Spam Experts. I was quite impressed with Brooke and Megan, especially as they are able to throw around terms like SMTP and and actually know what they're talking about. I had several enjoyable conversations with them, but had to eventually pull myself away and stop walking by their booth lest they think I was being overly flirtatious (or more likely, creepy).

My boss asked me to visit one of our vendors (OpenSRS) to say hi to our account rep. I stopped by several times and met everyone but. Apparently, our account rep wasn't working the booth that week. However, all the staff there were very outgoing and friendly, as were the staff at eNom (their competitor and also one of our vendors). The eNom people were pretty aggressive though. I could barely walk by without one of them tackling me and trying to sell me on their domain registration and SSL certificate services. Once they learned that we were already a customer, they were much more conversational.

Of particular note was thesslstore.com vendor. I first encountered this company in the swag bag of literature each attendee receives when registering. Instead of the usual whitepaper or datasheet provided by the other vendors, thesslstore.com had what looked like a direct mail sales letter. Same layout, same copy, same formulaic style of the "make more money with less work by joining our program today!". Instant suspicion. I thought they were a multi-level marketing company based around selling SSL Certificates at first. Later, sitting in the hotel/convention restaurant with my coworker, we saw thesslstore.com crew walk in. Six to eight people, all in high-contrast white button-down shirts and black pants, with a cultish look about them. I was scared to talk to them, for fear that they might gang-pressure me to join their MLM cult. After a while, I relaxed and took a look at their website, their "partner program" and found nothing out of the ordinary. Why did these people worry me so much? They looked really unnatural. The women were attractive, but...plastic. They looked like barbie dolls walking around. It dawned on me that the attractive women working for thesslstore.com weren't really employees. They were hired promo models! That would explain why they didn't make eye contact when passing people in the hallways. Every time I passed one of them in the hallway she would just look at me from the corner of her eyes, as if I were about to sneak up from behind and snatch her purse or possibly even talk about 128-bit SSL encryption.

I was also able to meet Anirban with StopTheHacker.com. They specialize in website malware scanning and reputation management (ie, Google malware warnings in search engines, etc). Since we are a relatively new partner with them, we discussed their billing system module, a possible cPanel module (I gave him some suggestions on how it might work and integrate for web hosts), and lots of general chit-chat.

The climax of the week came on Thursday when the Space Shuttle Discovery launched it's last mission. Cape Canaveral isn't very far from Orlando and hotel guests usually rush to the front parking lot to watch the shuttle launches. Unfortunately, this was all I got to see. Final Discovery Launch, view from Gaylord Palms Resort and Convention Center

I haven't quite decided if it was a letdown or not. I've been a fan of the Space Shuttle since the early launches in the 80's, when I was in elementary school. I remember when the teachers in my school would pause class, roll over the TV and we would watch the launch. It was always very exciting. Later, my fifth grade teacher (Vivian Woods) was an applicant to be part of the Teacher-in-Space program, the ill-fated mission of the Space Shuttle Challenger. We watched the Challenger explode that day on live TV.

The impression I got of the Discovery launch from my vantage point was more like a meteor or shooting star, just going the wrong way. It was pretty neat, but I really wish I had gotten a better view. However, it looks like there will be two more opportunities to schedule another trip to Florida...this time to Cape Canaveral.

I'm occasionally amazed when people complain about Facebook's policies. Facebook sells personal information that is freely given to it. What business do these people think Facebook is in?

Most people don't understand the business model at all. So here is an explanation of Facebook's business model and why Facebook does what it does.

You know about commercial radio, right? You listen to some top 40 music for free with periodic breaks for the local furniture store's obnoxious "Going Out of Business" ads.

Most people think the commercial radio station is in business for the listeners. But that's never been the case. The listeners do not pay money to the radio station. The listeners are not the customer. The listeners are the product.

Got that? Let me say it another way.

Radio station sales people go door to door to local businesses and say, "if you advertise on our radio station, we will guarantee 20,000 sets of ears during this time period, with 37% of those being suburban white female, stay-at-home-moms, 42% white males, employed with a salary of at least $40,000 per year or more, and 21% others." If your business caters to suburban stay-at-home-moms and their employed husbands, they will sell you access that audience (in 15 second slots) for a fee. Or the sales rep may say, "we have predominantly urban audience with 87% African-Americans, 10% white, and 3% Latino/Asian/other".

The radio station is in business for the advertisers, who are their customers. They sell a product, called an audience (you) to their advertisers. They have a lot of information on that audience, mostly the demographic breakdown such as race, gender, and income.

Loyalty cards work in a similar manner, except it's a little more personal. You sign up for a loyalty card at your grocery store and you get discounts for shopping there. In this case, you are the customer because you pay the money. But with each swipe of the loyalty card, your grocery store knows everything you purchased that week. And next. And they have your phone number, address, race, gender, age and all the other information you gave them with you signed up. So they may know that you are a 35 year old, white male who buys Morningstar frozen chicken patties. They can use that information to target sales and coupons toward your specific needs. If that 35 year old male is buying Morningstar chicken patties, maybe he'll also buy the Morningstar sausage if we give him $1.00 off. Maybe Morningstar is introducing a new flavor and wants to target a coupon campaign to existing buyers to try it out. Sounds fairly harmless.

Except, they sell that information to marketing partners. Did you notice that your Kroger Plus card now gives you discounts at Shell gas stations? Now Shell Oil knows that the 35 year old male who buys Morningstar chicken patties drives a vehicle that has a 15 gallon tank and takes 87 octane gas, and he fills up about once a week. At a specific gas station near the address registered on his loyalty card.

Kroger also prints ads on the back of their store receipts. They're not targeted ads...yet. But I'm betting they will be soon.

So again, your loyalty card is giving you something for free (discounts on what you're already buying), and they sell you as a demographic - complete with shopping habits - to their marketing partners.

Now here comes facebook. Poor old Facebook, plagued with privacy complaints that they are taking all this personal information and selling it to everyone.

What did you expect? This is the nature of their business model. They give you access to a free service, and then sell that demographic information to advertisers, application developers, and the others who are Facebook's *real* customers. They are doing nothing that hasn't already been done.

There is absolutely no difference between Facebook, a commercial radio station and loyalty cards. When you give your information to a third-party in exchange for a free service, you can trust that they intend to use it for their monetary gain...every time.

Years ago, I discovered the benefits of reusing pallets when I managed a [now-defunct] I Sold It franchise (I Sold It was a chain of eBay drop-off stores). We had a large item to ship that required a pallet for freight shipping, and I didn't want to spend $100 on pallets for a single use ($20 per pallet, quantities of 5 or more). A short search around the rear of our building yielded a couple of pallets, one of which was usable.

When I realized how many pallets are discarded every day around the Atlanta area, I initially thought it might be a good side business to gather up discarded pallets and sell them at a discount. Unfortunately, the economics of dumpster diving for pallets was not very profitable: a business owner is not likely to pay regular price for a used pallet, and to sell on any large scale involved a large commitment of time and resources - hours of driving and gasoline for the car to find and gather them. Though I had space to store them, gathering them up until I had enough to sell wasn't really an option. So my fantasy of a recycling pallet business dissolved as soon as I thought of it.

Adirondack Chair: Photo blatantly copied from Shelton's Flickr page, but used in a fair-use reporting context. If you click on the image to view all his photos, maybe he won't send me a DMCA takedown notice.I had a pleasant surprise at Indie Craft Experience this past weekend. While browsing the usual selection of vendors selling knitted hats, recycled felt skirts, and other crafty stuff, I came across a guy sitting in a wooden chair that looked suspiciously like it was made from pallets.

I have a tendency to stop at any booth featuring wooden furniture items (especially at art fairs, where handmade woodworkers display beautiful and amazing furniture). So there was no hesitation to stop and look at this interesting piece of furniture.

Shelton Davis (the guy sitting in the chair) was there representing Repurposed Goods, a project to reuse discarded pallets into new creations. The chair was made completely from recycled pallets, and he also had on display a birdfeeder. He was selling complete DIY building plans for the chair and bird feeder, as well as a DIY "Ikea-style" kit for the bird feeder that one could build immediately.

All the plans were printed on recycled paper, and he also sells the plans as downloadable PDFs through his Etsy store. It turns out Shelton is an industrial designer, so not only is he able to create plans for pallet repurposing, he is able to convey the instructions in a simple and clear document.

I have his birdhouse instructions, and plan to purchase his Adirondack chair from his Etsy store once I have some time to devote to a bigger project.

I think I have found a reason to start stacking up pallets again.

Recently, I was called to assist with a server that was constantly getting bombarded with HTTP connections and causing Apache to hit MaxClients. It took a couple of minutes to track the IPs with the most connections using this little command:


netstat -tpnC | grep httpd | awk '{print $5}' | cut -f1 -d: | sort | uniq -c
1 65.55.25.149
4 67.141.163.123
1 67.195.111.40
1 71.247.16.159
28 137.244.215.55


This showed a large number of connections from an IP 137.244.215.55.
Since we have ExtendedStatus enabled in httpd.conf, it's a simple matter to find what site is getting hit:


# lynx -dump -width=160 http://localhost/server-status | grep -e '...[1-9].*' | grep -v OPTIONS


Sure enough, there's the IP and the site getting hit, along with the request URI. I like to use this to see when a comment spammer is POSTing to Wordpress blog or otherwise trying something malicious. The site getting hit was a Fantasty Football site.

However, I was a little concerned because the IP that had a large number of connections was a .mil IP address.


# host 137.244.215.55
55.215.244.137.in-addr.arpa domain name pointer uhhz-wpa-001.robins.af.mil.


So I simply used ConfigServer Firewall (a nice front-end to iptables) to block the address for an hour.


csf -td 137.244.215.55 3600



9:45:00 AM Other Tech: block the .mil!
9:45:45 AM Me: blocked 'em.
9:45:49 AM Me: just for an hour though.
9:46:39 AM Me: in case it's the airforce cybercommand thingy that's investigating a terror suspect and enemy combatant.
9:47:05 AM Me: because a fantasy sports site is where all the terrorists hang out....


All was well and good, but shortly after blocking it another .mil address hitting the same site with a large number of connections:


netstat -tpnC | grep httpd | awk '{print $5}' | cut -f1 -d: | sort | uniq -c
1 124.115.4.197
1 186.18.102.180
1 38.99.98.113
1 65.55.25.144
1 68.216.159.34
1 68.240.147.150
32 131.51.128.21


After blocking that one, a wellsfargo.com address showed up, then a few other random corporate addresses. This beginning to concern me, since this resembles some sort of DDOS behavior from a bunch of infected PCs (and the idea of .mil and Wells Fargo computers being infected didn't sit well with me). But it wasn't a very effective attack, since only one or two IPs would hit at the same time. And why would someone attack a Fantasy Football site with military and banking computers? Sure there are better better targets than that!

Next step was to watch the logs for a bit (tail -f /path/to/domain/access_log). I noticed some odd behavior. Usually, a browser hits a site, requests a page, and then requests all the supporting files (CSS, javascript, images, media files, etc), usually listing the original referring URL along the way. This was a Wordpress blog, so most traffic was fairly normal along these lines. But grepping for the specific IP addresses in the log showed a more unusual pattern: a single request from a generic user-agent string ("Mozilla 4.0 (Compatible)"), followed by a large number of requests for all the links on the page. Something like this:


131.50.151.28 - - [16/Oct/2010:11:39:28 -0400] "GET /baseball/wp-content/themes/SomeTheme/style.css HTTP/1.1" 200 404 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201009 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201007 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200910 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200912 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201006 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200908 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200911 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/xmlrpc.php HTTP/1.1" 200 404 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201010 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200909 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201008 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200905 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200811 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/xmlrpc.php?rsd HTTP/1.1" 200 408 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200809 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?p=5 HTTP/1.1" 200 408 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200808 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200810 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200904 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200906 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?p=1961 HTTP/1.1" 200 411 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200812 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?p=1989 HTTP/1.1" 200 411 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200907 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"


I watched this for a while, scratching my head. The "Mozilla/4.0 (compatible;)" was suspicious. It was immediately obvious that it was some kind of bot or spider. Bad bots like to disguise themselves or try to pass themselves off as real browsers to avoid detection or redirection based on their behavior. So I was beginning to think this was a really bad indexing/search spider. Except that it's hitting this one single site, and there there were also referrer links in some of the lines indicating traffic from other sites. And bots are usually operated from a single IP address - they don't spring up from other IPs when the first one is blocked.

Troubleshooting is often a team effort, and it certainly helps to discuss a problem and brainstorm ideas.


11:08:53 AM Me: i'm beginning to think that our military is not infected with bots, but are goofing off with fantasy football, which scares me even more.
11:10:31 AM Other Tech: The useragent is weird though
11:11:23 AM Me: yeah. maybe they're behind a proxy server that grabs all the linked pages for immediate caching.


Sure enough, a quick Google for "Mozilla/4.0 (compatible;)" yielded some hits of exactly that. Behind the corporate doors of Wells Fargo and various Air Force bases are a bunch of people reading up on Fantasy Football, and their collective proxy servers (probably Blue Coat) are slamming the server with a ton of requests to pre-fetch all other linked URLs from the first page, so that each visitor is hitting the server with dozens of connections.

Since this is a shared server, this is affecting not only the customer site in question, but all other sites hosted on this server. It's obvious we can't block by IP address, and there are too many variations to block entire ranges (which is a bad idea to begin with).

My solution was to redirect on the User-Agent string until the issue died down. I created a single HTML page on the server's main DocumentRoot (outside the customer's virtual host) and added the following lines to the customer's .htaccess file:


RewriteCond %{HTTP_USER_AGENT} ^Mozilla\/4\.0\ \(compatible;\)$
RewriteRule .* http://host.example.com/denied/ [R]


The actual index.html page in the Redirected URL sums it up like this:


Your "web accelerator" proxy is causing problems with our servers and customer sites.
Sorry, but you will not be able to access content here.
Please contact your IT Support department for assistance.


Addendum:
I found the script on this page very helpful when testing the User-Agent string in my .htaccess rule. While I am quick to telnet to a webserver to pass an HTTP request a simulate a browser visit, I don't know all the details of the HTTP protocol (including the format of User-Agent string). Scripting this to quickly connect to localhost, pass the request and the User-Agent and see if I received a 200 or 302 Redirect was extremely helpful.

As the [former] Customer Service Manager at A Small Orange, comments, questions, and even complaints were frequently brought to my attention.

One common question is "why do you offer so little disk space compared to your competitors?" Or to phrase it another way, "why are your shared hosting plans so stingy?"

This is an understandable question. From the customer's viewpoint, they can (as of 2010) buy a 1TB disk drive from Newegg.com anywhere from $59 to $89. "Drives are cheap!", they cry. "Add more drives! I've got 3TB in my desktop PC, why can't you do it on my server?"

Servers don't quite operate this way.

For one thing, the hard drive in a server is expected to be reading and writing data almost constantly 24x7, instead of spinning up occasionally for downloads, torrents and gaming. That constant disk reading/writing will shorten the usable life of the drive. When a server's hard drive dies, it's not just one customer's data that is lost...it can be several hundred customers' data. So a server needs some failover, or redundancy. For a low-cost shared server, that might mean two drives in a mirrored (RAID-1) configuration so that one drive fails, the other one keeps going with a copy of the data until the first one can be replaced. A higher-end server might even have four or six drives, with some combination of mirroring or data protection (RAID 0+1 or RAID 5, for example) so that one or more drives can fail without the losing valuable customer data.

With this redundancy alone, we've gone from doubling the cost (two $90 drives for $180) to sextupling the cost (six drives at $534).

Now consider the performance of the drive. The cheapest $59 drive on Newegg would be be a "green" drive. It is a low power, low speed (5400RPM) drive, and has a very low cost per-Gigabyte. I have a couple of these drives in my home storage box and they run just fine. But I don't access my data at home in the same way a web server will. There is an $89 drive on Newegg that is a little bit more powerful - mostly faster (7200RPM). It's great for a PC desktop or home storage unit, and maybe even a low-traffic server.

But when you have constant usage, that drive needs to access server data fast and go to the next request because the slower the hard drive is, the fewer browser hits those websites can handle. The standard SATA drive has a 7200RPM rotation speed, and that's pretty good for desktop usage and low-end servers. For higher performance, SCSI and the newer SAS drives are still the best. A 300GB SAS drive currently runs about $279 and they're *fast*. They spin at 10,000-15,000 RPMs, and the SCSI protocol they use offers superior performance in server environments.

So now, instead of $180 for 1TB of storage (two SATA drives, mirrored), we now have higher performance drives giving us 300GB of mirrored storage for $558. That's 300GB of fail-safe, high-performance server storage for $558 versus a single $89 1TB drive in your PC. Three times the cost, but twice the speed and performance.

For even better performance, using a hardware RAID controller instead of software RAID is important. This hardware is at least $200, so now you get 300GB for $758 or more.

Apples and Oranges indeed!

This should provide some perspective on the differences between consumer and business server storage. The next logical question is, how do other hosting companies provide so much more storage, like 50GB or more per customer?

First, many hosting companies use a control panel software (such as cPanel) on their servers. This software is used on a single server, with a limited number of hard drives (usually 2, 4, or 6) and turns it into a profit-machine. You have a set number of customers per server, and when you run out of space, you buy a new server with a limited number of drives and start filling it up with customers.

Using A Small Orange's Medium hosting plan as an example, a single 1TB drive would allow up to 600+ customers to (1000GB divided by 1.5GB of space per customer). Of course, A Small Orange is not using single desktop hard drives, but mirrored high-performance drives. The drives actually are 146GB SAS drives, so about 97 customers could share that space if they all had the Medium plan.

The more customers a hosting company can stuff onto a single server, the more money you make. So many hosting companies use a technique called overselling. This means the hosting company will offer more space per customer (say, 10GB or 50GB or even unlimited storage) knowing that the most customers will never use it all. If every customer on that server filled up their 10GB or 50GB of space, the server drive space will fill up very quickly and cause problems. To prevent this from happening, there are clauses in the Terms and Conditions that define other limits, usually number of inodes. This prevents customers from filling up the disk with a large number of small files. Additionally, there are usually "fair usage" clauses with "unlimited storage" plans to allow the hosting company to suspend service if the customer is using more than a fair share of the overall disk amount.

With the price of Storage Area Networks and Network Attached Storage devices becoming more accessible to smaller operations, some hosting companies are moving away from fixed storage like physical disks. By implementing a high-performance network storage, a given server can increase it's storage capacity on demand to meet the increasing storage demands of its customers. As this becomes more common, larger storage quotas (*without* overselling) will eventually become more commonplace.

I was invited by Mike Shinkle to participate on a couple of panels this Wednesday at The Business of Wordpress conference.

I'll be participating in two sessions:

• Operations: Support, Maintenance, Disaster Recovery, and Scaling
  
  This is a technical-oriented session targeting backups, security, and other technical concerns of a business Wordpress site. I'll be there to provide perspective and information as a hosting provider.
• 
  Navigating your Choices for Hosting
  
  This is geared more toward businesses looking to understand their hosting choices (shared, dedicated, cloud, etc).

This should be a fun conference and I look forward to participating!

Disclaimer: My employer is one of the Gold sponsors.

On a related note, I had a devil of a time trying to find a headshot for the organizers to use in my presenter bio. I settled on an old Facebook profile pic that's about two years old. They weren't entirely happy with it, since they preferred something less cropped that they could adjust for uniform consistency. My wife dutifully scoured her albums for other pictures of me, but everything was in some way not ideal for a headshot (too "silly", a busy background that detracted from the photo, out of focus, etc).

I was also asked for my website. Oh. *My* website.

I guess other people maintain websites. I'm usually fixing other people's websites, so I never quite get around to doing anything with mine. It even still uses the default theme colors (yuck).

There are lessons here. If you are going to be participating in public, you better be ready to present your public face.

In the meantime, I hope that all the Wordpress users at this conference don't notice this is a Drupal site...

One would think technology companies would be relatively clued-in to the nuances of business communications, and look for real, targeted opportunities to make sales. Blasting an scattershot email to a large number of random email addresses is about as effective as direct mail is these days (does anyone actually read junk mail?).

But alas, even Cisco is prone to those sales droids who still think that one has to market to (and consequently annoy) millions of unqualified prospects in order find one person willing to talk to them.

Gretta Spaulding is subject of our analysis today. Attached below is the missive that arrived in A Small Orange's helpdesk ticketing system.

The first thing we should note: this was not sent to a personal email address. This was sent to a helpdesk email address. This address is solely used for customers to contact the hosting company. It is not used for non-support, business correspondence in any way (we use our own email addresses for that). This address is never used for opt-in marketing communications. This tells us she's trawling for email addresses or purchasing email lists, probably from a somewhat shady supplier ("100,000 verified email addresses for only $19.99!"). So this is already an inappropriate correspondence, and is wasting the time of our helpdesk technicians who now must read through this and delete it when they could be more usefully addressing customer problems.

Second, even if the advertisement were in some way relevant to our business, the helpdesk is probably the last place she should be sending correspondence. How many Level 1 Helpdesk Techs do you know of that are decision-makers and/or have any sort of purchasing authority? That's what I thought. Why Cisco approves of their sales staff targeting inappropriate prospects through unsolicited communications is beyond me.

Third, the subject line is a clear violation of the CAN-SPAM act, namely "don't use deceptive subject lines". I fail to see how "I've got your keys" has anything to do with WebEx products or services.

Looking at the headers information, this unsolicited email was sent through a third-party marketing list service, called Genius Network. A quick look at their home page shows a Wordpress blog with two pages. They at least include an abuse@ address, and I have already forwarded the email to them. I've also sent this to abuse@cisco.com, but I highly doubt anything will happen there. Cisco is such a spawling organization that it would probably be a low level admin that receives it, escalates it to his manager, who then has to forward it to the VP of Sales, who then has to make it filter down to Gretta Spaulding's direct superior. At any point in that chain, someone is going to throw it away thinking, "Oh, it's just one complaint. It doesn't matter."

[Update: I received an auto-reply from Cisco, also attached below. They're so immune to the idea that someone in their organization might be sending spam that they offered several alternate contact addresses for various aspects of Cisco security. But...I don't need to address Cisco security, I need Cisco to stop sending unsolicited commercial email!]

You may be asking yourself, "Why don't you just use the unsubscribe link at the bottom and end it there?" Because the helpdesk never subscribed to this list! I fail to see why I should have to perform extra work because Gretta Spaulding and Cisco decided to obtain an illegitimate list of emails to spam their WebEx service.

Note to Cisco managers, executives, and lawyers: before you try to file suit against me for libel, you may want to ask yourselves...does your reputation really hinge on a blog entry from one person, or from the millions of people who received your spam? Are any of the statements presented above patently false? (I maintain, "no"...re-read the spam email I posted).

Also for the lawyers: Cisco and WebEx are registered trademarks of Cisco Systems, Inc.