Emergent Chaos
I would estimate that 2/3 of the calls I get are from people trying to sell me things I neither need nor want. Of those, over half are outsourcing services. Of the remainder, recruiters are over half.
There are also people who call me for their services once a week. There's one particular outsourcing firm whose name is burned into my brain because of the number of times I've been subjected to it. I don't know how to spell their name, but I can sure pronounce it. There's also a recruiting firm that I know well, too. Each of these people I have asked to take me off their list, asked to talk to supervisors, talked to supervisors, yelled at them, ranted at them, and finally sworn at them, and yet I still get my weekly call.
As I was doing office stuff a few moments ago, I played a voicemail, and it was from my friends at Hadron Infotech, letting me know about their services just in case I have (a) developed a need I didn't have last week and (b) forgot their name. (One of my rants included telling them that when I do need such services, they will be the last people I call and sadly for them, I have no trouble remembering their name.)
Since I was doing office stuff, I let the message drone on, and got the litany of things they can do for me including, Java, Jay-mumble-E, Dot-Net, Pee-Haitch-Pee, AJAX, Perl, Ruby on Trains, updating your web site, ....
Wait a minute. Did he say what I thought he said? Ruby on what? I ran over to my computer, backed up the player, and ... Yes! Ruby on Trains! How delightful!
I'm still laughing. I hope you are, too. Maybe I'll get another laugh next week.
Photo "Ruby on Train" by theresa_l_reed.
Bookmark this post:
I've been in the hotel I am in for over a week now. It is a European hotel that has wireless, and you have to get an access card and type a six-character string into an access web page. That authenticates you, and you can go.
The problem I have today is that I can browse the net completely. But I can't do anything else. No email, no vpn, no ping, no traceroute, no nothing. If I telnet to a useful port on my own servers, I get a syn/ack/syn and no flow.
My hypothesis is that whatever does a redirect on port 80 to get you to the authentication web page is broken.
I've talked to first-line tech support at the provider who let it slip that he thinks its in the firewall at the hotel. This is consistent with my evidence. However, he won't let me talk to anyone who actually knows what "ping" is. I have talked to someone at my front desk, who has talked to the local IT person, and we've had mediated back-and-forths.
If I could actually talk to someone who knows what a web redirect is or even what a "port" is, I could let them know. If I knew the URL of the authentication page, I could tell them the problem. The local IT guy is presently talking to the ISP, but I told the gal at the desk that I'm an IT person, too, and if their IT guy will call me, then I will help explain the problem.
As a matter of fact, while writing this, I just connected to an https url, which redirected me to the authentication page, and now everything is working. This is how you're reading this today. So I know what their problem is and can tell them how to fix it. They just have to know that I know, and that I'm not a mere luser.
We need an IT secret handshake. Perhaps Randall Munroe can help. Remember those old stories about the Freemasons in some pickle or another who suddenly showed the handshake? We need one.
Update: The gal at the front desk has called back. The ISP and the local IT people have decided this is actually my problem. However, she also says that another guest has this problem. I explained this as much as I could to her, and told her to tell the other guest to go to an SSL web page to fix it.
Photo courtesy of photos.tjweb and selected because it matched a search for "authentication web page"
Bookmark this post:
I have been playing with Splunk, for about 45 minutes.
So far, I like it.
I've previously been exposed to Arcsight, but what I have more of an affinity for psychologically is not so much a correlation engine, but a great visualization tool that automagically can grok log formats without making me write a hairy regex. I have no idea whether I will use Splunk for anything real, but it made a good first impression. Since my budget is zero, the price of the non-enterprise version looks good, too. I am sure that for those of a less penurious station, there are many more fine contenders.
Bookmark this post:
In my last post on security, I promised a tale, and I ought to deliver on that before it becomes nothing more than a good intention.
Some time ago, so long ago that it no longer matters, I bought a piece of network stereo equipment. It was one of these little boxes that lets you play MP3s, etc. through your stereo. I got it because it was a cute little system running Linux, had a MIPS processor, a web site for developers, extension and enhancement tools in Java, and so on.
I used it for a couple of months, and played with the Java-based remote control application for it and then decided to do some more serious work on it. I rolled my eyes that it only had telnet to get to it, but telnetted to it and was met with:
#
which I just stared at for a moment. It didn't even register for a good twenty or thirty seconds before I had the wit to type
lsand was met with something akin to:
bin dev home mnt proc tmp boot etc lib usr sbin
and that didn't even register with me until I finally then typed
pwdand was met with
/
and I made a loud two-word exclamation, of which the former was "oh" and the latter is left as an exercise for you, Gentle Reader, but there are two obvious candidates.
Yup, for the last couple of months, sitting bear-ass nekkid on the Internet was a Linux box with open telnet and a root shell. No username, no password, just a root shell. I said the other obvious candidate word. I also considered (again) getting a firewall. My network doesn't have a firewall. Part of it is that I like the road feel of the packets whizzing by. Part of it is that by the time I open up enough ports to do useful things, I'm just closing down the ones that don't have services on them anyway. Part of it is also that of the three times I've had serious security problems on my network, one of them was because my IDS box got rooted, and one was because the firewall got rooted. For me, adding a firewall adds complexity, and that lowers security. (That last time was when I was traveling with my SO who wanted to send me an email from an utterly ancient netnews program that knows nothing of SMTP-AUTH. Never reconfigure your email infrastructure from five thousand miles away while jetlagged. A couple of days later, you will ask yourself, "I wonder why the SMTP server logs have gotten so big." Fortunately for me, I caught it before the blacklists did.)
I yanked the music box off the network and connected to it directly (one cable, just it and me). Looking through the thing, I didn't see what anyone who was now using it for anything. I checked the IDS logs and there was nothing that leapt out at me to as suspicious traffic. That seemed odd, because how could it not have been owned? I thought about it for a bit, and thought about it more as I reflashed the critter. Then I laughed, because I realized that the tools that probe for vulnerable boxes are not going to be looking for #. It was then too late to tell, but I allowed myself to think that maybe the box hadn't been compromised, as the evidence suggested.
With the machine rebuilt, I connected to it directly with telnet and started probing around for putting a password (like /etc/passwd). There was none. There was no SSH, either. I fulminated on the developer fora about this security stupidity. I found the instructions on how to build the right cross-compiled Linux setup to build binaries for it, and it was filled full of warnings about how to make sure you did this, set that compiler switch, and if you didn't, things wouldn't work, and you get to reflash the box.
This wasn't how I was wanting to spend my Saturday, so I turned the box off, and went to do something else. As I did, I thought about the situation. I became increasingly amused that (apparently) the box hadn't been compromised. I convinced myself that this is because the bad guys wouldn't recognize the box as vulnerable.
As I grumbled and thought more about how to lock down the box and then something occurred to me -- anyone who wants to own the box has to go to the same trouble to make it be a productive member of their botnet community as I do to do the opposite, but they're at a disadvantage because they also have to protect it from me. Since it's easier to find some unpatched Windows box than it is to set up a MIPS cross-compile sandbox, even if they can tell that has an open root shell, it's not economically viable. Think of it as Mutual Assured Annoyance, Economic-Based Intrusion Prevention, Security Through Stupidity, or proving old adage, "In the land of the blind lion, the one-eyed zebra doesn't have to run very fast."
A couple of weeks later, I solved the whole problem when a new product was introduced that did exactly what I wanted (to be able to play music on my laptop on my stereo) at half the price and no icky telnet. The poor little music box now sits face-down, forlorn, and dust-covered on a shelf.
Bookmark this post:
Adam comments on some breach commentary, and quotes Nick Owen saying that breaches are a sign of incompetence.
I can't let this stand un-commented-upon. I believe that that is a dangerous comment, and one that needs to be squashed early. It's like saying that a bug tracking system with lots of bugs in it is a sign of engineering incompetence. It actually means the opposite. A truly incompetent management team wouldn't know they'd been breached. A slightly less incompetent team would bury it under the rug. This is true for software developers as well as operations people.
This is a very dangerous comment because it rewards the truly incompetent who don't know how screwed up they are. It is a dangerous comment because it rewards the mendacious, who hide that they've been breached -- or who design their operations so they won't know when they're breached. Stop. You're going to set us backwards if you keep that up.
It doesn't matter how good you are, some day you will be breached. Accept that. As a consumer, that's a mildly unpleasant thing to think of, but it's true. However, you want people who lose your data to have the wit to know they've lost it, and the morality to own up to it.
I also want to comment on Allan Friedman's comment about Iron Mountain, as I've noticed the same thing, that many breaches involved Iron Mountain losing tapes. But I'm not an economist, I'm a guy who's spent times in operational groups, and I have an alternative hypothesis.
Let us assume an organization that makes daily backups and sends them to a data warehouse. Let us suppose that the tape monkeys have a Very Bad Day. Sam's on vacation. Ginger broke up with her boyfriend and came in late. Two tapes verified bad and had to be re-done, Networking misconfigured something and you couldn't get to C Building at all. The Iron Mountain guys come in to get the tapes from you, and you tell them the horror story. They say hey, no problem, just give them what you have. They'll take it off to the warehouse, and as long as there's no disaster tomorrow, it'll all be taken care of in the next incremental. The CIO never has to know. Whew! Thanks, Iron Mountain! You're a life saver.
Iron Mountain is being smart. The real customer is the supervisor of the tape monkeys, and if you help him shine, he helps you shine. Alas, they're being smart until lost data is not simply a gap in the backup history, it's a breach. Then this habit of mutual back-scratching all falls apart. If someone does an audit and finds out that a backup of the Order Database is missing, Iron Mountain takes the fall. All the paperwork says that the database was backed up, put onto tape 1723-A5, and sent to the warehouse. And therefore, so it was. Iron Mountain can't say, "Um, actually, for years now, we've been covering for our customers and letting them claim data was in the warehouse when we all know it wasn't." They just have to take it on the chin.
You know what? The real customers, the tape monkeys who have been let off the hook yet again know that Iron Mountain kept them out of even bigger trouble. They know that the Iron Mountain guys can't let them hand over an empty box any more. But they aren't going to switch to another company, either.
My hypothesis could be wrong. I don't know if it is. I can't admit to ever having been in a situation like my hypothesis. I am, however, a cynic, and I know that if Iron Mountain were in the habit of losing tapes, it may or may not show up in their stock price. But if they were in the habit of making the tape monkeys look more competent than they actually are, it is consistent with observed phenomena. It doesn't mean my hypothesis is right; heck, the magic blue smoke theory of semiconductor physics is consistent with observed phenomena. But when I noticed Iron Mountain showing up in a number of breaches, the smoke I smelled seemed to have a hint of electrolytic capacitor in it, and whiff of insulation.
Bookmark this post:
A bit of background.
Sun recently got hit with a 0-day that was 13 years in the making, by seemingly repeating a coding worst practice that bit AIX back in 1994 -- trusting environment variables under the control of an attacker. A slightly more complex variant bit Solaris' telnetd in 1995.
From the advisory (NSFW) at http://www.com-winner.com/0day_was_the_case_that_they_gave_me.pdf, as sent to me in an email:
/usr/src/cmd/cmd-inet/usr.sbin/in.telnetd.c
3198
3199 } else /* default, no auth. info available, login does it all */ {
3200 (void) execl(LOGIN_PROGRAM, "login",
3201 "-p", "-h", host, "-d", slavename,
3202 getenv("USER"), 0);
3203 }
Anyway, to save you valuable time, the pre-blogging department at EC has prepared a short summary of some posts you will see elsewhere on this topic.
"This proves open source is less secure than closed source"
This vulnerability is so Old Skool, it could have lain dormant like some sort of unexpressed genetic flaw. By making source available, Sun provided a road map to their own weakest link. You can hear the chortling in Redmond already.
"This proves the value of open source"
Low-hanging fruit like this is ripe for the picking. By harnessing the people power of a million eyeballs, cruft like this will be much more quickly eliminated. For all we know, Vista's telnet service is even worse.
"Don't they teach these kids to write half-decent code???"
[This one will be found in a blog containing a "Created with vi" emblem in the corner, and embedded RCS keywords showing its last mod time.]
(Note: The pre-blogging department accepts no responsibility for metaphors mixed in the provision of this valuable service. YMMV. Do not taunt happy fun ball.)
Bookmark this post:
Bookmark this post:
% prstat PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 14135 nobody 16M 12M sleep 60 0 0:00:11 4.2% mt-tb.cgi/1 14207 nobody 14M 11M run 55 0 0:00:08 4.1% mt-tb.cgi/1 14203 nobody 14M 11M run 56 0 0:00:08 4.1% mt-tb.cgi/1 14209 nobody 14M 11M run 54 0 0:00:08 4.1% mt-tb.cgi/1 14215 nobody 14M 11M run 54 0 0:00:08 4.0% mt-tb.cgi/1 14213 nobody 14M 11M run 57 0 0:00:08 4.0% mt-tb.cgi/1 14199 nobody 14M 11M run 58 0 0:00:08 4.0% mt-tb.cgi/1 14181 nobody 14M 10M run 48 0 0:00:09 4.0% mt-tb.cgi/1 14173 nobody 16M 13M run 58 0 0:00:09 4.0% mt-tb.cgi/1 14187 nobody 14M 10M run 48 0 0:00:08 4.0% mt-tb.cgi/1 14147 nobody 12M 9912K run 58 0 0:00:10 3.9% mt-tb.cgi/1 14169 nobody 12M 10M run 58 0 0:00:09 3.9% mt-tb.cgi/1 14185 nobody 16M 13M run 48 0 0:00:07 3.8% mt-tb.cgi/1 14195 nobody 12M 9992K run 58 0 0:00:08 3.8% mt-tb.cgi/1 14094 nobody 16M 11M run 55 0 0:00:11 3.7% mt-tb.cgi/1 14197 nobody 16M 13M run 54 0 0:00:07 3.6% mt-tb.cgi/1 14179 nobody 12M 9952K run 58 0 0:00:08 3.4% mt-tb.cgi/1 14159 nobody 12M 9768K run 54 0 0:00:08 3.3% mt-tb.cgi/1 14153 nobody 16M 9752K run 58 0 0:00:08 3.2% mt-tb.cgi/1 14167 nobody 12M 10M run 58 0 0:00:09 2.9% mt-tb.cgi/1 14247 root 4568K 4304K cpu0 59 0 0:00:00 0.3% prstat/1 Total: 73 processes, 146 lwps, load averages: 32.93, 31.82, 16.88 %
Somebody in the Ukraine has too much time on their hands :^)
Bookmark this post:
Many years ago, I needed to deploy a bunch of UNIX machines very quickly. When
I created the golden system image, it included an ntp.conf file that pointed
to a nearby public stratum 2 server not under my administrative control. This
was dumb, because I could (and should) have just had my boxen chime against a
local machine, and have it (and only it) hit the stratum 2 box.
About a week later, I got an email from the admin of the stratum 2 server
that politely pointed out these facts. A half hour later, cfengine had
fixed my error, and I red-facedly responded to the gent whose resources I
had ignorantly abused.
Well, a Danish member of the FreeBSD team is running up against the same
issue on a much larger scale, except that D-Link is the abuser, and they aren't fixing the problem.
Bookmark this post: