Category Archives: Metrics

Survey for How to Measure Anything In Cybersecurity Risk

Posted on by

This is a survey from Doug Hubbard, author of How To Measure Anything and he is
currently writing another book with Richard Seiersen (GM of Cyber Security at
GE Healthcare) titled How to Measure Anything in Cybersecurity Risk. As part of
the research for this book, they are asking for your assistance as an
information security professional by taking the following survey. It
is estimated to take 10 to 25 minutes to complete. In return for participating
in this survey, you will be invited to attend a webinar for a sneak-peek on the
book’s content. You will also be given a summary report of this survey. The
survey also includes requests for feedback on the survey itself.

https://www.surveymonkey.com/r/VMHDQ7N

A Mini-Review of “The Practice of Network Security Monitoring”

Posted on by

NSM book coverRecently the kind folks at No Starch Press sent me a review copy of Rich Bejtlich’s newest book The Practice of Network Security Monitoring and I can’t recommend it enough. It is well worth reading from a theory perspective, but where it really shines is digging into the nuts and bolts of building an NSM program from the ground up. He has essentially built a full end to end tutorial on a broad variety of tools (especially Open Source ones) that will help with every aspect of the program, from collection to analysis to reporting.

As someone who used to own security monitoring and incident response for various organizations, the book was a great refresher on the why and wherefores of building an NSM program and it was really interesting to see how much the tools have evolved over the last 10 years or so since I was in the trenches with the bits and bytes. This is a great resource though regardless of your level of experience and will be a great reference work for years to come. Go read it…

Seattle in the Snow

Posted on by

Seattle snow (From The Oatmeal.)

It’s widely understood that Seattle needs a better way to measure snowfall. However, what’s lacking is a solid proposal for how to measure snowfall around here. And so I have a proposal.

We should create a new unit of measurement: The Nickels. Named after Greg Nickels, who lost the mayorship of Seattle because he couldn’t manage the snow.

Now, there’s a couple of ways we could define the Nickels. It could be:

  • The amount of snow needed to cost a Mayor 10 points of approval rating
  • The amount of snow needed to cause a bus to slide down Olive way and teeter over the highway
  • 2 millimeters
  • Enough snow to reduce the coefficient of city road friction by 1%.

I’m not sure any of these are really right, so please suggest other ways we could define a Nickels in the comments.

Emergent Map: Streets of the US

Posted on by

This is really cool. All Streets is a map of the United States made of nothing but roads. A surprisingly accurate map of the country emerges from the chaos of our roads:

Allstreets poster

All Streets consists of 240 million individual road segments. No other features — no outlines, cities, or types of terrain — are marked, yet canyons and mountains emerge as the roads course around them, and sparser webs of road mark less populated areas. More details can be found here, with additional discussion of the previous version here.

In the discussion page, “Fry” writes:

The result is a map made of 240 million segments of road. It’s very difficult to say exactly how many individual streets are involved — since a winding road might consist of dozens or even hundreds of segments — but I’m sure there’s someone deep inside the Census Bureau who knows the exact number.

Which raises a fascinating question: is there a Platonic definition of “a road”? Is the question answerable in the sort of concrete way that I can say “there are 2 pens in my hand”? We tend to believe that things are countable, but as you try to count them in larger scales, the question of what is a discrete thing grows in importance. We see this when map software tells us to “continue on Foo Street.” Most drivers don’t care about such instructions; the road is the same road, insofar as you can drive in a straight line and be on what seems the same “stretch of pavement.” All that differs is the signs (if there are signs). There’s a story that when Bostonians named Washington Street after our first President, they changed the names of all the streets as they cross Washington Street, to draw attention to the great man. Are those different streets? They are likely different segments, but I think that for someone to know the number of streets in the US requires not an ontological analysis of the nature of street, but rather a purpose-driven one. Who needs to know how many individual streets are in the US? What would they do with that knowledge? Will they count gravel roads? What about new roads, under construction, or roads in the process of being torn up? This weekend of “carmageddeon” closing of 405 in LA, does 405 count as a road?

Only with these questions answered could someone answer the question of “how many streets are there?” People often steam-roller over such issues to get to answers when they need them, and that may be ok, depending on what details are flattened. Me, I’ll stick with “a great many,” since it is accurate enough for all my purposes.

So the takeaway for you? Well, there’s two. First, even with the seemingly most concrete of questions, definitions matter a lot. When someone gives you big numbers and the influence behavior, be sure to understand what they measured and how, and what decisions they made along the way. In information security, a great many people announce seemingly precise and often scary-sounding numbers that, on investigation, mean far different things than they seem to. (Or, more often, far less.)

And second, despite what I wrote above, it’s not the whole country that emerges. It’s the contiguous 48. Again, watch those definitions, especially for what’s not there.

Previously on Emergent Chaos: Steve Coast’s “Map of London” and “Map of Where Tourists Take Pictures.”

Your credit worthiness in 140 Characters or Less

Posted on by

In “Social networking: Your key to easy credit?,” Eric Sandberg writes:

In their quest to identify creditworthy customers, some are tapping into the information you and your friends reveal in the virtual stratosphere. Before calling the privacy police, though, understand how it’s really being used.

…
To be clear, creditors aren’t accessing the credit reports or scores of those in your social network, nor do those friends affect your personal credit rating. Jewitt asserts that the graphs aren’t being used to penalize borrowers or to find reasons to reject customers, but quite the opposite: “There is an immediate concern that it’s going to affect the ability to get a financial product. But it makes it more likely” that it will work in their favor,” says Jewitt. [vice president of business development of Rapleaf, a San Francisco, Calif., company specializing in social media monitoring.]

I’ll give Jewitt the benefit of the doubt here, and assume he’s sincere. But the issue isn’t will it make it more or less likely to get a loan. The issue is the rate that people will pay. If you think about it from the perspective of a smart banker, they want to segment their loans into slices of more and less likely to pay. The most profitable loans are the ones where people who are really likely to pay them back, but can be convinced that they must pay a higher rate.

The way the banking industry works this is through the emergent phenomenon of credit scores. If banks colluded to ensure you paid a higher rate, it would raise regulatory eyebrows. But since Fair Issac does that, all the bankers know that as your credit score falls, they can charge you more without violating rules against collusion.

Secretive and obscure criteria for differentiating people are a godsend, because most people don’t believe that it matters even when there’s evidence that it does.

Another way to ask this is, “if it’s really likely it will work in my favor, why is it so hard to find details about how it works? Wouldn’t RapLeaf’s customers be telling people about all the extra loans they’re handing out at great rates?”

I look forward to that story emerging.

2008 Breaches: More or More Reporting?

Posted on by

Dissent has some good coverage of an announcement from the ID Theft Resource Center, “ITRC: Breaches Blast ’07 Record:”

With slightly more than four months left to go for 2008, the Identity Theft Resource Center (ITRC) has sent out a press release saying that it has already compiled 449 breaches– more than its total for all of 2007.

As they note, the 449 is an underestimate of the actual number of reported breaches, due in part to ITRC’s system of reporting breaches that affect multiple businesses as one incident. This year we have seen a number of such incidents, including Administrative Systems, Inc., two BNY Mellon incidents, SunGard Higher Education, Colt Express Outsourcing, Willis, and the missing GE Money backup tape that reportedly affected 230 companies. Linda Foley, ITRC Founder, informs this site that contractor breaches represent 11% of the 449 breaches reported on their site this year.

I don’t have much to add, but I do have a question: are incidents up, or are more organizations deciding that a report is the right thing to do?

[Update: I wanted to point out interesting responses by Rich Mogull and Dissent.]

Laptops and border crossings

Posted on by

The New York Times has in an editorial, “The Government and Your Laptop” a plea for Congress to pass a law to ensure that laptops (along with phones, etc.) are not seized at borders without reasonable suspicion.

The have the interesting statistic that in a survey by the Association of Corporate Travel Executives, 7 of 100 respondents reported a laptop or other electronic device seized. Of course, this indicates a problem with metrics. It almost certainly does not mean a 7% seizure rate, as I’ve seen this inflated to. These seizures are such an outrageous thing that the people who have been subjected to them are properly and justifiably outraged. They’re not going to toss the survey in the trash.

I’m not sure how much I like the idea that Congress should pass a law to ensure that the fourth amendment is met. Part of me grits my teeth, as I think it should happen on its own. But if the courts aren’t going to agree, that probably has to happen.

Department of Justice on breach notice

Posted on by

data-breaches-carding-justice.jpg
There’s an important new report out from the Department of Justice, “Data Breaches: What the Underground World of “Carding” Reveals.” It’s an analysis of several cases and the trends in carding and the markets which exist. I want to focus in on one area, which is recommendations around breach notification:

Several bills now before Congress include a national notification standard. In addition to merely requiring notice of a security breach to law enforcement,200 it is also helpful if such laws require victim companies to notify law enforcement prior to mandatory customer notification. This provides law enforcement with the opportunity to delay customer notification if there is an ongoing criminal investigation and such
notification would impede the investigation. Finally, it is also helpful if such laws do not include thresholds for reporting to law enforcement even if certain thresholds – such as the number of customers affected or the likelihood of customer harm — are contained within customer notification requirements. Such thresholds are often premised on the large expense of notifications for the victim entity, the fear of desensitizing customers to breaches, and causing undue alarm in circumstances where customers are unlikely to suffer harm. These reasons have little applicability in the law enforcement setting, however, where notification (to law enforcement) is inexpensive, does not result in reporting fatigue, and allows for criminal investigations even where particular customers were not apparently harmed. (“Data Breaches: What the Underground World of “Carding” Reveals,” Kimberly Kiefer Peretti U.S. Department of Justice, Forthcoming in Volume 25 of the Santa Clara Computer and High Technology Journal, page 28.)

I think such reports should go not only to law enforcement, but to consumer protection agencies. Of course, this sets aside the question of “are these arguments meaningful,” and potentially costs us an ally in the fight for more and better data, but I’m willing to take small steps forward.

Regardless, it’s great to see that the Department of Justice is looking at this as something more than a flash in the pan. They see it as an opportunity to learn.

Security Prediction Markets: theory & practice

Posted on by

reckless-experimentation.jpgThere are a lot of great comments on the “Security Prediction Markets” post.

There’s a tremendous amount of theorizing going on here, and no one has any data. Why don’t we experiment and get some? What would it take to create a market in breach notification prediction?

Dan Guido said in a comment, “In security, SOMEONE knows the RIGHT answer. It is not indeterminate, the code is out there, your network is accessible to me and so on. There’s none of this wishy-washy risk stuff.”

I don’t think he’s actually right. Often times, no one knows the answer. Gathering it is expensive. Translating from “there’s a vuln” to “I can exploit it” isn’t always easy. For example, one of my co-workers tried exploit a (known, reported, not yet fixed) issue in an internal site via Sharepoint. Something in Sharepoint keeps munging his exploit code. I’ve even set my browser homepage to a page under his control. Who cares what I think, when we can experiment?

What would be involved in setting up an experiment? We’d need, in no particular order:

  • A web site with some market software. Is there a market for such sites? (There is! Inkling will let you run a 45 day pilot with up to 400 traders. There’s likely others.)
  • Terms & conditions. Some issues to be determined:
    1. Can you bet on your employer? Clients? Customers?
    2. Are bets anonymous?
    3. What’s the terms of the payoff? Are you betting company X has a breach of PII, or a vuln? Would Lazard count?
    4. What’s the term of a futures option? What’s the ideal for a quick experiment? What’s the ideal for an operational market?
    5. Are we taking singleton bets (Bank A will have a problem) or comparative (Bank A will have more problems than bank B.)
  • Participants. I think that’s pretty easy.
  • Dispute arbitration. What if someone claims that Amazon’s issue on Friday the 6th was a break-in? Amazon hasn’t yet said what happened.

So, we could debate like mad, or we could experiment. Michael Cloppert asked a good question. Let’s experiment and see what emerges.


Photo: “Better living…” by GallixSee media.

Security Prediction Markets?

Posted on by

In our first open thread, Michael Cloppert asked:

Considering the contributors to this blog often discuss security in
terms of economics, I’m curious what you (and any readers educated on
the topic) think about the utility of using prediction markets to forecast
compromises.

So I’m generally a big fan of markets. I think markets are, as Hayek pointed out, a great way to extract information from systems. The prediction markets function by rewarding those who can make better predictions. So would this work for security, and predicting compromises?

I don’t think so, despite being a huge fan of the value of the chaos that emerges from markets.

Allow me to explain. There are two reasons why it won’t work. Let’s take Alice and Bob, market speculators. Both work in banks. Alice thinks her bank has great security (“oh, those password rules!”). So she bets that her bank has a low likelihood of breach. Bob, in contrast, thinks his bank has rotten security (“oh, those password rules!”). So he bets against it. Perhaps their models are more sophisticated, and I’ll return to that point.


As Alice buys, the price breach futures in her bank rises. As Bob sells, the price of his futures falls. (Assuming fixed numbers of trades, and that they’re not working for the same bank.)

But what do Alice and Bob really know? How much experience does either have to make accurate assessments of their employers’ security? We don’t talk about security failures. We don’t learn from each other’s failures, and so failure strikes arbitrarily.

So I’m not sure who the skilled predictors would be who would make money by entering the market. Without such skilled predictors, or people with better information, the market can’t extract the information.

Now, there may be information which is purely negative which could be usefully extracted. I doubt it, absent baselines that Alice and Bob can use to objectively assess what they see.

There may well be more sophisticated models, where people with more or better information could bet. Setting aside ethical or professional standards, auditors of various sorts might be able to play the market.

I don’t know that there are enough of them to trade effectively. A thinly traded security doesn’t offer up as much information as one that’s being heavily traded.

So I’m skeptical.

Quantum Progress

Posted on by

quantum-computers.jpg

What is it about the word “quantum” that sucks the brains out of otherwise reasonable people? There has to be some sort of Heisenberg-Schödinger Credulity Principle that makes all the ideons in their brains go spin-up at the same time, and I’m quite sure that the Many Worlds Interpretation of it has the most merit. (In case you’re a QM n00b, the ideon is the quantum unit of belief.) Fortunately, there seems to be some sanity coming to reporting about quantum computing.

Just about every quantum computing article has a part in it that notes that there are quantum algorithms to break public crypto. The articles breathlessly explain that this means that SSL will be broken and the entire financial world will be in ruins, followed by the collapse of civilization as we know it. Otherwise sensible people focus on this because there’s very little to sink your teeth into in quantum computing otherwise. Even certified experts know that they don’t know what they don’t know.

Scott Aaronson has a good article in Scientific American called “The Limits of Quantum Computers” (only the preview is free, sorry) that gives a good description of what quantum computers can’t do. I’m pleased to see this. SciAm has been a HSCP-induced quantum cheerleader over the last few years.

I have been doing some research on the claims of quantum computing. I decided to pick the specific factoring ability of quantum computers, and produce some actual numbers about how we might expect quantum computing to develop. In other words, I’m going to be a party pooper.

The crypto-obviating algorithms in question are Shor’s algorithm for factoring and an algorithm he developed for discrete logs. I was surprised to learn that Shor’s algorithm requires 72k3 quantum gates to be able to factor a number k bits long. Cubed is a somewhat high power. So I decided to look at a 4096-bit RSA key, which is the largest that most current software supports — the crypto experts all say that if you want something stronger, you should shift to elliptic curve, and the US government is pushing this, too, with their “Suite B” algorithms.

To factor a 4096-bit number, you need 72*40963 or 4,947,802,324,992 quantum gates. Lets just round that up to an even 5 trillion. Five trillion is a big number. We’re only now getting to the point that we can put about that many normal bits on a disk drive. The first thing this tells me is that we aren’t going to wake up one day and find out that someone’s put that many q-gates on something you can buy from Fry’s from a white-box Taiwanese special.

A complication in my calculations is the relationship between quantum gates and quantum bits. For small numbers of qubits, you get about 200 qugates per qubit. But qubits are rum beasts. There are several major technologies that people are trying to tease qubits out of. There’s the adiabatic techlogies that D-Wave is trying. There are photon dots, and who knows how many semiconductor-based methods.

It isn’t clear that any of these have any legs. Read Scott Aaronson’s harumphing at D-Wave, more pointed yet sympathetic faint praise and these educated doubts on photonics. Interestingly, Aaronson says that adiabatic quantum computers like D-Wave need k11 gates rather than k3 gates, which pretty much knocks them out of viability at all, if that’s so.

But let’s just assume that they all work as advertised, today. My next observation is that probably looking at billions of q-bits to be able to get trillions of q-gates. My questions to people who know about the relationship between quantum gates and quantum bits yielded that the real experts don’t have a good answer, but that 200:1 ratio is more likely to go down than up. Intel’s two-billion transistor “Tukwila” chip comes out this year. Five trillion is a big number. We are as likely to need 25 billion qbits to factor that number as any other good guess. Wow.

The factoring that has been done on today’s quantum computers is of a four-bit number, 15. If you pay attention to quantum computing articles, you’ll note they always factor 15. There’s a reason for this. It’s of the form (2n-1) * ( 2n+1). In binary, 2n-1 is a string of all 1 bits. A number that is 2n+1 is a 1 bit followed by a string of 0s, and then a 1 again. These numbers are a special form that is easy to factor, and in the real world not going to occur in a public key.

This is not a criticism, it’s an observation. You have to walk before you can run, and you have to factor special forms before you can factor the general case. Having observed that, we’ll just ignore it and assume we can factor any four-bit number today.

Let’s presume that quantum computers advance in some exponential curve that resembles Moore’s Law. That is to say that there is going to be a doubling of quantum gates periodically, and we’ll call that period a “generation.” Moore’s specific observation about transistors had a generation every eighteen months.

The difference between factoring four bits and factoring 4096 bits is 30 generations. In other words, 72*43 * 230 = 72*40963. If we look at a generation of eighteen months, then quantum computers will be able to factor a 4096-bit number in 45 years, or on the Ides of March, 2053.

This means to me that my copy of PGP is still going to be safe to use for a while yet. Maybe I oughta get rid of the key I’ve been using for the last few years, but I knew that. I’m not stupid, merely lazy.

I went over to a site that will tell you how long a key you need to use, http://www.keylength.com/. Keylength.com uses estimates made by serious cryptographers for the life of keys. They make some reasonable assumptions and perhaps one slightly-unreasonable assumption: that Moore’s Law will continue indefinitely. If we check there for how long a 4096-bit key will be good for, the conservative estimate is (drum roll, please) — the year 2060.

I’m still struck by how close those dates are. It suggests to me that if quantum computers continue at a rate that semiconductors do, they’ll do little more than continue the pace of technological advancement we’ve seen for the past handful of decades. That’s no mean feat — in 2053, I doubt we’re going to see Intel trumpeting its 45 picometer process (which is what we should see after 30 generations).

I spoke to one of my cryptographer friends and outlined this argument to him. He said that he thinks that the pace of advancement will pick up and be faster than a generation every eighteen months. Sure. I understand that, myself. The pace of advancement in storage has been a generation every year, and in flash memory it’s closer to every nine months. It’s perfectly conceivable that quantum computing will see horrible progress for the next decade and then whoosh off with a generation ever six months. That would compress my 45 years into 25, which is a huge improvement but still no reason to go begging ECRYPT for more conferences.

On the other hand, it’s just as conceivable that quantum computing will end up on the Island of Misfit Technologies, along with flying cars, personal jetpacks, Moon colonies, artificial intelligence, and identity management.

But I also talked to a bigwig in Quantum Information Theory (that’s quantum computing and more) and gave him a sketch of my argument. I heard him speak about Quantum Information and he gave the usual Oooooo Scary Quantum Computers Are Going to Factor Numbers Which Will Cause The Collapse of All Financial Markets And Then We Will All DIEEEEE — So That’s Why We Need More Research Money boosterism.

He wouldn’t let me attribute anything to him, which I understand completely. We live in a world in which partisanship is necessary and if he were seen putting down the pompoms, he’d be fired. Telling middle-aged technocrats that the math says their grandkids are going to see quantum computers shortly before they retire will cause the research money dry up, and if that happens then — well, the world won’t end. And then where would we be?

Nonetheless, he said to me sotto voce, “There’s nothing wrong with your math.”

Photo is a detail from “Shop Full” by garryw16.

Are We Measuring the Right Things?

Posted on by

measuring-progress-gao-8496t.jpg

One of the reasons that airline passengers sit on the tarmac for hours before takeoff is how the FAA Department of Transportation measures “on time departures.” The on time departure is measured by push-back from the gate, not wheels leaving the tarmac. (Airlines argue that the former is in their control.) If you measure the wrong things, you create incentives for bizarre behavior.

Which is why I was fascinated to read the new GAO report, “Information Security: Although Progress Reported, Federal Agencies Need to Resolve Significant Deficiencies.”

While progress may be reported, PogoWasRight calls out:

The number of security breaches on government computers has quadrupled in the last 2 years – from just over 3,500 in fiscal 2005 to just over 13,000 in fiscal 2007.

If that’s progress, maybe we need some regression?

More seriously, I think it’s great progress that we are talking about the failure rates. Now we need to start to question the things being measured that allow the GAO to summarize that state of affairs as progress.

I wonder, where else are we measuring the wrong things?

[Update: I was measuring the wrong agency.]

Measuring the Wrong Stuff

Posted on by

There’s a great deal of discussion out there about security metrics. There’s a belief that better measurement will improve things. And while I don’t disagree, there are substantial risks from measuring the wrong things:

Because the grades are based largely on improvement, not simply meeting state standards, some high-performing schools received low grades. The Clove Valley School in Staten Island, for instance, received an F, although 86.5 percent of the students at the school met state standards in reading on the 2007 tests.

On the opposite end of the spectrum, some schools that had a small number of students reaching state standards on tests received grades that any child would be thrilled to take home. At the East Village Community School, for example, 60 percent of the students met state standards in reading, but the school received an A, largely because of the improvement it showed over 2006, when 46.3 percent of its students met state standards. (The New York Times, “50 Public Schools Fail Under New Rating System”

Get that? The school that flunked has more students meeting state standards than the school that got an A.

There’s two important takeaways. First, if you’re reading “scorecards” from somewhere, make sure you understand the nitty gritty details. Second, if you’re designing metrics, consider what perverse incentives and results you may be getting. For example, if I were a school principal today, every other year I’d forbid teachers from mentioning the test. That year’s students would do awfully, and then I’d have an easy time improving next year.