Beau Woods and Seth Carmody dig into CVSS, the Common Vulnerability Scoring System. It is useful as a shared taxonomy. It fails, badly, when it is used as a risk prioritization tool, especially for medical devices. This conversation covers why, what the FDA actually expects instead, and where the industry is heading.
Guest: Seth Carmody, VP Regulatory Strategy at Medcrypt. Former FDA cybersecurity lead and architect of the postmarket cybersecurity guidance. Connect with Seth on LinkedIn.
Transcript
Beau: All right, welcome to today’s conversation on behind the FDA cybersecurity guidance with myself, Beau Woods, and Seth Carmody of Medcrypt. We are very excited to be here this morning. we’re we’re all full of rants and giggles, so this might be an entertaining one for the viewers out there today. and hopefully it’ll give us some good fodder for conversation and and Bring some things to light that might not have if we were less caffeinated. Seth, how’s it going?
Seth: I’m doing excellent. Mostly because when it comes to dealing with models that are imperfect, I’m I’m just like, yeah, well, you know, all models are wrong, some are useful and I guess some piss people off. So CVSS is such a model, it was broken when it started, it’s still broken in ways, and every time somebody goes to replace it, it looks an awful lot like CVSS. So I guess some when they built that model, they got some things right. but it sounds like It’s increasingly causing consternation and and l less fit for purpose than ever. And I think I think you have something to say about that.
Beau: Yeah. Wow, you’re really you’re really leading me into that.
Seth: You’re not the only one. I don’t know what it is about the moment, but I’ve been fielding calls and complaints about CVSS, which is weird because you would think that I started it, you know, but I did not, and that I have control over it. Which I you might. I don’t know. We’ll we’ll figure out.
Beau: Yeah. So CVSS, it’s been around many, many years. I think like two thousand three, two thousand five, something like that. It’s really, really old. And the the problem that it was designed to solve is we didn’t have a common way to talk about specific vulnerabilities. It’s the same type of a problem, same shape of a problem that we had that that spawned CVE, the the common vulnerabilities enumeration. common vulnerabilities. I always forget what the the CVE stands for and the CWE but
Seth: Keep keep talking, I’m gonna look it up. I’ll be your like reference man. Go ahead.
Beau: Okay, nice. So CVSS started as a way to basically describe the characteristics of a vulnerability. and then they attached a a a scoring system to it, zero through ten, and no one seems to know. I’ve done a bunch of research on this, no one seems to know why 10 was the maximum, zero was the minimum. It’s just a fairly arbitrary scale. okay, fine. You gotta have a scale. If you’re gonna have a scoring system, crocodile to giraffe was not a valid one, not one that got considered, I don’t think, in the deliberations, but zero to ten, fine. You have an arbitrary scale. Then you go and rate things arbitrarily. and that is okay. They they CVSS and the organization that maintains it called FIRST describe it as a descriptor of technical severity. So it’s it’s all about the vulnerability itself, not at all in the base scoring system, about the context, the threats, the environment that it’s in, none of that. It’s it’s just this technical bug in whatever it is of configuration or software code or a certificate authority or whatever. it’s all about that. And so I don’t have a problem necessarily with CVSS in its design and conception. However, people who don’t bother to read the 300 page document that surrounds it or however long it is and all of the history and backstory of it, look at a zero to ten and say, Ooh, that’s a score. It must have a lot of rigor around it, and a three must be half as bad as a six. And, you know, an An eight point eight must be ten percent more risky than an eight. And it’s not about risk at all. in fact, this you know, the main determinant of risk and in whether or not you’re actually going to get popped is is anybody going to actually exploit this? And again, being a descriptor of the technical vulnerability itself cannot tell you adversary’s intent, it cannot tell you adversaries’ capability, it cannot tell you whether or not there’s tools. That exists to make something that seems almost impossible actually routine in day-to-day usage. And so it it is by design silent on those things. Now, over the years, they have updated it and and added some things to it to try and compensate for that. they’ve got the what is it, the threat scoring and the environmental scoring, and there are these optional things you can add onto it. But very few people do. and it’s it’s a bolt-on to the original construct itself rather than something that was originally part of the intent. and so it’s just it is not fit for purpose. There’s nothing in there about in the medical context, there’s nothing in there about patient safety. Does not have anything to do with clinical performance. It doesn’t consider a lot of the things that you would want. from a risk scoring system, or risk prioritization system for medical devices and for public health. and yet it’s thrown around as if it’s going to fix all the problems. or, well, at least it’s something. No, it’s the it’s deceptive. It’s wrong. And it is actively harmful. which is why, like you said, Seth, so many people are moving away from it. So many people are trying to find something better. And in fact, the FDA expects that you will have something better. And it’s called a risk assessment methodology. Like we already know how to solve this problem. You just don’t want to do the work. You want somebody else to do the work. You want
Seth: That’s a good time.
Beau: an arbitrary zero through 10 score to tell you the decision you need to make. But really these are can be really difficult decisions that you you have to have additional context. You have to get additional information from in order to actually know what it’s going to do. it’s not a, if it’s above an eight point zero on the scale, then automatically you do something. And if it’s not, you pretend it doesn’t exist. that’s just not a a good method. And I, I don’t know anybody who who uses that method, to be clear in in healthcare, but I’ve seen people in other industries do it that way. and it’s it it’s it’s just a poor fit for purpose. I’ve got a whole bunch more. There’s probably
Seth: Yeah.
Beau: gonna be at some point I’ll probably write a a big long rap, but let me see if I can recap that so it’s it’s not quite so caffeine bingey when when people go to read it.
Seth: good TV. Beau, it’s good TV. It’s like I’m picturing some sort of terrible technical reality show where people get into a room and it’s like technical standards people arguing over CVSS. Yeah, some fee
Beau: god.
Seth: few some will watch it. I mean it’s it’s watchable for some.
Beau: That is nightmare fuel. I can only imagine the type of people who would sit down and be like, all right, yeah, this is a this is a great thing for us to spend hours and hours
Seth: Okay.
Beau: and weeks debating.
Seth: Yeah, I have thoughts. Go ahead. Sum
Beau: So
Seth: up.
Beau: here’s my my main sequence of thinking on on CVSS. CVSS is a poor fit-for-purpose for healthcare and for medical device risk. CVSS is and always has been a descriptor of technical severity about the vulnerability itself. It is not a risk measure, expressly not a risk measure. One of the largest determinants of risk comes from whether or not. the vulnerability will be exercised or exploited. And CVSS base score is silent on that. It is also unreliable and imprecise. So different scorers will assign different scores to the same vulnerability. Over time, the score will change as as tools. Become available to reduce the difficulty of exploitability as That’s not even in CVSS. The one thing I was trying to give it credit for is not even a part of the base.
Seth: Do you think do you think that it has elements that should be paid attention to, right? Like attack vector. Should attack vector come into consideration when considering the technical severity and as a part of the consideration of overall risk it factors in?
Beau: Yeah, I I do think that as a descriptor of the characteristics and traits of the vulnerability itself, there’s a lot of value in CVSS. As a common taxonomy, a way for us to describe vulnerabilities, [CVSS is] very useful and very helpful. as it tends to be used as a way to prioritize risk or threat or patch it, whether to patch something. I think it is a A failure. It is is poor fit for purpose for that. There’s so much that gets left out of the equation when just looking at CVSS, and especially just looking at the CVSS base score, which is all that can reliably be published somewhere like National Vulnerability Database, because the other things change. The you know, the the threat score portion, the optional threat score portion changes with availability of knowledge of techniques, tools, of adversary willingness to use those things, the environmental portion of the optional add-on to CVSS changes in the specific deployment context, not just in the medical device, but how the medical device is deployed in the hospital or care provider environment, whether at home, you know, if it’s a wearable, whether it’s on you, what environment you’re in. and so it’s it is useful for what it is, not what people want it to be.
Seth: That’s probably the best succinct descriptor that I’ve heard, right? I think the tension that is felt and I felt since you know we started thinking about what could we introduce into policy, which ultimately was the post-market guidance to try to infuse elements of security, things that have been already, and I I use this with it’s doing a lot of work, worked out in the security community that we could give to the medical community to help them on the beginning journey of essentially vulnerability management. So if I take us back to we’re talking 2014, right? Which is weird to say I have a van that’s the year 2014, you know, 2014, it’s showing showing your age. but a lovely Honda Odyssey. just beautiful machine. definitely time has passed for sure. So that postmarket policy obviously needs an update. I just saw the FDA release their priority guidances for 2027. I believe post-market was on there for draft. it’ll be very interesting to see what they come out with. Anyway, I digress. there’s a graph a graphic in the postmarket guidance that basically says exploitability now and then safety on exploitability on the y-axis and then safety on the the x-axis. And that forms the basis of a post-market vulnerability management program, which in and of itself I think is the right direction, right? Hey, you’re considering failure modes all the time for your medical device. We now have to connect those failure modes to adversarial action.
Beau: Yeah. Yeah. I I I would love I think that that single graphic is one of the most useful things out of the post market guidance. And there’s a lot of useful things in the post market guidance, to be clear.
Seth: Hey, you don’t have to you know, I I it’s it’s it was twenty four vintage twenty fourteen. Works for wine, maybe not for college.
Beau: It is it is vintage twenty fourteen, but I think that that way of thinking about risk in a clinical health context is very, very valuable. having that graphic, the way that it’s drawn there, also brings people towards it so that it it gains a lot of prominence, which I think it deserves. and I I’ve seen that be a helpful guidepost for a lot of manufacturers when they go to think about should we do anything? And if so, what should we do to address this particular issue? Because it it is a without being a deterministic, you know, turn off your brain, just apply this one simple measure buzzfeed headline, it gives manufacturers a way to intervene in a vulnerability in a way that shifts the risk to a place where they’re they’re comfortable with that living in the overall clinical environment. And I think that that that’s an incredibly useful tool for manufacturers.
Seth: Definitely needs a revamp for sure. And we’ve been seeing, and I there’s the the percolation on CVSS is also a percolation on safety risk as well. and what we’re seeing is that existing the existing methodologies and and I think you know TIR 57 is the is the you know security risk management and then safety risk management and there are crossover points. I feel like that was a useful model, but it’s feeling quite dated and strange, strained. I have not talked to any manufacturer that is comfortable or has a perfect and flawless security to safety transfer. They’re all kinda a little clunky, and I’m really curious why that is and and I don’t anxious is not the right word, excited to dig in and figure that out. But I I do think that, you know, kinda The the point I I think I was driving towards that I didn’t realize that I’m realizing now is it was a crawl moment, right? We had this we had no notion of this in medical, as far as I understood. I’m coming on the scene at 2011. it was introd introduced from stuff that we saw. Manufacturers had no notion of vulnerabilities. In fact, vulnerabilities were equated with quality defects. And I’m not kidding you, they were shameful and and pe and also economically disadvantageous to even find. Cause qu if you had a quality defect in your device that was a problem in fact you didn’t get bonuses if you had a if you had to release if you had to do a field corrective action people’s livelihoods and you know their the their livelihoods were tied to that performance because quality defects have been systematized to be found and rooted out and cat like CAPA and like there are all these mechanisms to find it and vulnerabilities were this sort of this a similar flavor or vibe So the entire postmarket guidance, and in fact, I was just out at Boston TT Boston last week, and somebody asked me like what’s gonna be in the new postmarket. And I’m like, man, I hope they don’t change it too much because then I can’t get up there and say, I was the architect of it. but one of the things in there is that incentive structure to find vulnerabilities and do something with them. And
Beau: Yeah, for sure.
Seth: and what was interesting to talk about was hey, hey, this is a this is an experiment. We did we had no idea how that would work, but we knew the problem set was we need to find vulnerabilities and have a way of dealing with them. Right. So very very much a that graphic w was definitely a crawl moment, a crawl policy. And I think what we’re feeling is we’ve we’ve outgrown that outgrown that box, that policy and that thought pattern. And that’s natural.
Beau: Yeah.
Seth: I think I I think that’s actually a sign of healthy growth. It’s a sign of progress.
Beau: It is. Yeah, absolutely. anytime you can you can put something out there as a a starting point and then you come back around to it a dozen years later and saying, Yeah, it’s starting to get a little bit old now. we need to improve on it. that’s fantastic. And I think that there’s a lot of things in the post market guidance from twenty fourteen that were at the time groundbreaking and are still even today. not things that other regulators are doing and to their own detriment. you must have a policy in order to get some kind of backstop against having an automatic recall for some things. you have to have a coordinated vulnerability disclosure program. You have to have a vulnerability management program post-market. you have to participate in some of these information sharing organizations and groups where you’re actually getting information about new vulnerabilities. You have to have a way, an internal process to deal with those things. You have to have a a 30 and 60 day response to the the reports and the vulnerabilities. Like those, I think at least as much as anything else in there, catapulted medical device manufacturers, you know, the ones who follow it. into a much improved stance because in order to do those things, many of those things, you have to Restructure how you think about vulnerabilities themselves as not a quality defect, but a natural output of the software development process. And I remember having conversations at the FDA with folks who were, you know, died in the wool quality people. They’re like, I just can’t, you know, you should never ship a product with a bug. I’m like, well, you can’t eliminate all bugs. So then you should never ship a product. They’re like, wait, what do you mean?
Seth: Yeah.
Beau: And it was this clash of ideologies and mindsets. And we had to work through it. And I think that that was incredibly valuable for the folks there as well as for the medical device makers who benefited from having those types of conversations, where all of a sudden, when you have those two worlds collide of quality and cybersecurity, you’ve got to find a better way forward, not a middle ground. So to speak. it’s, you know, how do you apply quality to software? How do you apply quality to cybersecurity issues? and think about it that way. and I think that that again, the 2014 post-market guidance set the preconditions where it was more advantageous for medical device makers to accept that vulnerabilities existed. and build their operational processes as well as their technology to be able to adapt to that more agily than they had in the past.
Seth: Yeah, one of the it’s interesting that you’re picking up on that thread. Cause in some ways the post-market was more advanced than the pre-market, right? Because the pre-market came out in 2014. It was six pages long. It was built under duress, like not a lot of contemplation that went into it. Now the folks that built it, now it was sort of on the periphery. at the time the folks that built it had been thinking about it a long time folks that we’ve talked about before Brian Fitzgerald John Murray, folks that have been involved in software at the at the agency for a long time. and it’s what what what and I it’s probably lost in the 50 some odd pages that is now the pre-market. But the the best line the best the best cohesive force inside that diet, the the TLDR of the of the pre-market and it it’s progeny or it’s a progenitor is in the postmarket is it’s the secure development cyc life cycle. And what’s the objective of the secure device development lifecycle, which I think I coined secure product development framework? I wish I had a recording of me working on figuring out that name. I have there are absolute reasons that we could go into that are probably very dull and ex but exciting for policy wonks. But the notion of having a set of processes to find vulnerabilities and reduce their severity or reduce their number is the name of the game. That whole document and everything that we’re trying to do here is exactly that. It’s almost like in the complexity, we have to be able to go back to those framers. OWASP and the four questions of threat modeling. Threat modeling can get super hairy very quickly. Okay, what are we doing here? Okay, we’re figuring out what we’re building. okay, cool. well, what are we doing here? Why are we doing this? and I think those are that’s super useful. I do I do want to pull us back into CVSS because I I do think that’s a it’s a a to a topic that’s rich. I wanna just w I I got a list of scoring systems. I’m just gonna read them off. I don’t know if that that’s gonna be effective.
Beau: Go for it. Yeah.
Seth: TV. so let me just let me just do this. I’m gonna switch over my screen here. So we got the Singaporean method. it’s I I think it was released late a version of it was released late this year or in August of this year. But I know Singapore I think is using some combination of NIST 800-30 risk management framework, and I think which is sort and then a dread model. we’ve got CVSS. Obviously we’ve got a a histor what do you want to call it? Some people are still using two, some using three, we’ve got four in the mix. we’ve got permutations. I don’t know if people remember or know Chris Gates. but he devised what was called the ECVSS scoring system, which was I think for his take on embedded structures. I know Oleg Yusim at he was at Edwards at the time, now at Illumina, had the HHVSS or healthcare vulnerability scoring system. we have the MITRE rubric for applying CVSS to medical devices, which which in and of itself is a you know based on CVSS but takes a different approach and I’d love to dig into that while we’re here on the on on the call today. We have the exploit prediction scoring system. which was again, I think all of these were tension felt with CVSS and we gotta do something different. And of course, I would would be remiss if I didn’t mention our folks, well-established colleagues at Carnegie Mellon Software Engine Engineering Institute with the tongue-in-cheek name of a stakeholder-specific vulnerability categorization or SSVC, the opposite reverse of CVSS. let’s not forget our boy Billy Rios with his risk scoring system. and then there’s another one, Abed Graham. This is I forget where Abed is now, but he had a scoring system. I think he took it down off the website, also feeling the tension and trying to Get something closer to what, relieve that tension. And I’m gonna go on. Common misuse scoring system, common configuration scoring system, industrial vulnerability scoring system, the OWASP risk framework. And then there’s a bunch of things out of SANS, and then University of Trento, Italy talking about quantitative assessment of risk reduction for cybercrime. We are living the XKCD comic right now. Somebody should
Beau: Yes. Yeah.
Seth: put together a better scoring system, and we end up with another scoring system that is in it in and of itself looks a lot and feels a lot like CVSS and is imperfect in all the ways that CVSS is imperfect. So I I don’t know, I don’t know what what that bodes for the future.
Beau: Yeah, clearly this is a hard problem. So we should pick the most popular one that has the most name brand recognition. I don’t I don’t know how that even
Seth: Yeah.
Beau: enters somebody’s mind instead of let’s find something that’s fit for purpose. But apparently that’s a thing that standards organizations wanna do.
Seth: if you want more exciting tea I got a phone book I could read from around here.
Beau: Ha ha ha.
Seth: Do those even exist? Do they get published anymore?
Beau: Yeah. You you and Tracy Morgan reading the phone book,
Seth: Didn’t you win like a Tony for that or something in the in the show?
Beau: Yeah, yeah, in the in the fictional thirty rock show, I think you won a Tony for reading the phone book.
Seth: Just gotta sell it, man. Just gotta sell it. I I do think it’s it it’s interesting and the way the CVE it’s this is why we get mixed up, CVE’s and C P E’s very, you know, they’re aff affiliated, right? They’re associated. So C VE’s exposure.
Beau: Yes. Yes. The continuing professional education that you need for your certificates, right? That’s the the acronym that we’re talking about.
Seth: yes, of course. CVE exposure. The E is exposure. CPE enumeration and super easy to get mixed up. So I wanted to clear that up for the the audience at home. I do think I I I wanted to get back to the tension a little bit, Beau. I think there’s a there’s a couple things working against us. One, I think even if even if you are a subject matter expert and you feel comfortable in vulnerability management and dealing with the attributes of vulnerability management and comfortable in diagnosing the diagnosing the severity of vulnerabilities and potentially resulting risk of that vulnerability that simple volume simple volume the volume of of vulnerabilities is is far exceeds our ability to precisely evaluate them. I do think that is changing or could change, which is exciting. and I I do think there is an and I I explored this a little bit in 2020. I I think risk meth risk evaluation methodologies like ISO 14971 are actually What do they call it? trades tractability for fidelity. They’re actually oversimplifications. And if you we look at some of Nancy Leveson’s work coming out of Therac-25 and resulting safety, safety critical risk assessments, you know, I’m I’m stealing from her like her conclusions here and her work, right? That it’s like, well, if we look at the Columbia space shuttle disaster, safety calculation and risk calculation are far more complex. then we’ve we’ve given credit to. So I I think there’s this, but there’s always this push and this tent this feeling that as humans we want a model that we understand. ISO 14971 is is a model that we can all understand. And that’s what we based the security risk model on. But it’s trading some of that understanding and I get it to it it’s not quite fit for purpose. And I think that’s probably some of the underlying tension as well. Curious what you think about that.
Beau: Yeah, I first I’m a very, very big fan of Nancy Leveson’s work. I think it’s it’s incredibly clearly worded, understandable, and gets to like the real root of a lot of these issues. So like if we could replace 14971 with with Nancy Leveson 101, I think that we’d all be safer. I also think that you’re right. It is a a very, very difficult topic. it’s a one with a lot of different considerations to bring in. And that can be overwhelming, right? Just from a psychological standpoint. You look at all of these things, you have to think about and you’re like, that’s too much. I’m you know, I I am a domain expert in organic chemistry. And now I have to start factoring for, you know, all of these other things that I don’t really understand. And if I get one wrong, there’s a safety implication. I want something simple, clear, understandable. And I I think that that is a reasonable request to have. I don’t want to turn everybody into a cybersecurity expert because I think that there’s not enough room in anybody’s head for all the the cybersecurity stuff that you have to know. even even mine and yours is as big as our heads are. we can’t we can’t contain all the knowledge that exists in cybersecurity. nor all the knowledge that ever will exist in cybersecurity. And so it’s an impossibility. Things like frameworks really can help to distill that down to something that is, like you said, tractable. You can you can work the problem to a solution with an equation. Even if that equation is wrong, you get to the end of the problem and you feel satisfied that you got to the end of the problem. I think we’re things like CVSS get it wrong is it was never designed for the problem that we’ve been putting it towards. So, you know, you’re using an equation that is the type of answer you get, gives you something different than what you’re seeking. And it’s that misuse of those tools and those equations and those tractable solutions that’s what starts getting us into trouble. But again, if you’re if you’re coming from the outside, if you’re not a if you don’t know the technology, you don’t know the reasons behind C of ASS, you don’t know what it was built for or any of these scoring systems, you don’t understand the implications of that, it’s really, really easy to inadvertently misuse a tool.
Seth: Yeah, and I I I do think that from a standards perspective, I I wouldn’t say standard I it’s more s looks more standards than not, but I do think the FDA also could could help by clarifying w how. it should be done. and I think that’s a good position for them. That’s a good thing, a good problem for them to work on is to try to take the crow, you know, take the crawl strategy, crawl methodology, to the walk methodology. and you know, I I point to them, but I I also feel compelled to do it. I don’t know w why. but also in a position to do it, right? Because we’re helping customers do risk scoring all day every day and you can see the problems that are latent in it, right? So how how I think the experiment we have the data that we can do the experiments with, right? And say, yeah, we can we can run it.
Beau: Yeah. And I think i if you look back to I I was just recently looking at this. It’s not that I have an exact encyclopedic memory, but in some of the early guidance documents, pre market post market, they referenced CVSS. and then they started not referencing CVSS or not referencing it as much. And even when they did reference CVSS, it was here is a way to describe one aspect of exploitability which we care about. not you should use this to make risk decisions, right? And I think they even said like this isn’t a risk acceptance methodology, right? Which is absolutely the case. Right. Yeah. Right. and I I
Seth: Because that’s from first because that’s what first said. So when I read the spec, it was like, Yeah, this sounds important.
Beau: think you’re right. I think that they’ve may be reluctant to To step in where standards bodies and others are trying to to fill the gap, but they’re not doing it well enough. And so I think that there is a gap into which a better scoring mechanism could step. I just I’m not sure if there is one that exists, and maybe you know of one, Seth.
Seth: Yeah, I don’t. so I I think and I I think we we were in the the pre-meeting talking about how we have developed a Medcrypt recommended scoring system. without going into too too much detail, it’s trying to pull out the useful parameters and discard the others and combine that with Essentially device information to do this in an automated way. and that is supposed to get at a couple of the problems. One, the volume problem that we were talking about, right? Like it it’s consumptive and not useful to have to tick boxes in a spreadsheet. It is helpful to have a wrong answer written, and then somebody can say, let’s tweak that answer in the following ways. It’s just from a burden standpoint, even if automations are wrong, if they’re 75% correct, we can get to 95% correct with less effort. second part is like you talk, we went back to the math problem, right? And it sounds like a commercial, but it’s not a commercial. And just talking about the like what works inside cvss one of the is exploitability subscore like what what are the fact what are the contributing factors the attack factors like what is that score and forget that it’s what is it ordinal math, right? And you’re not supposed to add ordinal math. They take the exploitability subscore and then the impact subscore and they add them together. And that feels wrong to most people that are that are skilled in and and understand CVSS to its to its core. So we just take that out, essentially reject the impact score, take the exploitability and then combine that with device information that then allows you to actually combine an impact of the device or on the device. That’s roughly speaking what we tried. Now, there’s two traps, at least two traps. One, it looks a lot like CVSS. Right? So I love your phrasing the problem is maybe there isn’t a solution. And that’s probably why when people try to to solve it, they end up with something different but similar and flawed. so anyway, I I the I’ll I’ll close my I’ll close with what we’re working on. And I think we live in a time where we can quickly run a ton of experiments and actually figure out what will work and and get to if there is an answer or a model that’s closer to fidelity, we can find that faster and cheaper than we’ve ever been able to find do it before. And that’s what’s really exciting.
Beau: Yeah. Well I’m I’m excited to look over that and to to read more about it. Not that anything is better than CVSS, but anything is better than CVSS. and again, it’s it’s not that CVSS is bad or wrong, it’s that it gets misused. And I think that that that is a fair criticism of its application without criticizing the method itself. I think it still works, you know, it was essentially built for banks and retailers to be able to have some way to say, hey, this is bad. Here’s the taxonomy of how it is bad. The score gives a rough approximation. You know, it’s what close only counts in in horseshoes, hand grenades and and CVSS scores.
Seth: Yeah.
Beau: it’s a rough approximation of how bad it is, but don’t rely on that number because it’s not repeatable and Different people will come up with slightly different numbers. but it tells you at least that like it is worse than a different one or better than a different one that is, you know, very far apart from it. and I think that having those types of descriptors again are excellent for understanding the risk of the thing to plug into a broader equation. not to substitute for decision making.
Seth: Yeah, it’s interesting. because we’re talking about the history and behind the scenes at FDA, I wanted to based on your what you just said there, wanna talk a little bit about the mitre rubric for using CVSS because I think it’s a
Beau: Let’s dig it.
Seth: it’s a I I w I wanna say much maligned, I’ll just say maligned document and unused. and it’s unfortunate because I actually think it’s useful. It’s not used in a non-automated world. So let me dig into that. So where did it come from? so Back in I want to say 2016 or 17, realizing that CVSS, that two different people could use it and get two different scores, what what was a way to to constrain the methodology such that you could get some normalization over scores? And two studies were conducted into what we’ll call advanced culturally manufacturers. meaning that they allowed the FDA and MITRE to go into their four walls and sit in CVSS scoring vulnerability, vulnerability management sessions, taking lists of CVEs, sitting with their teams, listening to how they were discussing them, writing down what the conversations consisted of so that we could add additional constraints and considerations that needed to be selected, right? Essentially, to constrain and and focus the conversation. And I’ll give an example, like one of the things that co came up a lot, and it and it’s true is like when you talk about, well, if something’s wirelessly accessible, what’s the range? Right? And and and and there’s generally no good answer because Billy Rios will say, Well, I I can build n if I can build I’m I’m limited by the antenna I build, right? So and the power I can put through it.
Beau: Right. Yes.
Seth: it’s like okay, all right. But generally speaking, where does Bluetooth top out? Like so like it given that, okay, is it 50 feet? And so when you ask a question when you’re talking about attack vector, it’s like if you’re talking about Bluetooth as an interface or you’re talking about NFC as an interface, here are the considerations for for their you know wireless range with all those given caveats. It’s just a way to be like, we’re not talking about a hundred feet, we’re talking about ten feet. Okay. And I think that kind of thing helps when making selections in CVSS, which was essentially what it tried to do with some extra stuff on top, that you could more consistently get Gets two people to score similarly. And I believe the outcome of that study was that that was true, and that’s reflected in the content of of the rubric. Here’s the problem. that that additional, I think there’s and I I haven’t counted, but there’s like I don’t know, let’s say 15 additional parameters that need to be selected. Well, then we already talk about a vulner a a vol volume problem. people can’t handle the work they have. You’re giving me more work to do. Right. So I I I think it was an advanced, helpful but not useful. And it’s actually start from for me, I’m starting to think about it again. Like, this we might be able to implement some of the concepts in here with with automation. So really excited about revisiting that as well.
Beau: Yeah. and you say very few people few people use it, but there’s no,
Seth: I don’t know that. Maybe people love it. I don’t know. I don’t think I don’t think so.
Beau: there there’s vendors that sell like we use an MDDT approved tool to test your network for whatever. And and they’re relying on that and saying that because they lean on a what twenty twenty-one document that’s based on a twenty nineteen CVSS. Scoring mechanism that has completely gone away, that you should trust them more than you should trust somebody who doesn’t lean on, doesn’t use MDDP approved scoring mechanism. it it’s a it’s a series of logical leaps that are trying to to do parkour,
Seth: Ha ha.
Beau: basically mental parkour to get to a a point where you should think that their stuff is better. But you know, to that point, like the MITRE rubric, its intent was to to address an issue, like you said, which is good, which is like connecting CVSS, the predominant scoring mechanism and vulnerability description mechanism, to patient safety, clinical effectiveness, you know, health and public health. But I don’t think it’s been updated. it it was built on three point zero, CVSS three point z and this is gonna get a little geeky, but CVSS three point z was superseded by CVSS three point one. I think by the time the rubric was even out, there was a new CVSS, which took a slightly different method. And now there’s CVSS four point that changes a lot of things about how CVSS operates. So Even by the standards of, is this the best thinking of the CVSS group? it it fails that, which is again, it’s not a failure of the rubric as a point in time methodology, but it is it was also not exactly fit for purpose if part of the purpose was to ease the burden of of figuring out whether or not you should do something. and it is even behind on the base that it was stemmed from. so I I think my my takeaway from that is that it is probably past time to have a conversation about what is a better fit for purpose. You know, is there something that actually matches risk to health and public health that could be implemented even if there are necessarily gaps in it. recognizing those gaps and acknowledging that they exist, rather than trying to fill them in with something that’s inferior but will get overused and overrelied.
Seth: Yeah, I’d and I’m excited. I I think you know getting we are in an era where you know I was just having this conversation, I think it was at Amy Nexus, and I was speaking to a gentleman there, I’m forgetting the name of it, when we were talking about sort of Amy standard efforts. And he’s let he was very astutely thinking, how long does it take us or any standard body to put out a standard? And I’ll I’ll lump the mitre rubric in there with a standard because it takes effort to go back and revise and be make good decisions and choices and write write it up and get approved, get it reviewed and peer-reviewed and out. And he’s like, I things are moving so fast, I feel like that model is dinosaur. And I’m like, I think, I think it’s time to revamp. How we think about standards. And I I don’t know what that is, but I I I postulated, I was like, well, what if, right? Like what do you want from a standard, right? You want a standard implemented, right? a lot of standards you know, take considerable expertise and like time for people to be to understand how to do it, but That’s an old model. And I feel like you know, a lot of the AI revolution is sort of posed like, does it actually have to work that way? So I was like, well, what if Amy just shipped us, you know, like, vendors their whatever their digital requirements were? Right. And they worked on it run a required amount of time. hey, let’s take a look at this requirement, update it, and push it. Almost like Git. We pull it down. And then we’re helping clients against that that requirement today. it doesn’t have to be done in one monolithic you know, approach. And printed through paper and then download for people to read and digest. It’s like why don’t we just systematize it? You know what I mean? And I think that helps implement it, that helps get distribution, that helps break the prob you know, instead of going back and revising an entire standard document. Just, you know, work on a a a block at a time, something like that. And I was
Beau: Yeah, I think that there are different approaches, standards, frameworks, rubrics, like there’s different names for things. at the end of the day, they’re all an attempt at consistent governance and common communication. They they ease one of the best things that comes out of standards is that they ease transactional friction that can be a technical protocol transaction, Wi-Fi communicating across, you know, from a base station to an a laptop. It could be a procurement transaction. If you have these things, then we automatically assume that you’re good and we’ll let you into our sales channel. they can be other things, but they they ease transactional friction. I think that over relying on them To actually manage risk is a mistake. because look, it’s the same thing with the the vendor floor at Black Hat and RSA. Standards are basically the vendor floor at Black Hat and RSA, right? Day one, you spot an issue that needs to be addressed. You do some development work, galvanize institutional support to act on that problem. Then you start an initiative. You build a company, you start a standards bot or you start a new standard, you do something like that. Then you work to develop that over time. you get to a point of release and you’re maybe three years out, and then you hand it off to industry. Industry goes wild, it’s the best thing ever. They start adopting it, and then it’s two years before you get any kind of standard consistent adoption across the industry, by which point you’re already like. five to seven years from the original problem you were trying to solve. there is just no way that that process can work for something like cybersecurity that moves really quickly if it is done in a way that has some assumptions like most medical device standards assume. Laws chemistry aren’t going to be different in seven years. So you can lock you can lock those in. Those are good. If the laws of chemistry change in the next seven years, you have bigger problems than a standard. Instead, I think that where we should be going is more like here are the things to look at, the categories of things to look at, the types of things to look at, and how you’re doing better or worse on those. That allows you to then see where you are on a map and push to higher levels. And if somebody’s doing better and can publish it, then cool, you can chase that. And so, you know. It’s a selfish plug, the Hippocratic Oath for Connected Medical Device that I wrote with Eye on the Cavalry in 2015, 2016. it it tries to do that for medical device security. I think it holds up pretty well. there’s there’s five categories of things to do. it’s build defensible systems, you know, secure it, lock it down. it’s take help from others when you know a better way. So coordinated vulnerability disclosure. it’s make sure that you have the logs to be able to look at so that when issues do happen you can you can act on them. it’s have a mechanism to update. So this is, you know, whether it’s every time there’s a bug, we’re gonna replace the hardware for you and come out and do it, or whether it’s we’re gonna push software updates or whatever the mechanism is, but have a mechanism for for prompt and agile remediation. And then fifth is to build a way to ensure that a failure over here doesn’t affect the thing you actually care about over here. So some kind of containment isolation. and and I think that those are still five good principles to look at. That’s way beyond what CVSS it’s way beyond like how you score a vulnerability. but that’s the shape of a system that I would like to see more generally where it allows it allows you to look at the right categories over long periods of time to to move towards better and to gradually refine what you consider good enough as the capabilities mature and then Also to say this is no longer good enough. and to to be more or less transparent in telegraphing that so that you know don’t catch anybody out if they’re not on the standards body and all of a sudden their favorite risk mitigation strategy is now considered not cool. And they’ve got like three months or six months to get up to speed on whatever that latest TIR is or or ISO. standard that comes out.
Seth: Yeah, really interesting. I I I I think that you’re you’re pointing to something like, hey, there’s a a different model in here, right? Like what and I honestly I think we’re at a point. And you know, when I view standards effort, and I’m going back to standards because I do see CVSS as a state like it’s essentially a a de facto standard, right? The a bunch of people who are knowledgeable get in a room and they hack it out. Set it’s consensus standards in some ways, right? I think there needs to be a lot more data as opposed to opinion. Now the people people that know are bringing their opinion, which is effectively a type of data. But we’re managing vulnerabilities right now in healthcare, right? There’s no reason why we can’t sort of bottom up strategy. It’s like, hey, we tried this, we considered this, we quantified that.
Beau: Hey, we tried this, we considered this, we quantified that,
Seth: And it worked or it didn’t work, or it’s better or it’s worse in a certain way. And then all of a sudden, you quickly get to that model that’s it’s correct, better, and then you publish the standard, right? you, you it’s derived from first principles, it’s not divide derived
Beau: and it worked or it didn’t work or it’s better in the first place, then all of a sudden quickly get to that model that’s better depending on the
Seth: from research. and just two points on that: like, I think like EPSS exploit prediction scoring system is is a
Beau: and just two points on that. I think DTF’s exploit prediction storm is something is
Seth: study, right? and has data behind it, which I love, and then getting to the framework that’s flexible. Towards the Hippocratic Oath, which by the borrowed begged, borrowed and stole from in my policy wonk career. SSVC, so the Carnegie Mellon folks was a essentially a decision, decision flow, right? Like how how do I
Beau: Yes.
Seth: without getting into like ordinal math, you know, manage vulnerabilities? How do I systemically do it in an organization? what are things that I consider? It’s like we well, we we build a decision tree, right, to do it. So I think those
Beau: Yeah.
Seth: those two types of things are probably true. It’s just on us to put it together as practitioners in the space.
Beau: Yeah. Yeah yeah. All right, well I I think we have potentially beaten the the dead CVSS horse the ground.
Seth: No, we’re coming back. I can I can see take two or what is it? episode two for some reason.
Beau: unless you want to keep going, in which case we can rant all day.
Seth: I’m just waiting for the Nancy Leveson scoring system to come out, the NLS NLSS. Yeah.
Beau: LSS, I like it. I like it.
Seth: No, I’m tired. I’m beat. I can’t I I’m just it’s like we’re prize fighting or whatever, and I’m just I the need to take a corner here and get some water
Beau: Yeah. Well, Seth, it’s always a pleasure to talk. Always a pleasure to to rant at you over coffee with whiteboards in the background. and and go off on to our favorite subjects, including the behind the scenes on how Some of this stuff was actually done at the FDA in twenty fourteen to twenty nineteen, with all of the the work that you put in and the time that I was there. So thank you for the chat and we’ll see you next time.
Seth: Likewise. Cheers.