Developing Security Champions within DevOps

Chris: Hello, and welcome to another episode of CyberSpeak with Infosec Institute. Today's guest is Ty Sbano, Head of Security at Periscope Data. Ty is going to tell us about the concept, as created by OWASP, of security champions, specifically within the DevOps team and the way in which having an in house security champion can facilitate a more secure coding environment.

Ty Sbano is an information security leader with over 12 years of experience, mainly in financial technology organizations. Ty's career has been focused on developing application and product security programs for Lending Club, Capita One, JP Morgan Chase, and Target. Key areas of knowledge include developing security champions, threat modeling, secure code training, static code analysis, component analysis, dynamic analysis, penetration testing, and red teaming.

Ty graduated from Penn State University with a BS in Information Science and Technology, and from Norwich University with an MS in Information Assurance.

Ty, thank you for being with us today.

Ty Sbano: Absolutely. Thanks again for having me, Chris. I'm really looking forward to this chat today.

Chris: Terrific. So by the time this video gets posted, the most sacred month of all for cybersecurity people, Cybersecurity Awareness Month, will have probably ended. So tell us why Cybersecurity Awareness Month is important. What should coders, managers, C-suite and enterprises, respectively, be doing to take maximum advantage of the sort of awareness that gets praised in October?

Ty: Yeah, I think the National Cybersecurity Awareness Month is a great unified front for all of us in information security, and just because this will air a month afterwards, I don't think that's going to be a big deal.

Chris: Mm-hmm (affirmative).

Ty: It's a nice refresher for everyone to have that unified front, have that conversation, and I think every place I've worked at, Periscope Data included, once you start ticking into the awareness aspect of things, you get more, I'm gonna say incidents or people that are just aware, and that'll prompt that sneaker net for that communication, that conversation to say, "Hey, I think I saw this. Is this a problem?"

And I think if you're open as a security professional to have that dialogue, and that's on all front lines before that intake can happen, it's a very powerful thing so it becomes top of mind, and while us in information security wanna believe that it's always top of mind, that's not always the case.

And I think the National Cybersecurity Awareness Month team does a phenomenal job of putting together all this free content, and you get to be part of this great community, and even if the content's not for you or your organization, there's a lot of elements that you can pick and choose as you feel free to do so, and I think that's the power with it. It's just giving you opportunity to use something that's there.

So I'm a big fan, and being at Periscope Data from my past, I used to focus just on app and product security, as you mentioned, and I thank you for that wonderful intro. Here at Periscope Data, I'm doing everything information security, so end to end, that really enables me to not have to pay for a bunch of vendors or create a bunch of PowerPoints and content. I use images, I use snippets, and it just gives me quicker kind of time on investment where I don't have to sit down and create this like graphic, this presentation, and all these stock images. You know how content takes time.

Chris: Oh. Yeah. Yeah, yeah. Time and resources.

Yeah, so what were some of the highlights of this year's month, do you think? What were some of the, like the collateral you put out or whatever that was especially effective?

Ty: You know, I think the basics for me are really important. Business email compromise has been hot this year, and you look at the amount of account takeover or people attempting to squirrel money away because they're acting as your CEO, that's something I actively battle every day, and you look at email spoofing and enabling of alerts and sand boxing of certain emails with attachments.

But at the end of the day, it relies on the human control.

Chris: Oh, yeah.

Ty: So when I think about that, I will say there was an interesting scenario that popped up where I signed up with a new vendor for computer based training. And with it, when I kicked out the email, an alert I enabled as part of our service, it popped up and said, "Are you sure this sender is who you believe it to be? Because it looks like it's coming from the outside." Yet, this employee's name is from the inside. It knows from me, but it was through a third party that tagged me as the email came out, and the amount of touch points, the amount of people quoting it into kind of our Chat Ops channels, that makes me feel good because they're paying attention to those alerts that are coming up, and ultimately, they're protecting not only our organization and each other, but our customers.

And we're a data company, and we gotta get this right.

Chris: Yeah.

Yeah. Yeah, yeah. I've seen that exact email warning only recently. I feel like that's sort of been added within the last month, and I'm very grateful for it. Even though it was someone I've trusted, it was still like a thing to be looking out. You know?

Ty: Right.

Chris: So let's talk a little bit about OWASP's Top 10 Most Critical Web Application Security Risks for 2018.

Ty: Sure.

Chris: Now the list doesn't always get updated every year, but this year, it was updated, right?

Ty: So it wasn't updated in 2018.

Chris: Okay.

Ty: Probably not in 2019. That's every three to four years. So it was last year where I think there's a pretty good cadence, but I think as a person that's spent a lot of time on the outlier of the community, not actively engaging in projects, but I go and present. I've hosted events.

I'm a big fan of a lot of the content, but when I look at the early parts of my career, there weren't certain working groups when mobile apps were droppings, like the Blackberry app for the Blackberry storm, and I'm sitting at JP Morgan Chase as the security analyst, it was a different world where, where do you get the controls?

And there's this fantastic resource for web applications, and you have to understand the driver behind it. But also, you have to look at the maturity of our laws. It's now ingrained in a lot of security frameworks. It's ingrained in regulators, syntax and language of what they expect. It's ingrained in auditors, and then you're also seeing a lot of customer due diligence or vendor due diligence coming up and saying, "Hey, are you following OWASP?" Or, "Are you doing BSIM or Open SAM for maturity of your asset program?" And it's been great to see, over the course of my career, staring as a security consultant where I remember when people said, "Hey, websites aren't gonna really do anything. Business is always gonna be business."

And I'm sitting there writing code to build websites out for the Pennsylvania Office of Rural Health, and I look at ... Security's actually important. Now, I might not make a lot of money as I'm doing this in my career, 'cause not a lot of people take it seriously, but as I moved throughout my career, you see it now. There's a cybersecurity shortage. There's a huge demand. There's more maturity. There's great communities like OWASP, where people often provide information, and they share it freely among each other. And I think that's powerful.

Chris: So like you said, a lot of people do sort of internalize the list and sort of take it to heart as they're creating and so forth, but for organizations that don't, how should DevOps sort of come to internalize these issues and be on the lookout for them in their day to day procedures? How can they sort of ... What are some concrete ways they can kinda tighten up part of that?

Ty: I think it's a great talking point, and I'm happy to report like for OWASP's Top 10, I just did an internal training, because I think it's good to walk through and make sure your engineers and your DevOps teams are aware of what the expectations or kinda the outcomes of OWASP's Top 10 are.

And then when we deal with our customers, sometimes it's a matter of, "Do you have an App Sec training programs?" Like, "Oh, yeah. We teach people about the Top 10." But here's the reality for our product. You type SQL, you get charts.

And I joke around internally. I said, "You type SQL and you get injection," and everyone kinda laughs. But that's a part of the reality of how we set it up. We got multiple protocols in place, but our application is not the same as what everyone else contributed as part of the data.

So when you stop and take a pause, I think the Top 10's great, but your internal top end list, say it's the top five or top 10. I like a top five and then you focus on the top three for the year or the quarter, and you attack that. You say, "Input validation". We're gonna look at a framework around the board. "Security misconfiguration". We're just gonna have better base lines and gold standards for CIS benchmarks.

So I think for your DevOps team, it's a great talking point. It's a learning, and then if they're not already working with the security team, and they show up and say, "Hey, we looked at the OWASP Top 10. We use some of the checklists like ASPS application security verification standard", or, "We're using that proxy," I don't know of a security team that would be upset if you showed up as an engineer and said, "Hey, we're already doing all this great security work."

Maybe I haven't figured out all the results and like what to do with them yet, but we've turned it on. There's no security team that's gonna be upset about that unless they just don't wanna do the work and manage offline instead of it, right?

Chris: Yeah. Right.

Ty: But I think it's a powerful talking point.

Chris: Yeah. So sort of moving along here, one of your primary areas of expertise, as I mentioned in the bio is that you are someone who is in the business of developing security champions within DevOps teams. And so for those who don't know the concept of security champions, or only know it in its sort of more general application, what is a security champion?

Ty: So a security champion can be defined in a couple of different ways, and I think the best way for me to explain it was really how I came about the awareness of security champions.

And it was called Security Ninjas, and it was by Brad Arkin's team from Adobe, and there were a couple of individuals that were on his team, and they created the Security Ninja program. They basically used a belt ranking system. The rest of us in the financial services between some of the names I've worked at, early on, it was like, "Yeah, we have training." We check the box, we give them awareness.

But what's next? You know? How do you track that leveling up? And I think that's the gamified approach that makes it much more powerful. My intent for security champion, I think, is a little bit different than where maybe it's iterating into now.

What I believe a security champion to be is someone that's part of an existing product or an engineering team that's in a capacity not as a whole time security professional, but they carve out, say, 15% of their time to focus on security or know when to engage, and I think that's the more powerful aspect as opposed to ...

I work in security. I can't scale to 3,000 engineers, but if all those product teams and a couple of hundred security champs that I would ideally like to have, I need that risk model, I have to understand our inventory. Not every team needs a champion, right? And then when you start to break it down, you take a risk based approach to ingrain your best champions with the best knowledge of, "Hey, this is when we need to do a targeted penetration test", or, "This is when we should talk about a threat model", or, "Oh, crap. You've had 15 flaws in the backlog, and 10 of them had manifested as vulnerabilities. What are we gonna do about that?"

And having that honest dialogue allows for speed and quality to increases, and I used to say this a lot. I stopped saying it, but I think we should all know that security is a subset of quality, and if you have that chief intervention here helping you along the way, it's just increasing the table stakes and the quality of your software, right?

Chris: Right. Mm-hmm (affirmative).

Ty: So that security champion, that's a person. Man, it's just your may have been, your advocate, your ninja, your partner in crime to make sure that security is embedded.

Chris: Okay. So it sounds like your security champion is sort of not ... They're someone who's already kind of on your team, but like you said, they're carving out a portion of their day to do security champion stuff.

So what is the sort of day to day work of a security champion apart from what they're doing already?

Ty: Again, I think when you start to break it down, every team's gonna be a little bit different on the limit.

Chris: Okay.

Ty: One of my fun and maybe harder lessons learned is I think it's always great when it's an optional program. Many organizations, when they start to turn the corner and say it's, "Work hard" or "You must", or, "Thou shalt have a security champion for product team", the fun starts to dissipate, and when people are showing up to your trainings and it's because someone told them so, it's not gonna be great as opposed to that security champion that's opted in for that buffer of 15% of their time sake.

I'm here because I've heard I get to lean how to break things. I'm learning a lot about privacy and regulation, and maybe I can change my career path as well. And also, my favorite thing is our flick back and someone says, "I came because I wanted a bobble head and I heard if I get to platinum status, I get that bobble head." I'm like, "Well, that is also a truth. If that's your motivation, I'm cool with it." A bobble head.

Chris: Yeah. Whatever it takes.

Ty: Well, I'll learn a lot along the way to become more empowered to have security that's top of mind, so that it's really that training awareness, the cognition. They put on their hacker goggles, and they understand how to look at a web application.

And your web application for banking is much different than your time management system that sits internally where 50 users use it, from acquisition that is gonna be eventually sunset.

Like, how much time and energy you wanna put into that?

Chris: Right.

Ty: Problem not alone. Your banking application, that sits on the Internet and everyone has access to 24/7 and you have 99.96 uptime, that's a different story.

So I think their average day is they engineer, but they just have that little extra mindset for security cognition.

Chris: Mm-hmm (affirmative). And that also sort of ties them to the larger company as a whole, 'cause you're sort of acting as a conduit, I suppose, to insight some, and security ideas and stuff like that.

So let's sort of take a worst case scenario. You got a coding team. You want a security champion.

Ty: That's true. Yup.

Chris: And everyone puts their hand down. No one wants to volunteer for it. Like, what are some things you can do to sort of, apart from the bobble head, to sort of make it appealing? Like, how would you sort of initiate a security champion for your department, especially when people feel like they're already overbooked and don't have that extra 15% of the day to spare? What do you do?

Ty: Yeah. I think this is where carrot and stick always come into play, and you always want to avoid the stick. Always avoid talking about regulation, always talking about the requirements. I think the cool factor absolutely helps if you've made it fun, but you have to look within the team, and as part of maturity and growth and development, if you've created a tiered sort of security champion program, one of the bigger elements I always look at is how do you get that senior security champion?

And maybe it's someone that you've transitioned, and now they're part of your team. That's been my best use cases that I look back at a financial institution, or a personal friend that wanted to then become a champion, and then became even closer friends within the organization, and then through about a two year journey of their development, they started mentoring some of the other products.

So some of the other products are not getting the traction. I'll be like, "Hey, could you maybe help me?" And they're like, "Oh, I know X. Y and Z. They're actually interested in security, but they're not getting the time." They became that advocate for me on the visit side as I'm presenting, but at the same time, they were also helping tell the story, tell the narrative of, "In about six months, I think I'm gonna go work for Ty, and I'm gonna run this thing called static analysis, and we may have an opportunity in the future, but I have to prepare someone to meet this sale.

And this is gonna move us forward, and that was really powerful. And the better thing that came out of that is that second person that came through as that replacement also became a full time professional that eventually left the organization, moved to a massive retailer, and built out their static analysis program from the ground up, and then when I hear one of my former peers as part of the building security maturity model community say, "Where did you find this person?" I'm like, "I don't really find them. They came about because they were interested in security. They were a security champion that joined the team. We had some changes, and they stepped up."

And this is the result of their passion, their drive, and I think that's an element that you cannot force or impress upon someone. If they don't have passion for security, you can't fake that. Right?

Chris: Hmm. Right.

Ty: So you have to find that natural sort of integration of security capability in all this.

Chris: Well, as you describe it, I mean, I that sounds like quite a value at not only to your department, but also your career and just the company as a whole.

Ty: Absolutely, and I think when you look at career development opportunities and ... Again, when I started in the field as a professional, engineers weren't always treated as top dog. You know?

Chris: Mm-hmm (affirmative).

Ty: I didn't feel really confident during my time that, as I was coming up and creating programs like this, the training opportunities, giving people a chance to get sponsorship for a certification, like Certified Ethical Hacker or now the OSCP with a Bachelor of Security, that gives people an outlet and a direction that maybe they're not getting in their own teams or their own management tier.

So they start to see this investment. They see the change in the world where engineers are now creating companies. Engineers are presenting. There is no delineation between business and technology. It's all magically there. I think they're helping pioneer some of those skills along the way, so that career opportunity, 100% there.

Chris: Yeah. So when a security champion sees insecure code or the outbreak of insecure coding practices, or just bad policy, what does he or she do to steer the ship back to a safe trajectory?

Ty: Yeah. Be the voice of reason before your security professional has to intervene, or there's an incident. The worst case scenario is that you have a champion that realizes there is an issue, it goes undocumented, it gets deployed to production, it gets exploited, and then we have that dialogue of, "How did we not catch this before it went out the door?"

Well, we knew about it, but we didn't document it." And I've had that one happen once or twice, but at the same time, that's the coaching moment. Like, why didn't we document it if we knew this was gonna be an exposed API endpoint that should have been internal, but we needed temporarily for 60 days to be exposed so testing with this third party can just make sense? And I think that's where sometimes you get in that balance that we talked about earlier. It's a little tough when you're a full time engineer and now you're trying to balance security priority, but that dialogue should at least happen, and I think that's the important part, is they should feel comfortable. They should have the trust between you and them, and it's not just this snitching situation. It is protecting your organization. It's protecting your customers, and I think that is the important element to take away from that relationship, is if they see something, they're willing to comfortably walk up on that sneaker net and have that talk, Chris.

Chris: Right. Now yeah. Yeah, I guess that was gonna be kinda my next question here, is there's that feeling of being the snitch or adding more work, I suppose, to the tape or what have you. Like how do you, without sort of calling in the big dogs or the managers or something like that, like how does a security champion kind of get everyone on the same page, I guess, other than just good old fashioned persuasion.

Ty: Yeah. I think storytelling's always important, you know?

Chris: Yeah.

Ty: And you look back at those battle wounds or their scars of an incident that you had to go through when you couldn't tell more than like six people about it. They can impress that knowledge upon them. It's much easier to address a flaw that hasn't been realized instead of a vulnerability that is now an incident.

There is a little bit of fear, uncertainty and doubt in that, but If you can unpack or have a team unpack, "Hey, you remember that time we had to disclose to the public and how much our stock price was impacted? How the feelings internally changed, and all of a sudden, security had to bring that golden hammer that changed everything?"

We don't want that. We'd rather do the right thing up front. This is no different than if you find a quality or a defect up front, right? And you know it's gonna impact 10,000 customers, are you just gonna accept the risk without telling everyone, or are you gonna have that internal dialogue in your next standup to say, "I think it's important to champion this through and address it before this becomes a bigger problem."

So to me, I don't think it's always about the snitching or really tattle tailing. It's protecting the company. We're all here to win at the end of the day, and if you instill that mindset, that's ... You're avoiding losing.

Chris: Yeah. Now, practically speaking, does becoming a security champion have sort of a company cost at all? Like is there, I mean, apart from the training time, is ... You know? I mean, is this something that you're gonna have to kinda pitch to the C-suite as in, "I need this much money" or, "I need this much time away to sorta do the training"? Something like that?

Ty: It really depends on the organization. Where I'm at with Periscope Data right now, I interact with every engineer. We have about 30 engineers. We're always hiring. We're looking for more amazing engineers, and I think the element there is not every organization is going to need that dedicated, like. "I need $100,000 to have all this training. I need $20,000 to have all this swag."

I think there's an element that you find a balance, and at larger scaled organizations with thousands of developers, yeah, you're gonna need dedicated staff not only to kinda run the program on the security side, but also make sure that you have time on the engineering side to commit to these activities, and it's either, "Hey, we either staff up and double our information security team along with our App Sec team, or we instill this as part of engineering, your engineering team grows, your organization becomes bigger, better and stronger, and we're here to guide. Would you prefer the security team to be a guide, to be kind of this person leading you through this, or would you rather have someone telling you exactly what to do when they fundamentally do now know what the hell's going on with your application portfolio?"

And I think that's a very easy discussion, so I think it always depends on the organization, the complexity, and some of the risk model associated with it.

Chris: So let's go the other way. Previously, I was asking how you can get someone, if you have a team that doesn't ... Everyone feels overstretched and doesn't wanna be a security champion.

Ty: Right.

Chris: How do get them interested. Now what if you're on a coding team, and you wanna become your team's security champion, but your company doesn't necessarily see the benefit in having you take this extra time, how do you make your case as a good candidate? What accomplishments should you show? How can you explain what resources you'd need to do the job, and so forth?

Ty: I think ... This is a tough one. I've fallen into this a handful of times at larger scale organizations, and recently out of my past work where you have someone that's really interested in security. But I think is also takes a security team, like you as an individual. Like, I would sit down with that person to understand their drive or their "Why".

This is no different than an interview process, like you wanna understand that this professional really wants this, because while there's a lot of glamour, there's a lot of fun, there may be tchotchkes along the way, there is work. There is commitment. There is potential situations that you're gonna be part of an instant handling process, and that's not fully explained all the time because we don't know where that could lead.

Say we need to do an investigation of a database that just got pulled out of the organization 'cause Apache Struts' vulnerabilities are just popping up year after year, and it's tough to patch those things, and then how you do that sort of analysis for that decomposition. And when you do them, are you sleeping? Probably not. Right?

Like, and those are elements, as security professionals, we know we're getting into it, but we make sure that as we're either training, mentoring some of our dedicated staff, we have that honest sort of dialogue, I mean. And I think with security champions, it's the same thing. If they're interested, have that talk or have that coffee chat, and then if you can't get that direct calendar time that's all billable or however a company tracks time, say, "Hey, I have resources. You can do this after work. I will stay with you after work. I will meet up with you on the weekend," and those relationships pay dividends.

And again, those are the folks that go on to become greater security professionals either in the organization or outside with maybe better career opportunities.

Chris: Mm-hmm (affirmative). Yeah. So that, when we say, "15% less work to do, 15% of security champions" ... A lot of the times, that turns into 115% of your work day, but you know, obviously for the greater good.

So we mostly have been talking how security champions works with DevOps and coding, but obviously, in the last couple of years, the concept of a security champion has also been sort of extended to another concept, more of like a, almost like a security evangelist for the entire company.

Now have you seen any successful test cases of companies that have implemented these types of security champion programs, and seen a noticeable improvement in their security pasture?

Ty: I think you're just changing the name of that evangelist quarrel. At the end of the day, I think the security evangelist has been around for a while. The enumeration of the name, like a "BISO", a business information security officer, a "TISO", technical information security officer, it's basically a deputy CISO, right?

Chris: Mm-hmm (affirmative).

Ty: We're seeing that in lines of businesses, but it's no different if you do it at the product level. You have a dedicated resource or someone in that capacity taking on security. Now, to call that a security champion, I don't think it matters what you end up calling them at the end of the day, as long as the role and the recognition and the expectation of how they interact is fairly consistent.

The challenge, I think, for a lot of those BISOs and TISOs that I've had the opportunity to work with is every line of business is different, so they all want their own way or they all want their own function, or, "Hey Ty, we wanna do our own penetration testing. We don't want your team to do it" and be like ...

Just from a financial perspective, this makes no sense on what you're asking. What I'd like to do is make sure we have an enterprise agreement and not be going rogue everywhere.

Chris: Yeah.

Ty: But to me, that's more operational aspects, and if titling matters where your security engineer, analyst, officer, whatever it is, is dedicated to push the agenda, I think "evangelist" kinds comes down to the right core word that you called out, [inaudible 00:25:51].

Chris: Mm-hmm (affirmative). So okay. Wrapping things up a little bit, you said that the OWASP list's probably not going to be revised in 2019, but what do you think are some of the big challenges or developments in the rounds of vulnerabilities and secure coding, and so forth, and how can security champions be a part of that in financing?

Ty: You know, I think, as the world changes, we're seeing folks get closer and closer to software. The perimeter has disappeared. We've talked about that. I think where we're starting to step into next, I think I'll hit two points.

One is a deeper reality check in intelligence, and when we take a look at risk profiling, for example, that relies on a lot of candor to know what your products are, what data flows through them, and how they fundamentally work with other applications so you know that kinda feeling, you know that they'll want materials, and I think what we're going to see is more technical visibility into those products.

What APIs have are sensitive data. Where is it going? Is it tokenized, is it encrypted, or is it just in the clear? And we're gonna rely on a Cloud based service to identify or, or do we have to pay for a SaaS community to help us?

I think we're gonna get more visibility in the realm when it comes to knowledge, so that will just be a truth as opposed to, "I have to rely on candor and trust."

The second thing, I would say, is really ... We're seeing more integration of security directly into software engineering frameworks, so if you look at GO, like you have native static analysis kind of there now. There is more GO security. You're seeing that with like Groovy, and I know [Beckman 00:27:25] just got purchased. I don't know if [inaudible 00:27:27] so cool with it on the handle, but you're seeing more opportunity in the community where you have engineers or sometimes champions that are just part of these working groups, these communities that they're embedding security from the onset, so we're not gonna have that buffer overflow conversation for C++, C# and all of that because it's an internally functional code versus an externally facing application on .NET that is still susceptible to a buffer overflow.

So I think that's where we will see a massive change, and these security champions, these maven spokes that are building it into their base level understanding of how to build good software, that'll be the change, so people like myself may not need to have that full time App Sec job or build that security champion team because we should be pushing it further down stack, and it just becomes more readily available and expected knowledge.

Chris: Ty Sbano. Thank you very much for the very fascinating talk today. I really appreciate your insights.

Ty: Absolutely. Thanks, Chris. Just wanna say, "Thanks" to you and also Frank for setting this up on our side from Periscope Data, and I really appreciate the chat today.

Chris: I appreciate it as well, and thank you all for listening and watching.

If you enjoyed today's video, you can find many more of them on our YouTube page. Just go to YouTube and type in, "Infosec institute". Check out our collection of tutorials, interviews and past webinars.

If you'd rather have us in your ears during your work day, all of our videos are available as audio podcasts.

Please visit infosecinstitute.com/cyberspeak for the full list of episodes. If you'd like to quality for a free pair of headphones with a class signup, podcast listeners can go to infosecinstitute.com/podcast to learn more, and if you'd like to try our free security IQ package, which includes phishing simulators you can use to fake phish and then educate your colleagues and friends in the ways of security awareness, please visit infosecinstitute.com/securityiq.

Thanks once again to Ty Sbano, and thank you all for watching and listening. We'll speak to you next week.