Continuous Integration and Continuous Deployment (CI/CD) has become a must for many progressive organizations, and SAMI’s OpenCloud team is not an exception. We need to move fast to be relevant in IoT and IoT Cloud, which are arguably the most competitive areas today.
This trend often leads to security left behind the CI/CD process. For our team, however, security was the first-class citizen from Day One, and this is why we made CI/CD a cornerstone of our SDLC.
Embedding security into our agile SDLC process was not easy. It required significant changes to a common approach to enforcing security. There was no time to do it the old way; there was no such luxury as a two- to three -week-long period dedicated solely to security testing and verification.
Instead, we need to implement security as we go—every day, each time a “Deploy” button is pushed. Most importantly, we need to automate it as much as possible with a reasonable level of assurance that new dangerous security bugs have not been introduced.
Why security can’t be like QA
Automation is not only a security problem; QA needs to be even more agile to satisfy a “need for speed.” QA groups have likewise changed their testing methodologies to run comprehensive tests whenever new code is available and builds move along the pipeline.
But there is a significant difference between QA and security. Everybody who has been involved in static or dynamic code analysis knows that the number of false positives can be big in security testing. Another problem is that when a severity has been assigned by a tool, it usually needs to be reviewed by a human who can assign the right priorities.
This means we cannot automatically move all security bugs found by a security scanner to a bug tracking system. Manual review is still required if we want to avoid confusion and frustration in the release management pipeline.
We eventually realized that we need a security dashboard to solve the problems described above. The dashboard is not so much for executives to report the “state of security” as it is for security analysts—people who can make sense of the security bugs found by automated tools.
We wanted to have a tool that would give us a 360-degree view of security issues at a single glance, and also meet the following requirements:
- Supports as many exciting security tools and scanners as possible. These tools, their quality, and the cost of ownership model change very fast and we don’t want to be locked into any particular vendor.
- Integrated well with popular bug tracking systems. JIRA, our bug tracking system, is a must for us.
- API interfaces that allow integrating with our own homegrown security testing tools. We want to have targeted security tests that are specific to SAMI’s OpenCloud APIs.
- Reasonably priced. Staying within a budget is very important, even when it comes to security.
- An open source solution is an additional benefit, because we want to be able to change it fast if we feel that we need to.
- A community that uses the tool is a big plus, because asking questions and getting answers in real-time is very important for us.
- A well established engineering company that supports and maintains the product is equally important.
After looking around, we realized Threadfix was the best choice to meet all of the above requirements. After talking to quite a few Threadfix developers and listening to their presentations at various security forums, we realized that they are the ones.
A big differentiating factor was the open Threadfix API, which allowed us to build an application security architecture around it. As you can see in the below diagram, this architecture is very Threadfix-centric.
On the right side is JIRA, our bug tracking system where we export some (but not all) security bugs after manually reviewing them in the Threadfix dashboard. That reduces the “noise” and false positives that usually come from security scanners in big numbers.
Finally, on the bottom you see our configurable email notification system, which is triggered by newly found security bugs with severity higher than a pre-defined threshold (currently: medium).
All these components working together give us a very good and timely picture of application security and allow us to quickly remediate threats and implement security fixes.
Threadfix is no longer just a web application, but a service that other security and development tools are integrated with.
Contributing to Threadfix
We were able to deploy and start using the tool very quickly, but soon saw that we could achieve an even better level of automation and significantly increase the productivity of our rather small security team by implementing just a few more features.
The most important thing for us was to have a better integration with JIRA, which is known for its great customizability. Our developers have created a unique JIRA ticket layout for each of the different SAMI teams. An additional functionality allows mapping data from the Threadfix database to various JIRA fields.
Another feature that we’ve implemented is a tool and UI within JIRA that allows configuring and sending alerts if new security bugs are found.
All of this has saved a lot of time for our security analysts, allowing them to focus more on threats and bugs analysis, while reducing time on entering data into Threadfix and exporting it to JIRA.
Probably the most important outcome is that after discussing the new features with Threadfix, they’ve decided to make our code available for the whole Threadfix community. We’ve been extremely happy to be able to give back to Threadfix.
Who we are
Three major components that will be always important for any security project:
Since I’ve already talked about the process and the technology behind it, not mentioning the people would be unfair!
Our core security team is very diverse in age, background and ethnicity. Our extended security team is as big as the whole SAMI team, and even more diverse. Everyone is involved in security one way or another, because security is a team game.
The picture below was taken at our latest karaoke party, where we were better known as The Soul Men.
From right to left:
- Sanjay: contributed a lot to integrating security scans to CI/CD and feeding the data to Threadfix.
- Geoffrey: our secret weapon, as we call him; implemented all Threadfix improvements, and got excellent feedback from the SAMI security and Threadfix teams.
- Andrew: formally works for a dev team, but contributed a lot and implemented so many security controls in SAMI that we consider him one of ours.
- Oleg: that’s me, a bugging mosquito that makes everybody’s lives miserable by talking all the time about “how we can make our security even better” 🙂
There is no off-the-shelf cookbook that would explain in detail how security can be automated. But you can always find solutions, people and technologies worth playing around with.
Top image: Steven Janke