TL;DR

OSSF Scorecards provide a visible badge that lets people see that an open source repo is adhering to a set of practices that minimise risks, measured by a set of automated checks. Getting this right for a single repo can be an involved process, but with that experience in hand applying the learning to a larger set of repos can be fairly straightforward. Implementing Allstar first can help pave the way.

OSSF What?

The Open Source Security Foundation (OpenSSF or OSSF)…

…is a cross-industry organization that brings together the industry’s most important open source security initiatives and the individuals and companies that support them. The OpenSSF is committed to collaboration and working both upstream and with existing communities to advance open source security for all.

https://openssf.org/about/

OSSF Security Scorecards…

…assesses open source projects for security risks through a series of automated checks.

https://securityscorecards.dev/#what-is-security-scorecards

Why?

Over the summer we read about Dart and Flutter enable Allstar and Security Scorecards, which seemed like a good way to show that they cared about security. So if we followed in their footsteps then hopefully we can show the world that we (Atsign) also care about security.

Allstar first

Scorecards looked like a LOT of work, so we started with Allstar:

Allstar is a GitHub App that continuously monitors GitHub organizations or repositories for adherence to security best practices. If Allstar detects a security policy violation, it creates an issue to alert the repository or organization owner. For some security policies, Allstar can also automatically change the project setting that caused the violation, reverting it to the expected state.

https://github.com/ossf/allstar#what-is-allstar

Starting gently

We began with an opt in strategy for a single repo, and that evening Allstar did its thing and spat out a series of issues complaining about branch protection, and a missing SECURITY.md. It took a while to figure out exactly what was wrong with the branch protection settings, but once that was done the issue automatically closed.

Ramping up

Once it was clear what Allstar required from us Terraform was used to get a consistent branch protection config across all the public repos, and Git Xargs was used to ensure that SECURITY.md was in place. Allstar could then be enabled for that full set of repos.

We also tracked down a few repos with binaries in them that belonged in Releases rather than the repo itself.

Then Scorecards

Starting with the same pilot repo as before the Scorecards GitHub Action was dropped into place, and immediately noted 125 issues in the Security tab on GitHub. It turned out that particular repo was maybe the worst place to start, as it has a number of complex GitHub Actions, some of which run a bunch of things in Dockerfiles and Python. That all meant a huge stack of dependencies to be pinned, and a lot of token permissions in workflows to be restricted. Thankfully the StepSecurity App was very helpful in identifying minimal token permissions, and finding SHAs for Actions; leaving some mopping up work for the Docker and Python bits.

OpenSSF Best Practices

Maybe the hardest part on that first repo was completing the questionnaire for the OpenSSF Best Practices Badge Program. This scored us an 84% ‘In progress’, and maybe bumped the Scorecard itself by 0.1. It’s not a process I spent time repeating for the next bunch of repos.

Scaling out

Implementing Scorecards isn’t as scriptable as Allstar, and in the end each repo was done manually, so to keep the work manageable only the repos featured in our GitHub Org profile were added rather than all public repos. Doing that mainly consisted of adding the Scorecards Action, adding the badge to the README.md, and ensuring that dependencies were pinned and token permissions minimised.

The scores are in

At the end of the initial wave the scores looked like this:

I’m very happy with the mid 8s, and OK with anything in the 8s.

That 7.1 has already been improved to a 7.6, and those 7.4s just need some more PRs to run through the post branch protection regime.

ToDo

8.5 is a decent score (the Dart SDK is presently at 8.6, and plenty of the repos for OSSF tools are at or below that level), but obviously there’s still more that can be done, specifically:

• Fuzzing
• Implementing a Static Application Security Testing (SAST) tool
• Best practices for the other repos
• Signing binaries for the repos that have them (and automating that release process)

I suspect though that we’re now in a situation where 20% effort has got 80% score, and the remaining 20% score will take another 80% of effort.

Conclusion

Yesterday brought the first time showing a customer the Scorecards. They seemed impressed, and it registered as a clear signal of caring about security.

Dependency pinning has also shone a light onto things we’re using that aren’t in the Step-Security Knowledge Base (some of which now are), which perhaps we need to pay closer attention to.

atGitHub

For a little more on how Atsign use GitHub (and some links to more resources) check out atGitHub.