Skip to main content

Activity Scores: Repository and Contributor Scoring

Trusty establishes a connection between a repository and a package, and uses statistics and in particular, Principal Component Analysis, as a mechanism to assert the relative rating of a package versus other packages.

It retrieves metadata on ‘features’ of the repository and ‘features’ on the top maintainers/committers to a project. The repository features considered at launch are stars, forks, watchers, subscribers, open issues, networks, and active contributors. The contributor features that are considered are how many public repositories someone contributes to, public gists submitted, followers, and following counts.

Note that Trusty is an experimental service, and will be evolving rapidly; new features will be added on a continuous basis.

Together, metadata about the project and the top contributors to a project allows Trusty to generate a model using Principal Component Analysis. It looks at known malicious packages, and at a broad cross section of packages that are considered ‘known good’ and then determines which factors most effectively indicate whether something is good versus malicious. Based on the relative weight of ‘components’ it develops two 0 to 10 scores. The ranking system is linear, so a score of ‘5’ is the median, and ‘9’ would be the 90th percentile (i.e. better than 90% of known packages). A score of 0 to 1 represents the lowest 10% of packages by activity level.

The two individually calculated activity scores are:

  • Repository Activity Score - a relative ranking of the primary repository associated with the package.
  • Author Activity Score - a relative aggregate rank of the top contributors to the repository.

These two scores are averaged to render the Trusty Score.

Package Alternatives

Trusty is not only able to assess the relative activity associated with a package you might consider using, but also recommends alternative packages from the community that offer similar capabilities. It relies on generative AI to recommend alternatives that we then rank (based on their Trusty scores) and present. It only presents alternatives that have positive community ranks. Our hope is that we help individuals avoid problematic packages (that are, for example, no longer actively being maintained) and instead focus on packages that have healthy and vibrant communities supporting them.

A Word on Transparency, Open Metrics and Playing Fair

Stacklok is committed to openness, transparency and a community centric model of operations. We built Trusty as a mechanism to showcase the importance of software proof-of-origin, and to create value for developers when such information is present. While the service is experimental, we believe it is important to 'show our hand' and explain our approach to our prospective community:

  1. We believe in ‘finding truth in data’. Wherever practical we train impartial models on training sets of known good versus known compromised packages. We work hard to ensure that there are clear mathematical principles behind our scoring efforts. We will not fudge the numbers - not for ourselves (to make our own efforts look better) and not for anyone else. The numbers are the numbers. We respect that not everyone will agree with our approach, and we may well make mistakes that need to be corrected, but in the end we hope that everyone will agree that we are taking a fair hand.
  2. We believe in moving quickly and experimenting. We are proud of what we have built, but we can see some really cool directions we want to take things in to make it more useful to the community. We started with Principal Component Analysis, but are exploring a lot of different approaches to modeling. With that in mind Trusty is being shipped with an Experimental tag. We want you to kick the tires and let us know what you think, and most importantly we want your feedback to make it more useful to you.
  3. Transparency is critical. We do not believe it is okay to tell someone a package doesn’t look good without showing them what we consider good to be. We are going to strive to be transparent in how we generate metrics, but we also caution that the math is already a little complicated and likely to get a lot more complicated as we go. Also note that our appetite for transparency is somewhat at ends with point #2 (above), our need to evolve and get better quickly. We will do our best, but recognize that we will be moving quickly too.
  4. We are deliberately separating Trusty and Minder. Minder is open source tooling intended to help communities build more securely. It is intended to support communities and help them generate better operating postures, and thence better Trusty scores. This will never be ‘pay for play’. You don’t have to use Minder to get better Trusty scores. It is just a tool to help.