Let customers enforce code quality based on what matters to them

SourceClear was an application security tool that identified risks in open-source libraries, automatically creating issues for every vulnerability, undesirable license, and outdated library.

Overwhelmed customers contacted Support, requesting alerts only for the most relevant risks. The flood of less relevant issues slowed them down and dragged down their scan scores.

But “relevant” varied widely: some prioritized high CVSS severity scores, others focused on whether vulnerable methods were actively invoked, or whether GPL licenses were present.

• Customers had a powerful and flexible yet easy way to define issue creation and enforce system actions to match their needs.

• Organization administrators could allow individual policies per workspace or override them at the organization level for consistency.

• Customer frustration over excessive issues decreased, and adoption grew.

• This key feature helped SourceClear stand out in a competitive market and likely drove adoption.

• The impact was long-lasting—after SourceClear was acquired by Veracode, this policy-based approach remained a core feature, still in use today with expanded options.

• The success of this feature underscored the importance of balancing security enforcement with user control.

Support requests had already confirmed demand for this feature. Our Chief Product Officer also had it on the roadmap as table stakes to be competitive with other solutions, so we went directly to defining and designing the feature.

To complete this project, I would:

• Interview the Chief Product Officer who had domain expertise

• Define the behavior of the policy controls list (the individual rules making up a policy)

• Evaluate and design how to enter and exit custom policy mode and to formulate controls

Interviewing the Chief Product Officer provided both technical and usability requirements.

• From a technical perspective, the UI of the controls needed to map to the underlying code, meaning hard-coded sentences would not work. We needed to build a flexible syntax that could be put together like Legos.

• From a user perspective, flexibility was the top concern, since the chief complaint was that there was no way to stop certain types of issues from being created. Secondary to that was ease of use. The audience was technical, but that didn't mean the UI had to be difficult. Lastly, users should be able to tweak the default policy, revert to it, or create new policies from scratch.

Because the system created issues based on existing hard-coded controls, I had a sense of the maximum number of controls that might exist in a custom policy. Since that was a fairly small number, users could either edit a hard-coded list or create their own list. I explored both ideas, starting with a list of the default controls, all written out in sentences. I realized that this approach was both hard to parse and not flexible enough (see image below).

How could users begin using a custom policy?

Between two types of toggles that I considered to switch from the default to a custom policy, I selected a button group rather than a dropdown. Both accomplished the same goals, and the dropdown was more space-saving. However, until a user became familiar with the feature, they would have to click the dropdown to even see what the other choice was, whereas a button group listed both options up-front.

To make it really obvious how to edit and save, I used our big, standard green button for both options. The button group better matched the prominence of the Edit/Save button, unlike the easily overlooked dropdown. The more prominent button group better followed the information hierarchy as well.

Since the controls list wouldn’t be set in stone, how could someone create or edit a control? Any control could be about a vulnerability, library version, or license, so the options would need to depend on the object acted upon.

I settled on breaking each control into three modular sections that could grow over time:

• The resource is the top-level object which, at the time, was limited to "library" but could expand in the future.

  • Depending on the rest of the control, the library might be restricted to only direct dependencies, or be opened up to transitive dependencies (called indirectly by another library).

• The matcher is a comparison operator that defines how the resource is inspected.

• The descriptor is the trigger and its parameters:

  • a vulnerability of a certain severity, with or without a vulnerable method

  • a library, which should be the latest version

  • a license of a certain type

• For any vulnerability descriptor, the user could define whether to set the issue severity manually or by using the CVSS score.

• Finally, the system could break the build or continue with a warning.

The benefit of choosing this UX is that any statement could be created from these pieces, and users could create only the necessary controls without scanning a list of every permutation to find the desired controls to keep, edit, or delete.

After we defined these policies at the workspace level, our internal team raised the issue that our implementation was now flexible, but organizations may not always want that flexibility. Different controls for different workspaces would result in risk scores that were calculated on different sets of issues, making it hard to compare one workspace to another, and impossible for the security team to standardize their protocols.

To remedy this, we replicated the UX from the workspace level at the organization level, which only admins had access to. If an org-level policy existed, then it wasn't possible to create or update workspace-level policies. This gave customers more control over scan sensitivity, aligned with company policies.