Using Cloudsmith to store and distribute any type of file

A quick walkthrough of how you can use a private Cloudsmith repository to store and then distribute any type of file.

At Cloudsmith, we want to be your “one central source of truth” for your dependencies and package management needs. And in keeping with this ideal, we are extremely pleased to announce that we have added fully configurable transparent Proxying and Caching support for Debian packages.

Well, in short, it means that you can now use your private Cloudsmith repository for all of your Debian package needs – whether that is your own private packages or packages that you need from public upstream sources. Your private Cloudsmith repository is all that you need to handle both.

If you request a package from your Cloudsmith repository, and that package isn’t present in the repo, then Cloudsmith will automatically check any upstream repos that you have configured and will then fetch (and optionally cache the package for future requests) from the upstream.

This brings you several important benefits:

. Your Cloudsmith repository is the only repository you need to configure on your clients. No more need to configure multiple repos, and handle multiple authentication credentials etc. Configure the upstreams once in Cloudsmith, and that’s it done.

. If you have cached packages and dependencies that you require in your Cloudsmith repository, then if the upstream repo goes down, is otherwise unavailable, or if the packages are removed then you can still access your cached versions. No more breaking of build or deployment process due to an unreliable upstream.

. You can view details on what specific packages were requested from the upstreams. Gain insights into what you have, and what’s missing – or who and what else you are currently relying on.

. Cloudsmith repositories are backed by a performant, global CDN. This means that your own packages and those cached from an upstream are delivered with the same low latencies and speeds. Going further than this, with edge nodes in almost all geographic regions, your users will experience this performance wherever they are located. Distributed teams all benefit equally.

. All of your packages and dependencies in one place means it’s easier for you to implement the controls and security policies that you need. Multiple sources mean multiple management tasks. Keep everything in one place and keep a tighter hold on what you have.

Well, it’s easy. In your Cloudsmith repository, you’ll see a menu item called “Upstream Proxying”. This is where we will configure our upstreams. Simply click the “Create Upstreams” button and select “Debian” to create a new Debian upstream:

You are then presented with the “Edit Debian Upstream” form. This is where we enter the details of the Debian upstream we wish to use.

We add a Name for the upstream, it’s URL and a priority weighting- in cases of multiple upstreams this will determine the order in which they are checked for a package.

We can then choose to fetch and cache any requested package (instead of just fetching them), and to verify the SSL certificates provided by the upstream. In addition, we can choose to enable this upstream for source packages too.

Next, we select the distributions and architectures that we wish to use this upstream for, and finally, we can add optional authentication headers (for private repositories that require authentication) and also optional arbitrary headers, if you wish to send something custom along with your request.

And that’s it, we have now added a new Debian upstream to our Cloudsmith repository.

Behind the scenes, Cloudsmith will now start to index the packages available in the upstream repository. The upstream will be ready for use as soon as the indexing is complete.

So now, for the next request for a package that isn’t present in this repository, Cloudsmith will check the upstream and fetch it if it is available there and also cache it in the Cloudsmith repo (if you enabled caching) for future requests. It’s that simple.

Debian upstream proxying and caching support is just another step on our path to providing you with the centralized controls, security, management and visibility that you need to enable a modern, high-velocity and DevOps-first workflow for your package management needs. It means fewer things to worry about, less exposure to change and most importantly....

Caching and Upstream Proxying for Debian Packages

Today managing your licenses with Cloudsmith has become incredibly simple.

Now, with the help of our License Compliance UI, not only can you update the license associated with a package without needing to modify a package, plus you can also view statistics of how your overall licenses appear across all packages within a repository. Don't believe me? Have a look at the screenshot below:

From the overview section, we can see the breakdown of total packages by format, types of licenses used across all packages in the repository, and the number of unlicensed packages in the repository.

In this example, we want all our licenses to be uploaded with the MIT License; however, several packages are missing licenses, and one package has uploaded with the wrong license. Let's open the edit view for the django-guardian package and update it to use a MIT License.

At Cloudsmith, we endeavor to match the license defined within a package's metadata as accurately as possible. For example, the BSD license defined within this package's metadata is checked against a valid SPDX license.

The SPDX License List is a list of commonly found licenses and exceptions used in free and open source and other collaborative software or documentation. The purpose of the SPDX License List is to enable easy and efficient identification of such licenses and exceptions in an SPDX document, in source files or elsewhere. The SPDX License List includes a standardized short identifier, full name, vetted license text including matching guidelines markup as appropriate, and a canonical permanent URL for each license and exception.

Anytime Cloudsmith matches a license automatically, we will always provide a description of what the license was defined as within the package's metadata, our confidence of how accurate the match is, and the new license that has been applied. You can find the following description on the edit page for any license that has been automatically applied:

> The 

 license provided within this package’s metadata is a 

 SPDX license and was automatically added to this package.

Anytime the match is not accurate to a high percentage, or the license is not supplied, we leave the license empty for you to decide how you want to resolve it.

To change this license or add a new one, select a new license from the autocomplete/dropdown and click the Edit button to save this change.

Once all the packages within a repository have been updated with a license, the overview provides an easy way to confirm all packages are using the same license and all packages are licensed.

Simplify License Compliance

Using GitHub Actions to push packages to Cloudsmith

Dan walks through how to connect up GitHub Actions to push packages to Cloudsmith.

GitLab CI/CD is a tool that is built into GitLab. It allows you to create automated tasks that you can use to form a Continuous Integration and Continuous Delivery / Deployment process.

You configure GitLab CI/CD by adding a yaml file (called `.gitlab-ci.yml`) to your source repository. This file creates a pipeline, which will then run when a code change is pushed to the repository. Pipelines are made up of a series of stages, and each stage can each contain a number of jobs or scripts. The GitLab Runner agent will then run these jobs.

For an on-premise instance of GitLab, you can install the GitLab runner agent on your own instances, and it supports many different operating systems (thereby creating your own fleet of instances to run your pipelines). But to keep things simple in the following examples, we will use gitlab.com and the default hosted GitLab runner environments provided.

So why would you want to use Cloudsmith with your Gitlab CI/CD pipelines? Well, there are a few reasons:

Universality – Cloudsmith supports over 20 package formats, so whatever artifacts your project produces, you can find a home for them in a private Cloudsmith repository

Control – Cloudsmith private repositories provide extensive security and access controls, that have been designed to accommodate workflows such as internal development, deployment or even distribution to external customers. The fine grained permissions system available enables you to craft bespoke access control, and lock down or open up your repository as much or as little as you need.

Automation and Integration – Thanks to the Cloudsmith CLI and also native support for format-specific package managers, a Cloudsmith private repository can fit in seamlessly with other tools in your development or distribution processes. It provides you with a single source of truth across your packages and dependencies.

Performance and Reliability – As a cloud-native platform, Cloudsmith manages the availability and performance of your repositories. You don’t need to worry about managing a fleet of servers, containers or virtual machines. Global performance, backed by our ultra-fast CDN and multi-region infrastructure, ensures that your packages are delivered worldwide. Reliably, quickly and securely.

That’s not all. With features like custom domain support, configurable upstream proxying and caching, configurable edge caching rules, download logs / statistics and more, Cloudsmith aims to provide the best solution for all your package management needs. It really is the ideal tool in a high-velocity CI/CD workflow – precisely the type of workflow that GitLab CI/CD is intended to enable you to create.

OK, so let’s get started with a worked example. The very first thing you will need is some source code in a GitLab repository that you want to build. For this example, we will build Ruby source code into a 

Our project on GitLab has the following structure:

The important thing here, as previously mentioned, is the .gitlab-ci.yml file. This is where we define the GitLab CI/CD pipeline that will run when we push a change to the GitLab repo.

Let’s take a look at the .gitlab-ci.yml file:

The first thing we have specified is the image that will be used when creating the Docker container for the GitLab runner that will execute this pipeline, in this case, Ruby 2.5.

Next, we add an environment variable, RUBYGEMS_HOST. This is where we define the URL of the Cloudsmith package repository that we will push the result of the build to.

We then define two stages in this pipeline, the build stage and the push stage.

This stage is pretty straightforward. We just run the `gem build` command to build our Ruby source as defined in our cloudsmith-ruby-example.gemspec file. Following this, we have defined an `artifact` job, as we need to temporarily store the output of the build so that the push stage next can use it.

This is because different stages in a pipeline will run on a new runner instance, so a subsequent stage wouldn’t have the access to the package built in a previous stage. You could also perform any jobs required to build and push within the same stage, and therefore on the same runner to avoid this, but for more complex pipelines you’ll likely have many stages.

The push stage is a little bit more complex. This is due to the fact that we are going to push our package to a private Cloudsmith repo, which requires authentication. The Ruby package manager, gem, allows you to store your authentication credentials (in our case, the Cloudsmith API Key) in a credentials file, located at ~/.gem/credentials – But of course, we don’t want to check this credentials file into our GitLab repository along with our source!

So we can make use of GitLabs ability to add variables to the source code repository. We can create a file variable called “CLOUDSMITH_API_KEY”, and then as part of the push step, we add a job to move this variable into the required location before we run the `gem push` command.

You add this file variable in your GitLab repository settings:

Now, when our push step runs, the `mkdir` and `mv` jobs will create the required ~/.gem/credentials file (with our Cloudsmith API Key in it), and all without exposing our API Key in any logs on the runner instance, nor checked in with our source code.

The final job in our push step simply runs `gem push` to upload the package we have built to our private Cloudsmith repo, as Cloudsmith repositories offer full native support for the `gem` package manager.

All that is left to do is for us to make a change to our source, and the commit and push the change to our Gitlab repo. Let’s see what happens when we do that.

The Gitlab CI/CD pipeline starts, and the build stage executes:

The build stage runs `gem build` to build the package and then stores it in GitLab’s temporary artifact storage:

The Push stage then starts to execute once the Build stage completes:

The push stage downloads the package from the temporary artifact storage, creates the required `~/gem/credentials` file, and runs `gem push` which uploads the package to our private Cloudsmith repository:

And that’s it, the pipeline is now complete and reports as "Passed":

If we now go and login to Cloudsmith, and check our `examples-repo` repository, we can see that the Ruby gem we just built is present:

Integrating a private Cloudsmith repository with GitLab CI/CD pipeline is easy, whether you are building a package that Cloudsmith provides native tooling support for (like Ruby) or even if you are using the Cloudsmith CLI to push a raw file, or other binary artifact. You can add your own private Cloudsmith repository with just a couple of lines of configuration, and it just works.

Package management can be complex and difficult, and doubly so if you are trying to manage your own package management solution and infrastructure at the same time. Try it for yourself, and see the productivity and efficiency gains that you can get from a centralized, hosted, secure package management service.

Integrating a Cloudsmith repository with a GitLab CI/CD Pipeline

Using GitLab to push packages to Cloudsmith

A short demo of how you can use GitLab CI/CD pipelines to push a package to a private Cloudsmith repository.

A key factor in good package management is organization. How you organize and structure your repositories will help you to unlock the efficiencies and promises of modern DevOps processes.

There are, of course, a multitude of ways you can organize packages. You can group by version numbers, formats, architectures, filetype and more. At Cloudsmith we have supported this by extracting as much of this metadata as possible when you upload a package and making this metadata available for searching/filtering.

Sometimes, however, the metadata just wasn’t granular enough – or it didn’t provide quite the nomenclature that you may have wanted. You wanted more. And we are extremely pleased to say that we have now added a new feature that greatly expands upon this functionality. Presenting:

So, what is new about this? And why does it matter?

Well, in short, we now give you the ability to add ANY custom tags to ANY package or container, either during package upload or after the fact. And you can do this via the Cloudsmith CLI or the Cloudsmith API.

Let’s say for example that you were using Cloudsmith to distribute your packages to end-users, and you have “Free” and “Premium” editions of your package. Well now you can simply add “Free” and “Premium” tags to the respective packages, and then create 

to give your users access to just the packages they should be allowed – without resorting to adding the edition into the filename or creating two separate repositories. You can manage it all from one central location. And of course, as Cloudsmith repositories are fully multi-tenant for package formats, you can apply these tags across all the package formats you use, all in one repository.

Or perhaps you would like to tag a package based on where it is deployed. You could tag a package as “rest-api”, or similar, to differentiate where in your production application the package is used.

Let’s start off by listing the packages in our demo repo. The CLI command for this is:

So, we can see that we have a collection of 

 in this repo, with various different versions. Let’s check one of those packages to see if it has any custom tags attached. The CLI command for this is:

OK, this package does not have any custom tags. Let’s now add one, and the CLI command for adding a tag is:

Great, we have now added a custom tag “free” to this package. We can repeat this for the other packages in the repository, varying the tags to suit our purpose. In addition, we can also specify that tags are “Immutable”, which means they can only be removed or altered by someone with Administrator permissions on the repository, or by the package owner. When we are finished adding tags, we can look at the repository on the Cloudsmith website:

You can see the new custom tags listed for each package alongside the tags that were automatically created from the metadata when the packages were processed after upload. We can now use these custom tags in any searching / filtering we need to do, or add them as a restriction on an Entitlement Tokens that we create for the repository:

The syntax for searching / filtering is the same syntax you use when creating a search-based restriction for an Entitlement Token, so it really is easy to create a set of access tokens that divide the repository up into subsets of packages.

Visibility of the attributes of your packages is important, it will help you not only structure your repositories, but can also help you gain insight on what package is where or what package a group of customers can access. And it’s not just the basic attributes of a package that you need visibility on, it’s anything about a package that matters to your organization and workflows. With universal package tagging you now have the ability to add your own searchable attributes to your packages, so you can define what is of importance.