Using GitLab to push packages to Cloudsmith

A short demo of how you can use GitLab CI/CD pipelines to push a package to a private Cloudsmith repository.

A key factor in good package management is organization. How you organize and structure your repositories will help you to unlock the efficiencies and promises of modern DevOps processes.

There are, of course, a multitude of ways you can organize packages. You can group by version numbers, formats, architectures, filetype and more. At Cloudsmith we have supported this by extracting as much of this metadata as possible when you upload a package and making this metadata available for searching/filtering.

Sometimes, however, the metadata just wasn’t granular enough – or it didn’t provide quite the nomenclature that you may have wanted. You wanted more. And we are extremely pleased to say that we have now added a new feature that greatly expands upon this functionality. Presenting:

So, what is new about this? And why does it matter?

Well, in short, we now give you the ability to add ANY custom tags to ANY package or container, either during package upload or after the fact. And you can do this via the Cloudsmith CLI or the Cloudsmith API.

Let’s say for example that you were using Cloudsmith to distribute your packages to end-users, and you have “Free” and “Premium” editions of your package. Well now you can simply add “Free” and “Premium” tags to the respective packages, and then create 

to give your users access to just the packages they should be allowed – without resorting to adding the edition into the filename or creating two separate repositories. You can manage it all from one central location. And of course, as Cloudsmith repositories are fully multi-tenant for package formats, you can apply these tags across all the package formats you use, all in one repository.

Or perhaps you would like to tag a package based on where it is deployed. You could tag a package as “rest-api”, or similar, to differentiate where in your production application the package is used.

Let’s start off by listing the packages in our demo repo. The CLI command for this is:

So, we can see that we have a collection of 

 in this repo, with various different versions. Let’s check one of those packages to see if it has any custom tags attached. The CLI command for this is:

OK, this package does not have any custom tags. Let’s now add one, and the CLI command for adding a tag is:

Great, we have now added a custom tag “free” to this package. We can repeat this for the other packages in the repository, varying the tags to suit our purpose. In addition, we can also specify that tags are “Immutable”, which means they can only be removed or altered by someone with Administrator permissions on the repository, or by the package owner. When we are finished adding tags, we can look at the repository on the Cloudsmith website:

You can see the new custom tags listed for each package alongside the tags that were automatically created from the metadata when the packages were processed after upload. We can now use these custom tags in any searching / filtering we need to do, or add them as a restriction on an Entitlement Tokens that we create for the repository:

The syntax for searching / filtering is the same syntax you use when creating a search-based restriction for an Entitlement Token, so it really is easy to create a set of access tokens that divide the repository up into subsets of packages.

Visibility of the attributes of your packages is important, it will help you not only structure your repositories, but can also help you gain insight on what package is where or what package a group of customers can access. And it’s not just the basic attributes of a package that you need visibility on, it’s anything about a package that matters to your organization and workflows. With universal package tagging you now have the ability to add your own searchable attributes to your packages, so you can define what is of importance.

Universal Package Tagging

Puppet is a Continuous Configuration Automation tool that’s you can use to automate the configuration of your entire infrastructure. You can use it to manage the configuration of anywhere from a few, to thousands of servers or devices.

The Puppet server is where you create and store your configurations and define which nodes specific configurations will be applied to. Puppet uses modules and manifests to describe how a machine should be configured and they are written in a ruby-like syntax that describes the desired state of a node.

The Puppet agent is installed and runs on each of your nodes. The agent pulls configurations from the Puppet server and then applies them to a node to bring the node into the desired state. It can be configured to periodically check the Puppet server for new or updated configurations (which would be the usual configuration in a Puppet deployment), or it can be started manually on a node as required.

The Puppet server must be installed on Linux but the Puppet agents can be installed on, and manage, Unix / Linux like systems, MacOS, or Windows. Installing the Puppet server and agent(s) is beyond the scope of this article, but for further details, you can see the official 

You can use a private Cloudsmith repository in combination with the automation power provided by Puppet to bolster your pipeline with more security and control over your builds and dependencies. Puppet is an ideal tool to automate the configuration of a Cloudsmith repository on nodes in your infrastructure, and to then also install any required packages and dependencies from this repository. The key is control - control over your packages via your Cloudsmith repository, and control over your builds and configurations via Puppet.

For this example, we will use Puppet to set up a Cloudsmith repository for 

 packages, but Puppet also provides official modules to support the most popular repository types, like yum, pip, 

On our example Puppet server, we will create a module to configure a Cloudsmith repo and install a package. I have created a directory for this module, called cloudsmith-repo. In this directory, I have also created a manifests directory, and this is where we create our manifest (called init.pp).

Let’s look at each bit of this manifest and explain what it does.

Our manifest starts with a class called cloudsmith-repo, the same as the module name.

First we add the puppet “apt module” with:

Since our Cloudsmith repository works as a native apt repo, we can use the class and types from the apt module to define the configuration of our repo.

The next thing we do is use the apt:key type to set up the GPG key for our repository. We need to add the 20 Byte Fingerprint of the repository GPG key, which we can get from the signing keys section of our Cloudsmith repo, and the URL for the key:

https://dl.cloudsmith.io/TOKEN/OWNER/REPOSITORY/cfg/gpg/gpg.8-BYTE-FINGERPRINT.key

After this, we then use the apt:source type to set up the repository itself. We give the repository a short description, then add the URL for the repository - which also contains an access token as this is a private repository. In your Cloudsmith repo settings, you can create as many of these access tokens as you like, and add restrictions to each token. Or you can also use HTTP basic authentication instead, we support both types of auth:

comment => 'An Example Cloudsmith Repository',

https://dl.cloudsmith.io/TOKEN/OWNER/REPOSITORY/deb/debian

We will also add our Debian distribution version, and we also specify that this is a Debian source and binary repository, as Cloudsmith repos support both types of Debian package:

Finally, we declare the package that we would like installed, and we use ensure to specify the version of the package we would like - in this case the latest version:

So, putting that all together and adding in details for the repository and owner etc, we would have something that looks like this:

id => '7DF1831D968E392DAFD802A97D4D4CE49534374A',

https://dl.cloudsmith.io/QOH0JBRgKQx5lE7S/demo/examples-repo/cfg/gpg/gpg.7D4D4CE49534374A.key

https://dl.cloudsmith.io/QOH0JBRgKQx5lE7S/demo/examples-repo/deb/debian

~> package { 'cloudsmith-debian-example':

As stated previously, you would normally configure Puppet nodes to periodically check in with the puppet server to pull and apply configurations, but we will just kick off the puppet agent by running it manually with:

The puppet agent will now connect to the puppet server and will retrieve and apply the configuration:

We can then check to see that the Cloudsmith repo has been added as a source for apt in /etc/apt/sources.list.d/ :

And if we check that our cloudsmith-debian-example package is now present (with apt-cache showpkg cloudsmith-debian-example), we can see that it has been installed:

It really is quick and easy to set up a Cloudsmith repository and deploy your packages using Puppet. We try to ensure that things "Just Work", and when using the built-in apt_module support in Puppet (or other modules to support other repository types, like yum or pip) you should be up and running in minutes, not hours.

Using a private Cloudsmith repository allows you to level up the control you have over your packages and software artifacts, and using Puppet allows you to automate and control your infrastructure configurations. It's a great combination.

Deploy Packages From a Cloudsmith Repository With Puppet

Getting Started with Alpine Linux and Cloudsmith

How to get set up and started with your own private, secure repository for Alpine Linux Packages on Cloudsmith.

https://help.cloudsmith.io/docs/alpine-repository

Ansible is an open source continuous configuration automation (CCA) tool. You can use it to automate the management of the configuration of host systems. For example: installing and configuring applications, services, security policies; or to perform a wide variety of other administration and configuration tasks.

You also can use Ansible with a provisioning tool (such as the excellent Terraform, from the awesome folks over at Hashicorp) to automate the entire build and deployment of your infrastructure, and take steps towards true Infrastructure-as-code DevOps practices.

Ansible is also a perfect partner for your artifacts and assets stored in your private Cloudsmith repositories. As your Cloudsmith repositories can be your single source of truth, with all the access control and permission control that they provide, they give you another secure layer to your infrastructure build and management. By providing you with control of the sources of your packages that you use with your Ansible configuration automation, you insulate yourself further from any upstream changes.

Getting started – The Ansible Playbook.

An Ansible playbook is where you define the series of tasks that you wish to perform to ensure the configuration of your hosts is in the desired state. As an example, we have created an Ansible playbook that will install and configure a Cloudsmith repository for 

, and then install a package from this repository

- name: add Cloudsmith Repository GPG key

- name: install Cloudsmith Example package

Let’s break down what this playbook will do.

This is where we specify which of our hosts these tasks will run against. The hosts and their addresses are defined in a separate Ansible Inventory file and for this example, our inventory file is just a single host.

These hosts can be bare metal servers, cloud instances or virtual machines etc. You can define different groups of hosts within a playbook for different tasks and in this way a single playbook can manage a single machine or several hundred or even thousands of hosts.

This is where you specify the user on the host machine that the tasks will run as

This is where we specify the tasks themselves.

The first task is the “add Cloudsmith Repository GPG key” task. This task uses the apt_key ansible module to retrieve the GPG key from our Cloudsmith Repository and install in on our host system. We specify the URL for the GPG key and as this is a private Cloudsmith repository the URL contains an embedded entitlement token to authenticate for read only access:

We also add the `state: present` – which means Ansible will install the key if it is not already installed, and if it is already installed it will do nothing. This is important, as it means our task will be idempotent, meaning no matter how many times we run this task the outcome, and therefore the configuration will always be the same. We will see this `state: present` in all our ansible tasks in this playbook. Finally, as all apt operations on our host system require sudo permissions, we add `become: yes` which tells ansible to run this task with elevated permissions. Again, we will see this on all tasks in this playbook.

The second task is the “add Cloudsmith Repository” task. This task uses the apt_repository Ansible module to install our Cloudsmith repository on our host system. We specify the URL for the repository, again containing our embedded entitlement token for authentication:

The final task is the “install Cloudsmith Example package” task, and this task uses the apt Ansible module to install a deb package. We just need to specify the name of the package we wish to install:

And that’s it, that’s all we need in this playbook to get this repository set up and our package installed.

To run a playbook, we use the `ansible-playbook` command:

ansible-playbook -i cloudsmith-demo-inventory cloudsmith-demo-playbook -K

the -K flag tells ansible to ask for our sudo password, but you can also store secrets and password encrypted using Ansible Vault – which means you could check the encrypted vault file into your version control system, alongside your playbooks themselves. But to keep things simple for this example, we will just have ansible ask us for the password:

When we run this playbook, the output was as follows:

You can see that the three tasks ran successfully, and report that they made a total of three changes. If we now check what packages we have installed our Debian host, we can see our cloudsmith-debian-example package:

And finally, if we were to run this playbook a second time (or any subsequent number of times), it reports that all tasks are OK, no changes need to be made:

While this is a very simple example, we hope that it demonstrates the possibilities and the power of using a configuration automation tool like Ansible in association with a private Cloudsmith repository. It gives you the ability to automate not only common system configuration tasks, but deployment of your own packages, your own custom configurations and even the software build of your entire infrastructure. You can version these builds in your version control systems, track changes and provide an audit-trail-of-truth all the way back to your packages, stored securely in your own private repositories.

We hope you have very happy continuous configuration automation!

Deploy Packages From a Cloudsmith Repository With Ansible

In this blog, we will go through an example of how you can use the 

 to provision resources in Cloudsmith, such as repositories and entitlement tokens.

 is an awesome Continuous Configuration Automation tool. It is used to provision, update and manage infrastructure resources such as Cloud instances, containers, physical machines and more.

It is a firm favourite among developers, due to its brilliant community and mix of power and simplicity. At Cloudsmith, we use Terraform to provision and manage our environments (following a rather disastrous outing with an alternative tool), so we are big fans ourselves.

Terraform uses providers to interact with different APIs and expose any required resources. In the case of Cloudsmith, the Cloudsmith Provider allows Terraform to access the Cloudsmith API and manage two resources – Repositories and Entitlement Tokens.

Let’s take a walk through setting up Terraform to use the Cloudsmith Provider and create some resources.

The first thing that we need to do is to add the following to the "required_providers" in our example Terraform module:

This tells Terraform to download and install the Cloudsmith Provider, and which version of the provider to use.

Next, we need to add our Cloudsmith API Key, for use by the Provider:

Of course, you should never add credentials or secrets in plain text like this in a Terraform module. You should add them as environment variables, or better yet – use a proper secrets management tool like HashiCorp Vault. Also, Terraform Cloud also allows you to store these credentials securely for free.

But in the interests of simplicity, and to keep this example brief, we will just add our API Key here for now.

We now just need to run “terraform init” to install the provider automatically.

Then we add our namespace data source. The namespace data source is used to identify the Cloudsmith account that the resources we want will be created under:

The “slug” is the only required argument. It is the user-facing identifier for the Cloudsmith account, which appears in URIs.

And now we get to creating our resources. The first resource we will create is a repository:

namespace - The namespace is determined from our namespace data source that we added previously.

slug - A user-facing identifier for the repository, which appears in URI’s

The second resource that we will create is an entitlement token. Entitlement tokens provide read only access to a repository, and we can add restrictions to them to control what package they are allowed to access within the repository, and for how long:

namespace - The namespace that this token will belong to, which will be determined from the namespace that the repository we create belongs to.

repository - The repository that this token will be created for, which will be determined from our repository resource.

We have also added two optional arguments to this token, and they are two token restrictions.

The first restriction sets a maximum number of downloads that this token can be used for, and the second restriction sets specific packages, tagged “basic”, that this token is allowed to access. For more detailed information on tokens and restrictions, please see our token 

And that’s it. That’s all we need to add to create a repository and an associated entitlement token with restrictions.

Our complete terraform module now looks like this:

If we now just run "terraform plan" we can see the actions that terraform will perform:

The plan confirms that terraform will create two resources, a repository and an entitlement token.

Now, to go ahead and create these resources, we simply run “terraform apply”. We get the same output as we did from the “terraform plan” command, and we are asked to confirm that we want to perform the actions.

When we enter “yes”, Terraform then creates the resources that we have specified:

If we now log into Cloudsmith and check our repositories, we can see our new “terraform_example” repository:

And if we check the entitlement tokens within the repository, we can see the token that we created via Terraform, complete with token restrictions:

Terraform and Cloudsmith is a powerful combination. It allows you to automate the creation of your package repositories and associated access tokens in a controlled, reproducible way. Also, as you have now defined these resources as code, you can version them and treat them as you would any other source code that you have.

You can find the full documentation for the Terraform provider 

You’re one step further down the DevOps path, long may it continue.

Introducing the Cloudsmith Terraform Provider

 preparations are already underway globally. If you're not familiar with the event, it's a month-long celebration of open source during October by Digital Ocean. Hacktoberfest promotes contributions to a diverse range of open source projects on Github and welcomes for first-time contributors to get involved with the community.

It's easy and the perfect time to get involved or even to rejoin the open source community. If contributing to your favorite open source project sounds appealing, wait, because it gets even better! Hacktoberfest is open to everyone across the world. Contributors can sign up anytime between October 1st and October 31st to become edible for fantastic swag in the form of t-shirts and stickers from Digital Ocean. Submitting four quality pull requests to a public Github repository qualifies each contributor to swag, and what fantastic swag it is! Check out last year's fantastic t-shirt!

At Cloudsmith, we use open source tools and dependencies in all of the software we build and use daily. We love giving back to the community and frequently contributing enhancements and bug fixes to libraries and package management tools to help our customers and our own internal needs.

Helping to support the open source community is important to us. That's why we provide 

 for all our package formats and provide open source tools to help users get the most out of Cloudsmith as their universal package repository.

This year, like many previous years, will see members of the team contributing to a range of open source projects for fun and to scratch our own itch by making something we love even better. Still, we're all about giving back and want to help empower developers to contribute to tools they enjoy using. That's why we will be creating a range of issues suitable for first-time contributors and experienced developers to make code and documentation improvements across some of our open source projects:

As a team, we are looking forward to engaging with the community we love, and creating new awesome software together!