Using Semaphore CI to Push packages to Cloudsmith

A short demo of how you can set up a Semaphore CI workflow to push a package to a private Cloudsmith repository.

At Cloudsmith, our mission is to make it as easy and straightforward as possible to get your packages into our system from where you can manage and distribute them to your heart's content.

To this end, we recently spent some time with Atlassian's Bitbucket team at 

 that makes it simple for Bitbucket users to easily turn their source code into distributable packages and deploy to Cloudsmith in just a few lines of YAML.

Pipes build on top of Bitbucket's excellent 

 functionality which allows users to automate their build and deploy processes, whatever form they may take.

Bitbucket Pipes are, at their simplest, the combination of a Docker container and some YAML-formatted configuration. Pipes are designed to remove what would otherwise be repeated or verbose configuration from a user's pipeline. Pipes can be shared across repositories or organisations and make many common tasks almost trivial.

The Cloudsmith pipe is included in Bitbucket's collection of officially maintained pipes and is available by default for any user that wishes to include it in their pipeline.

To use the pipe you'll first need an application or library that can be packaged into one of the formats that Cloudsmith support. You can see examples of Python and Javascript libraries in 

For the purposes of this example we'll assume you're using the Python example from the link above.

First, ensure that you can package your code using a Bitbucket pipeline by editing 

 in your repository. You can do so either via Bitbucket's web UI or using your normal editor and Git workflow.

For the example library, a basic pipeline that builds a package when a new tag is created looks like so:

This pipeline will build a new package each time you push an appropriate tag you your Bitbucket repository. The built package is then discarded as we have not yet added further instructions to tell Bitbucket what to do with it.

Next we'll add publish the package to Cloudsmith using our official pipe. We'll need to provide an API Key that the pipe can use to authenticate with Cloudsmith. You can find the official documentation for pipeline variables in the 

The API Key should be added as a "secure" variable (so it doesn't leak into logs) with the name 

Once authentication is configured, you can add the pipe configuration to your pipeline.

If using the web UI to configure your pipeline, you can select the Cloudsmith pipe right from your browser, from within the list of supported pipes as in the image below:

If using your local editor, you can follow the instructions in the 

Once added, your pipeline YAML should look something like so:

This configuration instructs the pipe to push artifacts to the 

 repository, with the API key we stored earlier. We're uploading a 

And that's it! Your pipe is now configured and ready to run. You should be able to push a new tag to Bitbucket and see your pipe run.

Bitbucket's pipeline UI provides a great overview of the status of your jobs:

Once pushed, you should see your shiny new package in the Cloudsmith UI, ready for download and install using your preferred tools:

Cloudsmith's Bitbucket pipe provides the easiest and simplest way for Bitbucket users to push their assets to Cloudsmith. Our official pipe is maintained by the Cloudsmith team and you can be sure it'll be kept up to date with all changes and features as we release them.

As always, if you have any questions about the pipe (or anything else), we're available at 

Cloudsmith + Bitbucket

Today, almost every service now is offered in a “Cloud” variant. But what does that really mean? Are all clouds services equal?

It’s easy to see why so many vendors rush to add a Cloud edition/variant of established software they sell. Undoubtedly, there has been a move to Cloud services across the industry, as more and more organizations seek to take advantage of the higher reliability and lower total cost of ownership that Cloud platforms promise.

The thing to watch out for is the major difference between Cloud Hosted and Cloud-Native.

When Cloud providers started to emerge; spearheaded by AWS who saw the opportunity to monetise the tooling and spare capacity they had in their data centres. We began to see other vendors jump to offer their software (which had previously been sold for on-prem use) on these Cloud platforms, hoping to capitalize on the emerging trend.

A lot of “enterprise” software was never designed to run on a Cloud platform. It was never architected to take advantage of the paradigm shift that cloud computing enables, and you are left with effectively paying (a premium, in some cases) for the same software, just running on a virtual machine on a Cloud platform. This is Cloud hosted. A self-contained software solution, typically cannot take advantage of what the Cloud offers - for instance, how would it know to load balance requests? Or scale up. Most of the time, it is limited by the resources it has to hand. The fixed cpu and memory it has been allocated.

On the other end of the spectrum, you have software that has Cloud built into its DNA. Software that relies on features that are only possible on Cloud platforms in order to work as it does - think fully distributed workloads, fault tolerance, and scaling for global performance.

Every superhero has an origin story. This is ours.

Cloudsmith is designed from the ground up as a Cloud-Native application. From day one, it has been architected to take advantage of all the technology that Cloud provider platforms offer.

This is why we don’t offer Cloudsmith in an on-prem variant. It’s not just a package management solution, it’s a secure, ultra-fast delivery platform for software, harnessing the power and flexibility of the Cloud and Edge computing.

Cloud-Hosted or Cloud-Native? Discover Why Cloudsmith Was Born in the Cloud

In this blog, we will walk through the process of configuring a private Cloudsmith repository as an artifact source for a Harness Continuous Deployment pipeline.

Harness is a Continuous Deployment platform that allows you to easily automate the deployment of your software to your infrastructure and environments.

The Harness Platform has two main components:

 The Harness Manager is where you manage your deployment pipelines and also where you store your deployment configurations. It is available as a SaaS or on-premises running in your own infrastructure. In this blog, we will use the SaaS version at app.harness.io. You can use the Harness Manager (or the API, or a trigger in your pipeline) to start and view your pipeline execution.

 the Harness Delegate is software you install in your environment (locally or on a cloud platform of your choice), and run as a service. The Delegate connects to the Harness Manager to receive tasks and then executes these pipeline tasks on your chosen platforms.

Installing the Harness delegate and building a pipeline from scratch in the Harness manager is beyond the scope of this blog, but Harness provide 

We are going to set up our Cloudsmith repository as a docker registry in Harness and then use Harness CD to deploy a docker image to AWS Elastic Container Service.

First, we log into the harness manager, and then go to “Setup”:

At this point it’s worth explaining some of the Harness key concepts, particularly Cloud Providers, Connectors and Applications :

Cloud Providers describe your infrastructure, such as AWS, GCP, Azure or even your own Physical Data Center. I’ve configured AWS as my Cloud Provider here and I have provided the AWS Access Key and Secret Access Key. I’ve added my AWS Secret Access Key as an encrypted secret using the included Harness Vault, and it’s always correct to use an encrypted secret and a secrets manager of some sort. Never, ever add your credential or secrets as plain text as part of a deployment pipeline.

Connectors are where you integrate repositories, registries, or monitoring tools. The Connector type we will configure here is an “Artifact Server”. This is the important bit for this blog, as it is where we add our Cloudsmith repository.

This is really simple, as Cloudsmith repositories are fully compatible as a native docker registry. All you need to add is your Cloudsmith registry URL – (docker.cloudsmith.io/v2/), your Cloudsmith username and your encrypted password.

I’ve added my Cloudsmith API key as an encrypted secret, again using the included Harness Vault.

You could also use a Cloudsmith Entitlement Token, and you can see our 

 to find out how to authenticate to a Cloudsmith repository using a token.

The next thing we configure is an Application. Applications are how you organize a deployment project in harness and consist of Services, Environments, Workflows, Pipelines, Triggers and Infrastructure Provisioners:

Services represent your applications and microservices, and this is where you declare the artifacts that you wish to use, any configuration variables and container specifications.

This is the first thing we define for our application.

I have selected my Cloudsmith artifact repository as the source server, and for my docker image name I have entered my Cloudsmith organization name “demo”, my repository name “examples-repo”, and my docker image name “cloudsmith-example-app”. I also choose here to deploy this docker image to AWS ECS:

An environment represents your deployment infrastructure, such a test, staging or production. This is the second thing that we define for this application.

We have defined a “TEST” infrastructure here on AWS here, and this is also where wer define things like the region and cluster to be used, and the IAM execution role to use:

A workflow includes the Service, Environment, and Infrastructure definitions we have added already, and then uses these for the deployment steps. A workflow is the third thing we define for this application.

There are different types of Workflows, such as basic, canary and blue/green. For this example, we will use a blue/green workflow that configures an AWS load balancer, and then deploys or upgrades the containers running on our ECS cluster. You can also add things like rollback steps and notifications:

A pipelines model is a collection of one or more stages, containing Workflows for Services and other deployment and verification steps, such as Approvals

We have set up a very simple single-stage pipeline here that just runs the Workflow we defined above:

A Trigger is used to automate deployment in response to an event, such as Git events, a new artifact in a repository, a schedule or even the success of another pipeline.

We have set up a trigger here that will start the Pipeline when a new docker image is pushed to our Cloudsmith Repository:

We have a simple python flask project, that just prints a simple message. We just need to create a docker image from this project using the ‘’docker build’ command:

This command also adds two tags to the image, a ‘latest’ tag and a ‘2.0.0’’ tag.

Now, we just push this docker image and tags to our Cloudsmith repository, exactly as you would do with any docker registry:

That’s the docker image pushed to Cloudsmith, and we can see it now in our Cloudsmith Repository:

As we have set up a Trigger in Harness, Harness will then poll any configured artifact repositories every 2 minutes for the presence of a new artifact. As a result, our Pipeline will then start to execute:

We can follow the stages in our pipeline, and the steps in our workflow here, and see the log output. For example, you can see the output of the upgrade containers step here:

Once the deployment has completed successfully, we can hit the address of our AWS Load Balancer in a browser and we can confirm that our docker image has been deployed to our ECS cluster, and the message that this example application prints:

Our automated deployment to AWS ECS was successful, our docker image was pulled from our private Cloudsmith repository, and it was all managed and controlled by Harness CD.

Integrating a Cloudsmith repository with a Harness CD pipeline

Collision 2021 - Securing Software Pipelines with Continuous Packaging

Building automation and security into software supply chains requires packaging source code, dependencies and containers into logical, versioned units.

But, in 2021, how engineers package their software is more vital than ever, requiring a serious refresh with a name befitting its focus on security within the cloud.

Now on-demand, watch our talk from Collision Conference 2021 where we explored Continuous Packaging and how it can help secure your delivery pipelines, from development through to deployment.

Using Harness to deploy packages from Cloudsmith

A short demo of how you can use Harness CD to deploy a docker image from a private Cloudsmith repository to an AWS ECS Cluster.

It was revealed just a few days ago that US Federal investigators are looking into an intrusion and insertion of malicious code into Codecov.

As many readers here will already know, Codecov is a software auditing tool that analyses your source code to check for the amount of test coverage. The intrusion targeted the Codecov bash uploader, which is a script that provides a way to send coverage reports to Codecov.

It has been determined that attackers were able to target the bash uploader script due to an error in a Docker image creation process, which effectively leaked a secret credential. This allowed them to update the bash uploader. The malicious code added to the script scraped all environment variables that are present when the script runs and sent these to an as-yet-unidentified third party.

Specifically, the following line was added to the bash uploader:

This means that if the bash uploader script was running as part of a CI process, then it would copy all of the environment variables that this process required (such as API keys, tokens, credentials for other required services etc), and leak them to the attackers.

Codecov found out about the unauthorized access on April 1st, but their investigation determined that beginning as far back as January 31st, 2021 there were periodic unauthorized alterations of the bash uploader script.

Codecov said that a customer informed the company of the issue on April 1st after noticing a discrepancy between the SHA1 checksum for the uploader posted publicly on GitHub and the checksum calculated from the Bash Uploader they had downloaded.

Simply put - not enough people were looking and checking, or more specifically, not enough people had the correct tools and processes in place to simplify such checks.

You need a single source of truth for all packages and dependencies used in your environments. Somewhere you can check all package checksums quickly. A single pane of glass for all your software artifacts, together with a good API where you can get this information and automate your checks.

The advice given by Codecov here is good:

We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders.

You can easily determine any keys, credentials or tokens that may have been compromised in your CI environments by simply running the ‘env’ command in your pipelines. Any environment variables that are output should be considered possibly compromised. You should both change/rotate them and perform an audit of their use.

This underlines the importance of two things we at Cloudsmith talk about as benefits of Continuous Packaging - Provenance and Isolation.

 - checking the signatures (if present) and checksums of artifacts that you consume. This revealed that the bash uploader script had been modified.

 - Keeping a cached copy of the bash uploader script allowed this customer to have a point of reference to compare the checksum with the publicly available version.