Using Harness to deploy packages from Cloudsmith

A short demo of how you can use Harness CD to deploy a docker image from a private Cloudsmith repository to an AWS ECS Cluster.

It was revealed just a few days ago that US Federal investigators are looking into an intrusion and insertion of malicious code into Codecov.

As many readers here will already know, Codecov is a software auditing tool that analyses your source code to check for the amount of test coverage. The intrusion targeted the Codecov bash uploader, which is a script that provides a way to send coverage reports to Codecov.

It has been determined that attackers were able to target the bash uploader script due to an error in a Docker image creation process, which effectively leaked a secret credential. This allowed them to update the bash uploader. The malicious code added to the script scraped all environment variables that are present when the script runs and sent these to an as-yet-unidentified third party.

Specifically, the following line was added to the bash uploader:

This means that if the bash uploader script was running as part of a CI process, then it would copy all of the environment variables that this process required (such as API keys, tokens, credentials for other required services etc), and leak them to the attackers.

Codecov found out about the unauthorized access on April 1st, but their investigation determined that beginning as far back as January 31st, 2021 there were periodic unauthorized alterations of the bash uploader script.

Codecov said that a customer informed the company of the issue on April 1st after noticing a discrepancy between the SHA1 checksum for the uploader posted publicly on GitHub and the checksum calculated from the Bash Uploader they had downloaded.

Simply put - not enough people were looking and checking, or more specifically, not enough people had the correct tools and processes in place to simplify such checks.

You need a single source of truth for all packages and dependencies used in your environments. Somewhere you can check all package checksums quickly. A single pane of glass for all your software artifacts, together with a good API where you can get this information and automate your checks.

The advice given by Codecov here is good:

We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders.

You can easily determine any keys, credentials or tokens that may have been compromised in your CI environments by simply running the ‘env’ command in your pipelines. Any environment variables that are output should be considered possibly compromised. You should both change/rotate them and perform an audit of their use.

This underlines the importance of two things we at Cloudsmith talk about as benefits of Continuous Packaging - Provenance and Isolation.

 - checking the signatures (if present) and checksums of artifacts that you consume. This revealed that the bash uploader script had been modified.

 - Keeping a cached copy of the bash uploader script allowed this customer to have a point of reference to compare the checksum with the publicly available version.

Thoughts On the Codecov Breach

At Cloudsmith, we strive hard to ensure that private Cloudsmith repositories work with any build and release process that our customers use. Our mission is to be the universal package management solution that any modern DevOps workflow requires. Thanks to the power of the Cloudsmith CLI and API, it’s very easy to integrate our platform with any of the popular build tools out there, and in this blog, we will take a look at how you can set up your Drone CI pipelines to push the results of your builds to your own private repository.

Drone CI is an open-source Continuous Integration platform. It has a container-first approach which means that you utilize Docker containers to execute the steps in your build pipelines.

Installing and setting up Drone itself is beyond the scope of this blog, but it’s easy – the components of a Drone build environment are provided as Docker images so you can get set up and running very quickly.

A Drone environment consists of two main pieces of infrastructure. The Drone Server and Drone Runners.

The Drone server is the centralised place where you manage and observe your build pipelines and configure settings and secrets. It is installed via a lightweight Docker image that you can deploy on a cloud instance, a container service or even directly on your own hardware. It’s also available as a cloud-native solution that is free to use for open-source projects.

For this tutorial, I have installed the Drone server directly on an AWS EC2 instance.

The Drone runners are standalone daemons that poll the drone server for pipelines to execute. In this tutorial, I will make use of the Docker runner that I have installed on my local machine, but you could equally install it on a cloud instance or container service also. There are additional types of runners that are optimized for different use cases such as a stateful pipeline that needs to store files on the host machine between pipeline executions, but we will look at a simple build here that doesn’t require anything stateful.

Once you have set up the drone server, and one or more runners, you then connect the drone server to your source repositories using OAuth. I’ve already set this up and you can see the source repositories that the Drone server now has access to:

Let’s go ahead and activate it for my demo source repository:

You can see some of the main settings here, including the name of the drone configuration file from the source repo (.drone.yml)

The Drone configuration file is a YAML file that you place at the root of your source repository and it is used to define your pipeline steps. It’s actually a superset of the docker-compose file format, and it’s designed to be easily readable by anyone looking at your source.

The next thing we need to do is add a couple of secrets to the environment for this build pipeline.

The Cloudsmith API Key. We will need this as we need to authenticate push the package that we build to our Cloudsmith repository

Docker Credentials. We are going to use our own Docker image to execute this build, and this image is stored in a private Cloudsmith Docker registry. We can get these credentials from our ~/.docker/.config.json file after we do a normal `docker login`

It is, of course, a great idea to store these secrets encrypted on the Drone server as we are doing here. You do NOT want to check secrets like these into your source as part of any configuration files.

Once we have everything activated as above, we can now build the project.

I have a simple “Hello World” example project in C, that I want to build into a Debian package. Let’s put together a drone configuration file for this project. It starts off by defining the kind as a pipeline, and the runner type as a docker runner, and we have just named it Cloudsmith Demo:

Next, we just specify the OS and architecture and we then get into defining the steps that will execute when this pipeline runs:

After this, we start to declare our steps. We just need a single step here, a build and push package step, and we also declare the docker image that we will use to execute this step:

image: docker.cloudsmith.io/demo/examples-repo/drone-base:latest

This is a docker image that I have prepared already, and just so you can see what is required for this build, the Dockerfile looks like this:

# Do system updates and install dependencies

RUN apt-get -y install git ruby ruby-dev rubygems build-essential python3-pip

It contains the tooling required to compile the C source, and then package it into a Debian package. It also contains the Cloudsmith CLI, which we will use to push the package to our Cloudsmith repository.

You could install all the tooling using additional commands in the step, and forgo the use of your own Docker image. But it’s cleaner, and also makes better use of drone’s “container first” approach to build your own image with any pre-requisites for the pipeline, and use that to execute the step.

Next, we set up our Cloudsmith API Key as an environment variable, from our drone server:

We then execute the ‘make’ command to compile our C source, use ‘fpm’ to package this into a Debian package, and then use the ‘cloudsmith push deb’ command to push the built package to our Cloudsmith repo:

- fpm -f -s dir -t deb -v 1.0.1 -n cloudsmith-drone-demo .

- cloudsmith push deb demo/examples-repo/debian/buster cloudsmith-drone-demo_1.0.1_amd64.deb

Finally, we also pull in the secret we added to our Drone server that will allow Docker to pull our Docker image with our prerequisites from our private Cloudsmith Docker registry:

So, putting all this together, our complete .drone.yml config files looks like this:

Now, all that we need to do is to make a change to our source and push it to our source repository to start this build.

If we check our Drone server after pushing a change to our source, we can see that our pipeline has started to execute:

Let’s have a look and see what happened. It started by cloning our source repo,

And then our “build and push package” step executed:

You can see that it has pulled the docker image we are using to execute this step, and then executed ‘make’ and ‘fpm’ to build our Debian package. Finally, it pushed the resulting Debian package to our private Cloudsmith repository. That’s the pipeline execution finished successfully.

If we now go to our Cloudsmith repository, we can see our new cloudsmith-drone-demo Debian package – the result of our automated build pipeline, present as expected in our own private repository:

It’s easy to automate pushing the results of your Drone pipeline builds to a private Cloudsmith repository. With the simple addition of the Cloudsmith CLI (or a Docker container that has the Cloudsmith CLI), you can be up and running in just a few minutes, and benefit from the security, control and centralised management that a universal package management solution like Cloudsmith provides.

Integrating a Cloudsmith repository with a Drone CI pipeline

Using Drone CI to push packages to Cloudsmith

A short demo of how you can set up a Drone CI pipeline to push a package to a private Cloudsmith repository.

Software development by distributed teams is nothing new. But since 2020, it’s no longer just teams that are globally dispersed, it is the individual team members themselves. Remote working is the new normal. So in this unpredictable, “modern” world we’re in, how do you put together a solution that delivers for every single team member, no matter their location?

How do two (or more) people work together on the same project, if availability times for integration and deployment have different latency due to old thinking, old infrastructure limited due geographic locations. If a package containing a crucial bug fix takes nearly 15 minutes to replicate from San Francisco to London or Tokyo, it severely hampers rapid rollout of fixes - something that could potentially affect the bottom line of your business.

Organizations have been grappling with the challenges that distributed teams present for some time now, and many different tools and services were created to lighten this burden.

Take source control, for example. Organizations were quick to embrace distributed version control systems, and then went one step further when they shifted these to the Cloud. CI/CD build pipelines are no different, having moved from on-prem build servers to - you guessed it - the Cloud.

But these only solve some of the problems that distributed teams face. What about artifacts and package management?

Historically, one solution was to stand up an on-prem package management system, either as an instance in an organizations’ cloud or a physical server. But in the changing landscape now, this isn’t a great fit anymore. How do you ensure secure, performant access for every member of your distributed development teams? What about redundancy, backups, and high availability?

You may need to stand up many instances around the globe, in many regions - dealing with VPNs, replication, clustering, CDNs etc. Pretty quickly, you’ll probably need a full-time administrator, or a dedicated team to manage this. Costs here quickly spiral, both in terms of the infrastructure you need, the licenses for the software, and the dedicated personnel.

As with other components of the software development and delivery process, it's time for package management to move to the Cloud also, just like source control and CI/CD pipelines. You can reduce your total cost of ownership and at the same time, deliver a better solution for your teams. We believe this passionately, so we created Cloudsmith - a universal, web-scale package management system.

A fully distributed platform. Cloudsmith has been architected from the ground up to deliver best-in-class performance no matter where in the world you are. Multi-region, and backed by a global CDN. We have worked extremely hard to build a reliable, performant platform. Our reputation, and that of our customers, depends on it.

A fully managed service. You don’t need to be an administrator of the platform, it’s all fully managed for you, 24/7. One of our customers put it very succinctly when they switched to Cloudsmith: “We don’t want to be Sys Admins”. We are also here to provide support anytime you need it; our engineering team works closely with our customers to ensure that they get the very best possible value from Cloudsmith.

Secure by default. One of our core principles is security; you could almost say we are obsessive about it: fine-grained access controls, multi-factor authentication, event and access logs, and an extensive permission system. We are proud to say that we have never failed a security assessment by one of our customers.

Universality. With support for 24 package formats at present, and more being added all the time, you can simplify and centralize your package management. Also, with our multi-format repos, you can add all your package formats into the same repos; there are no restrictions on how you structure your repos.

Distributed teams aren't new, but for the post-2020 world, their automated workflows need to be much better for their distributed team members. A fully cloud-native package management solution is now an essential component for modern workflows.

Missing Infrastructure: Accelerated Worldwide Delivery

Security in software is now everyone’s problem. We can no longer simply rely on InfoSec teams or your equivalent Gary “he-likes-security” to handle security-related processes and issues. All software, tools, infrastructure, and services need to be trusted.

It is important to us at Cloudsmith to provide you with the ability to build that trust within your teams or with your customers.

Cloudsmith allows you to use your own domain name for your repositories. Not a subdomain of cloudsmith.io, but your actual domain name. This means you can use URIs like 

 to brand and identify your repositories.

So, why does this matter? Seems like a simple thing, a simple question, right? Well, there are a few facets to it that do actually make it quite important:

If you are distributing your software to your own customers, it’s important that your brand is reflected at every point in the distribution chain - from purchase to delivery. Using your own domain name for your repositories also reassures your customers and users that they are accessing an authoritative source.

And even if you are not distributing to external customers, using your own domain is a sign to your internal development teams that they are dealing with a company resource. It’s reassuring, and it makes integrating an external service like Cloudsmith that little bit more transparent.

Using your own domain name for your repositories means that you are in control of how you implement additional security features for that domain, such as DNSSEC (signed DNS request/responses) and CAA (enforced certificate authorities). It’s your domain, you control it, not Cloudsmith.

Also, it reduces typo-squatting. Using custom domains removes things like your Cloudsmith organization name, and even the repository name, from the URL. Therefore, they can't be entered incorrectly.

Cloudsmith Custom Domains are a good example of our “no vendor lock-in” philosophy.

Once you have set up and started to use your private repositories, you may have tens, hundreds or even thousands of build definitions, package manager configurations and scripts across your organizations’ projects.

Let’s say your private repositories are located at a domain such as 

. Should you at any point in the future decide to move your repositories to another vendor or location, you would then need to update all of these configurations. This could be no small task and quite the upheaval.

However, if your repositories are located at say 

, then you can just continue to use these URIs (provided, of course, that your new vendor also provides the same “no vendor lock-in” approach as Cloudsmith does!). Your configs, build processes etc can all stay the same. In the background, your packages can all be migrated to a new location and your processes need be none-the-wiser. It’s fully decoupled.

Custom Domains give you more control over the Identity and Security of your hosted repositories and also provide you with more flexibility going forwards. They are one of the key features that differentiate Cloudsmith repositories from alternative solutions.

Build Trust with a Custom Domain

It’s finally happened. After months of whispers, JFrog have announced the sunsetting date for Bintray - their distribution add-on to their long-standing on-premises Artifactory product. Bintray officially 

REMINDER: JFrog is sunsetting Bintray services on May 1st, 2021. Learn more about it in this blog 

So what does that mean for Bintray users? In short, users can no longer use the Bintray service and are instead redirected to Jcenter.

Fortunately, Cloudsmith is a direct replacement for JFrog Bintray. And Artifactory. And their X-Ray product.

Don’t get us wrong - JFrog has achieved a lot over the years and we would never publicly speak out against them. They are the distinguished competition, and we respect their accomplishments.

Today, more than ever before, our world became distributed. Teams of developers started working together remotely. Each person needs new ways to communicate and collaborate, and be able to replicate working environments in real-time. It’s no longer acceptable to have everyone in, or physically near, your company headquarters bask in the warm embrace of low latency downloads, while other geographically distributed teams have to put up with significant replication lag.

Cloudsmith was architected and built from the ground up, to solve this exact problem. From day one, our thesis was that package management and software distribution are the same things.

In a cloud-enabled world, there is no distinction when delivering artifacts. It just has to be:

A web-scale, collaborative platform, with security, redundancy and dependability built-in.

Backed by global infrastructure, you’ll get the same amazing experience in San Francisco, as you do in London, Sydney, or Tokyo.

We can see the competition repositioning themselves, to build towards where we are right now - where we started, and where we’ve been for years. And that’s good for you. It’s great for you. Two major league products going toe-to-toe in the market. That means more innovation from both companies and better products for engineers worldwide.

Packages and containers are the fundamental building block of software. Continuous Packaging is the glue between Continuous Integration and Continuous Deployment, providing much-needed visibility and control to secure the pipeline end-to-end. Providing provenance of all artifacts, and an essential isolation layer away from public upstreams.

We are confident that today, and tomorrow, we have the best solution to package management and the vision of how to solve the inherent challenges with building secure supply chains. Security not just at the left, but all the way through the pipeline.

Cloudsmith provides secure storage and lightning-fast Edge distribution backed by intelligent infrastructure. It's not just a backend with a CDN - it's so much more than that. It’s a Package Delivery Network.