Using Drone CI to push packages to Cloudsmith

A short demo of how you can set up a Drone CI pipeline to push a package to a private Cloudsmith repository.

Software development by distributed teams is nothing new. But since 2020, it’s no longer just teams that are globally dispersed, it is the individual team members themselves. Remote working is the new normal. So in this unpredictable, “modern” world we’re in, how do you put together a solution that delivers for every single team member, no matter their location?

How do two (or more) people work together on the same project, if availability times for integration and deployment have different latency due to old thinking, old infrastructure limited due geographic locations. If a package containing a crucial bug fix takes nearly 15 minutes to replicate from San Francisco to London or Tokyo, it severely hampers rapid rollout of fixes - something that could potentially affect the bottom line of your business.

Organizations have been grappling with the challenges that distributed teams present for some time now, and many different tools and services were created to lighten this burden.

Take source control, for example. Organizations were quick to embrace distributed version control systems, and then went one step further when they shifted these to the Cloud. CI/CD build pipelines are no different, having moved from on-prem build servers to - you guessed it - the Cloud.

But these only solve some of the problems that distributed teams face. What about artifacts and package management?

Historically, one solution was to stand up an on-prem package management system, either as an instance in an organizations’ cloud or a physical server. But in the changing landscape now, this isn’t a great fit anymore. How do you ensure secure, performant access for every member of your distributed development teams? What about redundancy, backups, and high availability?

You may need to stand up many instances around the globe, in many regions - dealing with VPNs, replication, clustering, CDNs etc. Pretty quickly, you’ll probably need a full-time administrator, or a dedicated team to manage this. Costs here quickly spiral, both in terms of the infrastructure you need, the licenses for the software, and the dedicated personnel.

As with other components of the software development and delivery process, it's time for package management to move to the Cloud also, just like source control and CI/CD pipelines. You can reduce your total cost of ownership and at the same time, deliver a better solution for your teams. We believe this passionately, so we created Cloudsmith - a universal, web-scale package management system.

A fully distributed platform. Cloudsmith has been architected from the ground up to deliver best-in-class performance no matter where in the world you are. Multi-region, and backed by a global CDN. We have worked extremely hard to build a reliable, performant platform. Our reputation, and that of our customers, depends on it.

A fully managed service. You don’t need to be an administrator of the platform, it’s all fully managed for you, 24/7. One of our customers put it very succinctly when they switched to Cloudsmith: “We don’t want to be Sys Admins”. We are also here to provide support anytime you need it; our engineering team works closely with our customers to ensure that they get the very best possible value from Cloudsmith.

Secure by default. One of our core principles is security; you could almost say we are obsessive about it: fine-grained access controls, multi-factor authentication, event and access logs, and an extensive permission system. We are proud to say that we have never failed a security assessment by one of our customers.

Universality. With support for 24 package formats at present, and more being added all the time, you can simplify and centralize your package management. Also, with our multi-format repos, you can add all your package formats into the same repos; there are no restrictions on how you structure your repos.

Distributed teams aren't new, but for the post-2020 world, their automated workflows need to be much better for their distributed team members. A fully cloud-native package management solution is now an essential component for modern workflows.

Missing Infrastructure: Accelerated Worldwide Delivery

Security in software is now everyone’s problem. We can no longer simply rely on InfoSec teams or your equivalent Gary “he-likes-security” to handle security-related processes and issues. All software, tools, infrastructure, and services need to be trusted.

It is important to us at Cloudsmith to provide you with the ability to build that trust within your teams or with your customers.

Cloudsmith allows you to use your own domain name for your repositories. Not a subdomain of cloudsmith.io, but your actual domain name. This means you can use URIs like 

 to brand and identify your repositories.

So, why does this matter? Seems like a simple thing, a simple question, right? Well, there are a few facets to it that do actually make it quite important:

If you are distributing your software to your own customers, it’s important that your brand is reflected at every point in the distribution chain - from purchase to delivery. Using your own domain name for your repositories also reassures your customers and users that they are accessing an authoritative source.

And even if you are not distributing to external customers, using your own domain is a sign to your internal development teams that they are dealing with a company resource. It’s reassuring, and it makes integrating an external service like Cloudsmith that little bit more transparent.

Using your own domain name for your repositories means that you are in control of how you implement additional security features for that domain, such as DNSSEC (signed DNS request/responses) and CAA (enforced certificate authorities). It’s your domain, you control it, not Cloudsmith.

Also, it reduces typo-squatting. Using custom domains removes things like your Cloudsmith organization name, and even the repository name, from the URL. Therefore, they can't be entered incorrectly.

Cloudsmith Custom Domains are a good example of our “no vendor lock-in” philosophy.

Once you have set up and started to use your private repositories, you may have tens, hundreds or even thousands of build definitions, package manager configurations and scripts across your organizations’ projects.

Let’s say your private repositories are located at a domain such as 

. Should you at any point in the future decide to move your repositories to another vendor or location, you would then need to update all of these configurations. This could be no small task and quite the upheaval.

However, if your repositories are located at say 

, then you can just continue to use these URIs (provided, of course, that your new vendor also provides the same “no vendor lock-in” approach as Cloudsmith does!). Your configs, build processes etc can all stay the same. In the background, your packages can all be migrated to a new location and your processes need be none-the-wiser. It’s fully decoupled.

Custom Domains give you more control over the Identity and Security of your hosted repositories and also provide you with more flexibility going forwards. They are one of the key features that differentiate Cloudsmith repositories from alternative solutions.

Build Trust with a Custom Domain

It’s finally happened. After months of whispers, JFrog have announced the sunsetting date for Bintray - their distribution add-on to their long-standing on-premises Artifactory product. Bintray officially 

REMINDER: JFrog is sunsetting Bintray services on May 1st, 2021. Learn more about it in this blog 

So what does that mean for Bintray users? In short, users can no longer use the Bintray service and are instead redirected to Jcenter.

Fortunately, Cloudsmith is a direct replacement for JFrog Bintray. And Artifactory. And their X-Ray product.

Don’t get us wrong - JFrog has achieved a lot over the years and we would never publicly speak out against them. They are the distinguished competition, and we respect their accomplishments.

Today, more than ever before, our world became distributed. Teams of developers started working together remotely. Each person needs new ways to communicate and collaborate, and be able to replicate working environments in real-time. It’s no longer acceptable to have everyone in, or physically near, your company headquarters bask in the warm embrace of low latency downloads, while other geographically distributed teams have to put up with significant replication lag.

Cloudsmith was architected and built from the ground up, to solve this exact problem. From day one, our thesis was that package management and software distribution are the same things.

In a cloud-enabled world, there is no distinction when delivering artifacts. It just has to be:

A web-scale, collaborative platform, with security, redundancy and dependability built-in.

Backed by global infrastructure, you’ll get the same amazing experience in San Francisco, as you do in London, Sydney, or Tokyo.

We can see the competition repositioning themselves, to build towards where we are right now - where we started, and where we’ve been for years. And that’s good for you. It’s great for you. Two major league products going toe-to-toe in the market. That means more innovation from both companies and better products for engineers worldwide.

Packages and containers are the fundamental building block of software. Continuous Packaging is the glue between Continuous Integration and Continuous Deployment, providing much-needed visibility and control to secure the pipeline end-to-end. Providing provenance of all artifacts, and an essential isolation layer away from public upstreams.

We are confident that today, and tomorrow, we have the best solution to package management and the vision of how to solve the inherent challenges with building secure supply chains. Security not just at the left, but all the way through the pipeline.

Cloudsmith provides secure storage and lightning-fast Edge distribution backed by intelligent infrastructure. It's not just a backend with a CDN - it's so much more than that. It’s a Package Delivery Network.

Cloudsmith, the JFrog Bintray Alternative & Replacement

"Build It Yourself, They Said. It Will Be Worth It, They Said" Dev Week Enterprise Keynote Session

“We’ll build it ourselves!” We’ve all heard it, seen it, and likely been directly impacted by the decision to build a custom, in-house solution rather than use an existing one. Whether it’s a CI/CD tool, artifact management solution, or even the entire DevOps tech stack, it’s a common misconception that building it internally is easier, cheaper, and faster. When, in fact, the complete opposite is true! Modern development processes demand high levels of availability and performance, so building a custom system that provides both of these isn’t a trivial undertaking. So why, then do enterprises continue to build in-house? And what exactly are the benefits of purchasing an existing solution? And how does this impact the quality of their overall organization? Join Cloudsmith’s Dan McKinney in this session as he answers all of these questions while helping attendees understand the true difference between building and buying DevOps solutions, how to make the best decision for your organization, and the benefits of an existing solution made to solve the problems you face.

Getting Started with Docker and Cloudsmith

A short demo of how to get started with Docker and Cloudsmith, including building and pushing a Docker Image to a private Cloudsmith repository and then pulling and running the image.

At Cloudsmith, we are proud of our partnership network. We work hard to collaborate and integrate with the best DevOps tools available. Building performant, modern workflows requires frictionless integrations between tooling.

Integrating Cloudsmith repositories with your CI/CD processes facilitates rapid development. Build, test, and deploy/distribute your software in repeatable, performant ways.

CircleCI is a best-in-class CI/CD platform, providing easy to configure, performant continuous integration/deployment processes. And as Cloudsmith is a CircleCI partner, deploying the results of these processes to your Cloudsmith repositories is simple with our CircleCI Orb.

A CircleCI Orb is a packaged, reusable piece of configuration for CircleCI. It enables you to eliminate repetition in your CircleCI config, helps to automate repeat processes, and helps you integrate CircleCI projects with other tools.

For example, you could install the Cloudsmith CLI as part of your CircleCI config and then use Cloudsmith CLI commands to upload build artifacts to your Cloudsmith repositories. But by using the 

, these additional steps are abstracted away for you, speeding up the development of your CircleCI projects and providing you with a simple, easy-to-use syntax to push your artifacts to Cloudsmith.

The first thing you need to do is create an account on CircleCI. Just like with Cloudsmith, you can get started for free, so you can follow along with the steps in this blog at zero cost.

You create your CircleCI account using your GitHub or Bitbucket credentials, and you then authorize CircleCI to access source repositories on these platforms. Once logged in, on your CircleCI dashboard, you can view a list of your GitHub or Bitbucket projects that you can start building right away:

CircleCI embraces `configuration-as-code`. The entire CI/CD process that runs when triggered by a commit to your source repository is controlled by a single file - the config.yml file. This whole CI/CD process is called a “pipeline.” A pipeline consists of one or more Workflows, Jobs, and Steps.

So let us take a look at how you define a pipeline in a config.yml file.

The first thing that we specify in our pipeline is the CircleCI configuration version, and then immediately following this, we define an orb that we would like to use – the CircleCI Cloudsmith Orb:

Next, we define a Job. Jobs are collections of steps which run any commands or shell scripts that we need. Each job must declare an executor, which is where the job will run. An executor can be a docker container, or a Linux, Windows or macOS VM.

For this example, we declare a Job called “Publish”, which will run on Docker Container with Python 3.7 provided by CircleCI – but you can, of course, use your own Docker images also:

jobs: publish: docker: - image: circleci/python:3.7

Following this, we declare the steps that get executed as part of the Job:

steps: - checkout - cloudsmith/ensure-api-key - cloudsmith/install-cli - run: command: | sudo apt update sudo apt-get install ruby ruby-dev rubygems build-essential -y sudo gem install --no-document fpm make fpm -f -s dir -t deb -v 1.0.1 -n cloudsmith-circleci-demo . - cloudsmith/publish: cloudsmith-repository: demo/examples-repo package-format: deb package-distribution: debian/buster package-path: ./*.deb

The first three steps check out the source from our repository, ensure that we have added our Cloudsmith API key as an 

 in our CircleCI project settings, and install the Cloudsmith CLI.

Next, we run a series of commands that install the dependencies that we need to build our example project. For this example project, we will create a Debian package, and for that, we will use a packaging tool called FPM. FPM is written in Ruby, so we need to install Ruby, some related Ruby packages and FPM itself (as a Ruby gem).

Finally, we declare the cloudsmith/publish step, from our CircleCI Cloudsmith Orb. For this step, we need to declare the Cloudsmith repository that we wish to push our built package to, the distribution and version that the package is for (as it is a Debian package) and the path to the built package itself. And that’s it – that is all the steps needed for this pipeline.

The last thing we add to our pipeline configuration is a workflow. Workflows define a list of jobs and their run order. We only need one workflow for this example pipeline, one that runs the “Publish” job (and therefore, the steps it contains) we declared previously:

workflows: cloudsmith_publish: jobs: - publish

version: 2.1 orbs: cloudsmith: cloudsmith/cloudsmith@1.0.4 jobs: publish: docker: - image: circleci/python:3.7 steps: - checkout - cloudsmith/ensure-api-key - cloudsmith/install-cli - run: command: | sudo apt update sudo apt-get install ruby ruby-dev rubygems build-essential -y sudo gem install --no-document fpm make fpm -f -s dir -t deb -v 1.0.1 -n cloudsmith-circleci-demo . - cloudsmith/publish: cloudsmith-repository: demo/examples-repo package-format: deb package-distribution: debian/buster package-path: ./*.deb workflows: cloudsmith_publish: jobs: - publish

All that is left to do now is add this config.yml file to our source project, in a directory called .circleci at the root of the project, and then commit and push to our Github or Bitbucket repository.

Once we have pushed this (and any subsequent commits) to our source repository, our pipeline will be triggered and will run our workflow:

We can click on the job name (in this case, “publish”) to see detailed console output from each step as it runs:

And once the pipeline has completed successfully, we can check the output from the “Publish Package” step:We can see that our CircleCI pipeline has successfully uploaded our Debian package to our Cloudsmith “examples-repo” repository. If we now login to Cloudsmith and have a look at the repository, we can see the package is present:

Now, I’m sure that the steps involved in building your projects would be much more complicated than the example outlined above. However, thanks to the 

, pushing the result of your build to a Cloudsmith repository should be no more complex than demonstrated.

CircleCI Developer Hub - cloudsmith/cloudsmithInstall the Cloudsmith CLI and publish packages to Cloudsmith.cloudsmith/cloudsmith

The Cloudsmith CircleCI Orb really does simplify the entire process of getting your build artifacts into your own secure, private repository – and then onwards to other CI/CD processes that your build is a dependency for, or even for direct distribution to your customers.

Increase your build velocity, enable repeatable workflows and level up your DevOps, with CircleCI and Cloudsmith. In short, 