Multi Format Repositories

A quick look at how Cloudsmith repositories are fully Multi-Format, and how you can mix and match different package types in one repository.

At Cloudsmith, using Multi-format repositories, we provide a simple and flexible solution to deploy and distribute your software artifacts.

Multi-format repositories allow you to store artifacts of different formats in the same place. Organize your packages by environment, project, package type, or whatever way you see fit- we are not opinionated about how you organize your packages or containers.

Cloudsmith is the most flexible, universal package manager on the market, with Multi-format repositories and support for all package formats.

In the not-too-distant past, it was common to see organizations characterize themselves as Java or C# houses- it’s rare nowadays to find a project that is developed entirely in a single language.

A common project may use React for server-side rendering, Kubernetes, and Docker for orchestrating microservices and have microservices built using Python, Java, and Node.js. Different frameworks and languages are used depending on the expertise and problem they are tackling- backend, frontend, containerization- there are loads of good reasons to have multiple languages in your project- Who are we to judge your tech stack!

Many package managers don’t support a broad range of packages. That means you’ll require several repositories (Public or Private) and therefore several places that you need to manage, several places that you’ll need to integrate with your build processes, and several places that you’ll have to ensure your team has reliable, performant access to.

Other package managers don’t let you store different packages in the same repository. They make you create a repository for all your NPM packages, another for your Maven packages, and another for your Docker images. At Cloudsmith, you can do this no problem, but you don’t have to 😊

What exactly is a Multi-format Repository? Cloudsmith’s Multi-format repositories can house all your packages in one single location- no matter what package format you are dealing with!

You can throw your Docker images in with your NPM packages, your NuGets with your Helm Charts- if it makes sense for your project and your team, go for it.

Your Multi-format repository will still work as expected with your native packages management tooling. You can use the native tooling for all supported packages- ‘docker pull’, ‘pip install’, or ‘npm install’ for example- all from the one single repository!

So while you have a single repository and a single set of processes for managing, sharing, and controlling your software assets, you lose absolutely nothing when it comes to functionality.

The benefits of Multi-format repositories aren’t just limited to development environments either. If you are distributing your software to your own customers, and you provide it in a number of different formats, wouldn’t it be nice to offer all those formats to your customers from a single location? You no longer have to worry about pushing/publishing to multiple different repositories.

Multi-format repositories make it easy to develop, deploy, and distribute software.

If you are worried about losing some information by changing your repository structure, don’t worry- you can enrich your packages with Universal Package Tagging.

Cloudsmith Multi-format repositories mean you can apply these tags across all the package formats you use, all in one repository.

With universal package tagging, you now have the ability to add your own searchable attributes to your packages without complicating the repository structure.

We are not opinionated around how you organize your repositories. There are many ways to organize your repositories- below, we explore how you can structure your repositories around logical groupings of environments, projects, or package type.

A popular structure for customers in Cloudsmith is to have your repositories mirror your production pipeline and then use webhooks to automate deployment and promote assets.

A typical setup is to have a development, test, staging, and production repository. Once you upload an asset, you can promote it through the repositories: development -> test -> staging -> production. This structure helps keep the integrity of the asset, a provenance trail, and without the bandwidth cost of downloading from staging then re-uploading to production.

Nice and simple, easy to automate against- it’s a classic!

I love seeing all the packages associated with a project in the same place, especially when working on a project which uses a microservices architecture with a diverse tech stack. Use your repository to help Developers visualize how the project works together.

Storing all your packages associated with a project in the same repository is a simple and practical way to organize your packages.

Of course, if you want to create a repository for each format you use, you absolutely can do that - there is nothing about Cloudsmith’s Multi-format repositories that would prevent you from doing so.
We like this structure, and it fits beautifully into many DevOps processes, but we don’t like that other providers limit you only to use this structure.

Multi-format is the best fit for modern tech stacks

Modern tech stacks use multiple languages, frameworks, and tools. Cloudsmith’s Multi-format repositories are ideally suited to this environment- a flexible universal package manager that can store any package format in whatever repository structure you choose fit.

Reduce the complexity of deploying software and gain the flexibility to structure your repositories to suit your team and processes using Multi-format repositories.

Modern Tech Stacks need Multi-Format Repositories

Getting Started with Go and Cloudsmith

A short demo of how to get started with Go modules and Cloudsmith, including pushing a Go module to a private Cloudsmith repository and then pulling the module as a dependency in a Go project.

Docker Security Scanning

A walkthrough of Docker Security Scanning with Cloudsmith.

Get Started with Cloudsmith's Docker Security Scanning

 is an awesome tool, created by Hashicorp, with a vibrant community of open source modules for multiple cloud providers. It’s not hard to understand why it’s a favourite amongst developers; especially when it's often faster at supporting new features than some of the cloud providers!

Terraform enables its users to manage components in their architecture, ranging from the configuration of the services used in their chosen cloud provider (such as 

Amazon Web Services, Google Cloud Platform, or Microsoft Azure

), to software run on those services (like 

). Terraform even supports configuring some SaaS services you may interact with (for example, 

Using Terraform to define your desired configuration means that not only will you offload the work of creating the components you need, but also make the creation of these instances reproducible, allowing you to create, update and destroy components of your architecture as needed, all with one command (and a review of the changes before executing them!).


To make modules created by the Terraform providers available to all Terraform users, Hashicorp runs the 

 which allows users to upload modules, making them publicly available to all Terraform users.

Cloudsmith's private module registry fully supports the 

, allowing users to access both private modules available via Cloudsmith and public modules via the public registry.

Cloudsmith's private module registry also enables developers to:

Share custom modules for managing infrastructure among their team

Utilize private modules in your organization without the overhead of managing their storage

Distribute private modules as a commercial offering

Make modifications to public packages while having control of how you publish them (publicly accessible, private to your organization or open-source)

Capture the exact state of modules you depend on at a particular version

Control (allow list/deny list) at an organization, repository, and module level

Getting started with Cloudsmith and Terraform only takes a few steps. First, you need to create a Cloudsmith account and a repository to which you can upload your modules (a terraform provider for doing just this coming soon!).

If you need to install Terraform, you can find a getting started guide on the 

.

Cloudsmith's private module registry should work with all supported versions of Terraform, however, we recommend using version 

. You can check the version you have installed by running:

For this example, we'll create a module which prints out "Hello, world!".

First, we need to create the directory which will contain the module:

 file, this is the entry point for your module and will contain the code required to generate our message. To generate this message we must add the following to the 

Now that we have the code necessary for our module, we need to package and upload it to Cloudsmith. To achieve that we need to make use of the *nix tool 

To ensure we don't upload some unnecessary files, we exclude your terraform state file(s) along with 

 which can contain authentication credentials.

The name of the terraform module is critical here, as the 

terraform registry API requires this specific format

, along with several other requirements that you should review when creating modules.

To upload the module to Cloudsmith, you can use either the application's UI, or the Cloudsmith CLI. For this example, we'll be using the CLI, which has steps for installation available 

With the CLI installed and configured to authenticate successfully with Cloudsmith, we can upload our module by running:

Once uploaded, you can view your module in Cloudsmith.

Cloudsmith provides fully featured Terraform module repositories on all plans, flexible enough for use whether you’re hosting public packages for a public or open-source project, or private packages for your company’s internal needs. We're extremely proud to be able to support the Terraform ecosystem with this tooling.

You can find further, context-specific information, including detailed setup and integration instructions inside each Cloudsmith repository.

Announcing Private Terraform Registry Support in Cloudsmith

The movement away from on-premise and towards the Cloud is unstoppable. Even the 

 government is on board with their plans to “accelerate movement to secure cloud services, including Software as a Service (SaaS), Infrastructure as a Service (IaaS), and Platform as a Service (PaaS).”

There's many types of infrastructure to consider when selecting software. We aim to uncover the true difference of the key two, being on-premise software and cloud-native software below.

Cloud-native software and technologies are an approach to architecting, building, and managing workloads built in the cloud. This means that cloud-native software has the Cloud inherently built into its DNA, taking full advantage of all the technology that Cloud provider platforms offer. This results in the ability to scale for global performance.

On-premises (or on-prem) software uses IT infrastructure hardware and software applications that are hosted on-site, and relies on constant maintenance and upkeep from dedicated resources. On-prem is quickly becoming the legacy approach in artifct, as organizations are realizing the power of the Cloud.

What exactly is the difference between on-premise and cloud-native?

On-premise (on-prem) software is deployed, hosted, and maintained by your organization. While you can host your on-prem software in the cloud to remove some infrastructure and maintenance costs, you really need cloud-native software to take full advantage of the Cloud.

Cloud-native software is designed, developed, and deployed to take advantage of modern cloud computing, using techniques like micro-services, modularity, containerization, continuous integration, and continuous deployment.

When you access your software over the cloud, that usually means using the SaaS licensing and delivery model. You access your software over a browser or via an API, and you pay for what you use, like a utility bill.

Cloud-native software using SaaS licensing empowers you to focus on growing your business while only paying for the services you use.

Advantages of Cloud-Native Software over On-Premise Software

Many new companies today are choosing to offload the overhead of maintaining the tools and infrastructure they rely on to cloud-native services. The use of on-prem software is mainly due to legacy or regulatory reasons.

For example, the innovation coming out of Cloud-first FinTech companies compared to the risk-averse old-school banking industry shows what you can do when unshackled from your on-prem mindset. Now you see older banks like 

 accelerating their use of the cloud to remain competitive.

There are many advantages of cloud-native over on-premise software:

Cloud software is hosted for you. You don’t have to worry about maintaining your “on-prem” software or infrastructure- No updates, no security patches, no replacing obsolete hardware. Cloud-native technologies allow businesses to reduce the total cost of ownership for businesses, especially when you factor in the staff costs of maintaining “on-prem,” never mind just the licensing fees.

Scalability is another area where the cloud excels. Cloud-native software can quickly re-adjust its resources to meet demand. A company experiencing rapid growth can use the cloud to expand its infrastructure and computing power. In contrast, the same company using on-prem infrastructure would have to quickly invest in more hardware, software, and Engineers to keep up with rapid growth. On-prem software cannot compete with Cloud-native software in terms of scalability and flexibility.

“On-prem” software solutions tend to be faster the closer you are to the infrastructure. It’s no longer acceptable for everyone near the company headquarters to have a responsive low latency experience, while other geographically distributed teams have to put up with significant replication lag. This lag stifles collaboration and productivity.

Cloud software can use techniques like content delivery networks (CDN) and edge caching to provide better performance for distributed teams without requiring you to implement complex global replication or maintain globally distributed infrastructure.

A secure system needs a secure building, training, constant security updates, high availability, monitoring, and disaster recovery infrastructure. Although many companies that host their software on-prem take security very seriously, it is expensive and consumes many working hours.

Cloud providers are driven to focus on security as their business and reputation depend on providing a robust and secure service. As a result, cloud providers use highly sophisticated security tools and resources beyond the reach of most in-house teams.

 projects that from 2019 to 2029, there will be a 22% increase in the number of jobs available to Software Developers. It’s already hard to find Software Engineers, and the best ones want to work with the latest technologies. We need to embrace tooling that takes advantage of the automation and scalability of cloud-native technologies- freeing up your engineering and server resources to build your products.

When moving from “on-prem,” it’s crucial to select cloud-native tools, like 

Some vendors offer their on-prem software on Cloud platforms. However, these legacy offerings were not designed to run on a Cloud platform or architected to take advantage of the paradigm shift that cloud computing enables. You are left with effectively paying on the double for the same on-prem software, just running in a virtual machine in the Cloud. These Frankenstein hybrid software solutions inherit the same scale and distribution problems as their on-prem alternatives.

Cloud-native software works in tandem with Cloud Infrastructure- secure, fully distributed workloads, fault tolerance, and thanks to containers, it can move from cloud to cloud with ease and scale in or out rapidly.

Cloudsmith- your very own Cloud-Native packaging tool

Package management is an example of a tool that would benefit from being Cloud Native. Package management tools develop, deploy and distribute software packages.

Cloudsmith is a Cloud-native package management tool that makes life simpler for your engineers. Don’t worry about infrastructure, patching, upgrades, replications, or scaling. Our cloud-native architecture enabled us to develop a smart CDN for software packages- or our Package Delivery Network (PDN). The PDN is optimized to ensure lightning-fast delivery for deploying or shipping licensed software to your customers.

We believe that the elasticity, flexibility, security, and scalability of the Cloud allow us to support infrastructure and service beyond the capability of traditionally provided “on-prem” tools.

Shift to the cloud away from on-prem software by choosing Cloud-native software to empower innovation, stay secure, reduce costs, and scale as your business needs change.

Go Cloud-Native or Go Home

In early 2020, threat actors breached the build systems of SolarWinds and used this access to add malicious code into one of SolarWinds products. The product, called “Orion”, is very widely used and deployed by tens of thousands of companies, including many Fortune 500 companies.

Then, from around March 2020, when SolarWinds’ customers updated their installations of Orion, the malicious code embedded gave the threat actors a backdoor which they could then use to install additional malware, spy on, and exfiltrate data.

By now, this is a very well-known, maybe even the most famous, example of a software supply chain attack.

In short – a lot! Unfortunately, the SolarWinds’ incident was not the last time a software supply chain was breached. Software supply chain attacks have been continuing and could be accelerating.

In February, it was discovered by security researchers that many public package repositories and registries could be used to implement a 

, which allowed the insertion of malicious code or packages into several major companies’ build and development processes, including Apple and Tesla.

threat actors had breached a very commonly used development tool, CodeCov

. And just a few days ago, it was revealed that the primary public registry for Winget, Microsoft’s new Windows 10 package manager, was being filled with 

corrupt, duplicate, or possibly malicious

Despite the amount of news that these incidents have generated recently, unfortunately, software supply chain attacks are not new.

 development environment was distributed, which, when used to develop software for iOS, inserted malicious code into dozens of iPhone apps. And in 2019, the same thing happened to Microsoft’s Visual Studio, which resulted in 

In 2017, another software supply chain breach launched the 

. The software updates for a Ukrainian accounting product were compromised and caused an estimated $10 billion of damages.

And also, in 2018, a group of threat actors compromised software updates from the manufacturer ASUS and the computer cleanup tool CCleaner. These particular attacks are believed to be 

 successful supply chain attacks perpetrated by this group.

Overall, these attacks have affected thousands, if not hundreds of thousands, of users and systems. And often, the difficult part is determining exactly how large the blast radius is.

There are a few reasons why we see an increase in software supply chain attacks, so let’s just touch on a couple of the most important ones.

The first reason is that software security is not new. For years now, we have been conditioning users to “not click links in unsolicited emails”, “Check that the site you are using is genuine and secure”, “protect username/password/credentials”, etc. As users, we have started to implement better opsec; we are more aware of threats and how to combat them.

So, as a result, the threat actors have had to move up the chain. Rather than target individual users and systems, they target the very processes by which users obtain their software. After all, who wouldn’t trust a software update that has been delivered from a software vendor's distribution platform and signed by the said vendor? Everything we have been conditioned to question and check passes in this case.

The second reason is that the blast radius is potentially so much greater. Rather than targeting thousands of users or systems and compromising just a few, an attack like that on Solarwinds results in a single compromise expanding to many thousands of compromised systems and networks.

The threat actors can leverage the distribution networks of a software vendor to do the dirty work for them. To let thousands of Trojan horses out into the wild and run wild. The payoff is just so much greater for a successful supply chain attack, that it represents a significantly larger return on investment to an attacker.

At Cloudsmith, we believe that the solution comes in the form of Continuous Packaging.

Continuous Packaging is a development methodology that offers the observability and control to ensure that software is always verified, packaged, and ready to deliver.

We have talked about this before, and you can watch our talks on continuous packaging on 

. We think that continuous packaging is the missing link between continuous integration and continuous deployment – The cornerstones of a modern DevOps software development process.

It really comes down to using new best-in-class tools to help you have visibility and management over all packages and code used in your environments—providing trust, isolation, and performance.

· Minimise trust. Scan at source, during build and at packaging.

· Implement and own a single source of truth for packages.

· Be a curator of software packages, not just a consumer.

· Draw a thread from build to deployment.

We will dig a bit deeper into continuous packaging in an upcoming blog, so check back soon and watch this space.

 have recently announced efforts to improve the security of software supply chains. These efforts should result in guidelines for best practices and minimum standards that vendors should meet.

At its core, this is what Cloudsmith does. Cloudsmith provides a single source of truth for all packages and containers you consume as part of your development process. It enables you to isolate yourself from public upstream package sources and take back control over your software supply chain.

It’s time for a change. It’s time for an overhaul of how we securely develop software, and it’s time to get ready.