VMware Aria Build Tools Explained: Developers Q&A Session

I began using Build Tools a few years ago, even before it became open-source. Today, it is a much more mature, advanced, and reliable product than before, and it has permanently changed how I think and create my workflows. For the sake of using VSCode to develop my workflows, it’s worth trying Build Tools. And, of course, all other goodies like IntelliSense and unit testing.

I had the privilege of speaking with Venelin Bakalov, Senior Cloud Automation Engineer and Team Lead at the Broadcom VCF PS Center of Engineering. This team is responsible for developing Build Tools for VMware Aria.

As a big fan of this tool, I was excited that Venelin kindly agreed to answer a few questions. I believe many of you will find his insights interesting and enjoy this behind-the-scenes look at how this powerful product is developed.

Q. What was the initial motivation behind developing Build Tools for VMware Aria?

A. Build Tools for VMware Aria originated as an internal VMware project. It was previously known as IaC, toolchain, and vRealize Build Tools (vRBT). Originally distributed as a VMware Fling, it was open-sourced in the beginning of 2023. The main drive was to enable a developer approach when implementing and delivering vRA/vRO IaaC through a standard development lifecycle. This allowed more developer-oriented teams to be involved in virtual infrastructure automation and provide them with capabilities to use development best practices and extend their current process with:

• SCM Integration
• Peer Reviews
• Change Management
• Release Management
• Solution Traceability
• Unit Testing
• Static Code Analysis
• Building Distributable Packages
• Achieving versioning and immutability of the deliverables
• Artifactory integration
• Full Continuous Integration / Continuous Delivery (CI/CD) capabilities

Q. What key challenges did the team face during the early development phase?

A. The early development phase involved multiple challenges, both internal and external. The idea itself was quite complex to implement, development efforts and capacity limited, we really needed this toolchain internally to improve our own development process and enable developer oriented capabilities. The first versions were delivered with little documentation for use, and even less documentation for extending vRBT itself. This made it hard to get more people involved into development, not just consumption. At the same time we needed to introduce more features rapidly leading to lack of standardization which meant code reworks at later phases and enforcing fully blocking quality standards as soon as we were finally able to slow down the pace. The products around which vRBT was revolving were also going through a lot of changes and initially had limitations. We had to reverse engineer some vRO/vRA functionalities, search for hidden (internal development) APIs, adapt to changing APIs and find a strategy for backwards compatibility support. So I can say it was quite dynamic and demanding, but the end result was worth it! And the fact that the customer demand for the toolchain was high immediately after it was exposed externally was a sign that we are moving in the right direction.

Q. How has the adoption of this tool changed over time within VMware and among external users?

A. Adoption started slowly within our own team. We aimed first to get everyone familiar internally with the new mindset and how to leverage all the benefits the tool can provide, as our better understanding helped us drive the adoption externally. Once the Build Tools for Aria became a native part of our own processes and way of thinking, adoption within our customers grew rapidly and nowadays comes hand-in-hand with VCF products automation and 3rd party integrations.

Q. What are the biggest risks or obstacles that could slow down the tool’s adoption and future growth?

A. Complicated initial setup – the first steps might be a bit overwhelming for a new or inexperienced user. As we will discuss later, we are currently working on simplifying this step through documentations and hands-on tutorials

The required change of mindset has always been challenging, especially in bigger teams. For every team, internal and external, there is usually an initial push back, a phase which requires adaptation and accepting a new way of development. Utilizing the tool requires a change in mind set:

• in coding – from scripting to object-oriented programming, from drag-and-drop building blocks to IDE and creating a meaningful structure of source code files and projects, separation of the actual code from the consuming product
• in operations – looking at the bigger picture, incorporating CI/CD capabilities into the development process, understanding infrastructure as code principles, automating steps and integrating quality gates.

but it is important to understand that you could view the transformation as a journey to take on, not as a single-step approach

Although Build Tools for Aria enables all of these possibilities, it does not require them as a prerequisite. On the contrary – the initial change might seem scary at first, but it is important to understand that you could view the transformation as a journey to take on, not as a single-step approach. This would enable you to incorporate a certain step only when you and your team are ready for it.

Q. Can you share any major lessons learned from previous iterations or releases?

A. We need to be more flexible in terms of development frequency and majority. We should not hold back in developing or changing something only because it breaks backwards compatibility or limits support for a given feature. Developing with such a mindset leads to accumulated code complexity and business logic that becomes hard to understand and even harder to manage. If a feature is no longer needed or should not be supported for a reason, it is better to remove it and not block further extensions and enhancements. At the end of the day, there are plenty of options for dependency management which allow the consumer to combine certain legacy features for a given project while still benefiting from the latest and greatest for his other work.

Q. What were the biggest technical limitations that needed to be addressed in the past few years?

A. One of the biggest technical challenges we had to overcome is the limitation implied by the Mozilla Rhino engine – the use of ES5. This was truly something we had to solve, considering we wanted to promote better development experience and usage of OOP and SOLID principles within Aria Orchestrator development. It was definitely not an easy task, but we managed to find a solution – implementing our own Typescript transpiler and adding support for a Typescript based project type. This expanded the Aria Orchestrator development possibilities even further because now you can use TS to develop even Workflows, Configuration Elements, Resource Elements and Policies and make use of all the goodies that come with it like type-checking, TS-based syntax, decorators and so on (in the old ES5 based projects only Actions were supported as JavaScript code, all the other elements were XML-based). We are striving to continuously improve this and add support for newer features. The latest related update was in release 2.40.0 which added support for TS 5.4.5

Another step forward was the addition of more runtime environments in Aria Orchestrator 8. In order to support this new feature, we created a new project type – polyglot/ABX. The fact that Build Tools for Aria is Maven-based allows the user to define dependencies between polyglot and TS-based project types and call for, example, a vSphere PowerShell-based action from a Typescript class.

Q. What are the core technical challenges when developing and maintaining Build Tools for VMware Aria?

A. The wide area of knowledge and experience needed. In order to make meaningful contributions, you must have good knowledge of different technologies (Java, Typescript, Node, Maven, test frameworks and more), how they are used and how they interact in different submodules of Build Tools for Aria. You also need experience with VCF products (or at least the one related to the project type that you are contributing to) and understanding of the business case and how the end user expects Build Tools for Aria and the related product to communicate.

The need for an actual environment to use during implementation and test phases, which is related to creating and maintaining a complex set of products in virtual infrastructure.

Q. How does the tool handle versioning and backward compatibility with other VMware products?

A. It really depends on the products, their API and actual functionality and how it changed. To better explain this, I could give 2 examples:

• If you have to compare vRA 7 with Aria Automation 8, you can see that both of them serve similar functions at high level, but the core functionality has changed and evolved a lot. To tackle this, we had to create a separate project type to handle Aria Automation (vRA) 8 projects.
• On the other hand, products like Aria Operations have also evolved a lot but still rely on some core concepts and have similar API for some of the functions. This makes it easy to adapt because when we communicate with the product, we have access to the product version, and we can decide which API we want to use.

Q. Can you describe the architecture of Build Tools for VMware Aria and how it fits into the broader VMware ecosystem?

A. Build Tools for VMware Aria provides development and release management tools for implementing automation solutions based on the VMware Aria Suite (VMware Aria Automation, VMware Aria Automation Orchestrator, VMware Aria Operations, VMware Aria Automation Pipelines, Aria Operations for Logs) and VMware Cloud Director. The solution enables Virtual Infrastructure Administrators and Automation Developers to use standard DevOps practices for managing and deploying content.

Q. What are the security measures in place to ensure compliance with enterprise-grade deployments?

A. During the initial open-source phase, the whole project was imported into an internal security scanner that provides feedback and defines any issues that must be resolved before the project is even applicable for an open-source approval. Once imported, it allowed us to enable GitHub native features, such as CodeQL analysis and Dependabot. Recently, we also added Trivy scanner configuration. Any alerts or PR suggestions that are raised from those tools are reviewed and are only closed if there is a fix implemented or if the particular alert does not affect the specific use case for which Built Tools for Aria is used. Having said all of this, it is important to note that this is primarily a dev focused tool, meaning a lot of the common vulnerabilities are actually not relevant or not exposed to our use cases. However, as mentioned above, we try to keep things updated and security concerns from reports low.

Q. How does the tool manage integrations with third-party automation frameworks like Terraform, Ansible, or Kubernetes?

• Currently we do not have direct integration with them. However, we have created a more open project type – the so called “bsc” (stands for basic) archetype which would allow you to create a Build Tools for Aria managed package from any file or set of files. This would allow you, for example, to package your state definition files and plug them in together with your other Build Tools for Aria projects into a CI/CD pipeline or a set of executable scripts. You could even go further and use Aria Orchestrator as an automation tool to invoke those integrations while both the Orchestrator code and the integration files are packaged and version-controlled under a Build Tools for Aria project.

Q. Are there any plans to improve the developer experience, such as better debugging, logging, or CI/CD integration?

A. Talking about Aria Orchestrator debugging experience, unfortunately we are facing certain limitations. However, what we try to do to overcome these – whenever we work internally or with other teams, we always try to demonstrate and underline the importance of often overlooked development practices like detailed error handling, consistent logging, usage of all logging severities and controlling log size and even utilising our own product – Operations for Logs (vRLI).

Q. What performance benchmarks do you use to measure scalability, and how do you plan to improve them?

A. Some factors to consider in terms of performance are:

• Typescript compilation – we have optimised it a lot already
• Maven – we do not have a lot of options to improve performance on our side but the consumer can utilise standard options exposed by maven such as multithreading, caching, skipping unneeded phases and so on
• Import of packages in the target systems – we make the packages as lightweight as possible and work with the product APIs directly but of course the package size affects import speed
• Artifact Manager (artifactory) configuration and speed also affect performance

Q. How do you see Build Tools for VMware Aria evolving in the next 3-5 years?

A. Build Tools for Aria will continue to evolve in the same direction in which the Aria Suite develops combining new product technologies and use cases with the needs of the customers and open-source community.

Q. How do you balance enterprise-focused needs vs. developer-friendly enhancements in your roadmap?

A. This is usually a 2-step approach that automatically rebalances itself – first an enterprise-derived requirement is delivered as core Build Tools for Aria functionality. This usually also introduces some form of complexity either in the tool itself or in its usability components. The more that feature gets adopted by development teams, the more this complexity starts becoming more obvious and the need and ideas for overcoming it and making it developer friendly starts to gather. Of course, sometimes we have a shortcut – our experience or history with similar features helps us to sometimes identify potential risks early on so we can directly deliver a user-friendly experience all together with the new requirements.

Q. What are the top feature requests from enterprise users, and how do you decide which ones to implement?

A. The most common requests are related to support for newer versions of products and the related features and upgrade of technology versions. We are checking feature requests and open issues on a daily basis. We have a labelling strategy and based on the information provided in an issue and the knowledge of the reviewer, it can either be immediately approved/declined (small request, bug, invalid issue that is the outcome of environment configuration, etc.) or put on hold for triaging. We have a regular bi-weekly roadmap meeting that involves our main internal contributors and many of the architects at which we discuss open requests and ideas that are left for triage. During those sessions we review the technical feasibility of the request, its alignment with the architecture of Built Tools for Aria and the possible timeline to introduce the request as this can be affected by relations to other features or even technical restrictions and product dependencies. In case the related issue is even more complex – we initiate an internal meeting with an SME to decide how to proceed.

Q. Have you seen any unexpected use cases that made you rethink aspects of the tool’s design?

A. We have discussed ideas about rewriting the entire tool (Node, Go and other alternatives) in order to tackle speed, accessibility, consistency with parts of the tool itself (e.g. currently we have both Java and Typescript code) and other issues. However at the current state this is just an idea that we have discussed and it is neither planned nor crossed out as an option.

Q. What steps are being taken to improve documentation, tutorials, and onboarding for new users?

A. We are constantly trying to improve documentation and make the onboarding process easier. During the last year we have updated our requirements on what a PR should contain (apart from code, of course). Every contribution is validated for proper labelling – kind (feature, bugfix, documentation, etc), semantic version, affected area, programming language and other suitable labels. We introduced a PR title format that contains affected components, related Github issue, and a summary of the changes introduced. Labels and PR titles are then used for autogeneration of release notes which during the release procedure are manually validated and adjusted if needed. Apart from that every PR also needs to contain a proper Release.md entry that follows our updated template (e.g. any breaking changes need to include a suggested upgrade procedure and detailed explanation of the major change). If the PR contains a new feature, it also needs to have its own section in the documentation pages.

Another important task that we managed to close last year is the completion of the minimal infrastructure template – a very easy to follow through guide for setting up small scale infrastructure and integrating it with Build Tools for Aria. The infrastructure supports source control system, runs build pipeline and hosts and serves artifact packages, and is used to create and test a sample Build Tools for Aria project.

We have plans to completely revamp our existing documentation, updating contents, restructuring it and using an external engine to provide a better UI experience but it is still in the works.

Q. How does Build Tools for VMware Aria differentiate itself from other similar tools in the automation space?

A. Build Tools for Aria is specifically suited to work with products from the Aria Suite. The development team work on automating and integrating Aria Suite and third-party products on a daily basis and understand well the architecture of the different products and the business needs of the end-customer, driving Build Tools for Aria in the same direction. From that perspective it offers a wide range of unique capabilities that no other tool provides.

Q. How does the development team gather feedback from the open-source community or enterprise users?

A. There are a few main channels through which we gather feedback:

• Github issues in the opensource repository which I would say is the most utilised communication channel where anyone can present an idea or request a fix.
• Github discussions – an alternative to issues which is more focused on a chat-like approach and can be utilised for topics which are not yet qualified as a specific feature or bug request. Of course, if a discussion resolves into an idea to develop a new feature, an issue can always be opened from it. The other way around is also valid – an open issue (like a bug) might turn out to be an environment problem or configuration so the issue can be closed and a discussion can be derived from it.
• Direct communication with customer teams during our interactions with them.

Q. Is the project being actively adopted by VMware customers or partners? What has the response been like?

A. Ever since Build Tools for Aria became available as an offering from Broadcom and as a public Fling (initially, opensourced after that), interest has been really high and demand has only been increasing. As I mentioned earlier, there have been difficulties related to complexity but we are actively working through open communication, improving documentation and providing actual examples to make the onboarding process easier and drive adoption. Demand for new features, enhancements and bug fixes is also increasing which shows that customers have increased their proficiency in the tool and are finding new use cases and even catching corner cases in the current implementation.

I hope you found it interesting and valuable as I did.

Thanks to Venelin and the team for their time and that opportunity to talk about this great product!