Lessons learned from building an open-source side project

On Twitter you'll see jokes about folks purchasing domain names in the hopes of launching a side project. Then that domain comes up for renewal, taunting those never had the chance to launch that side project.

A couple of years ago I did the unthinkable - I built and released an open source side project and then purchased the domain afterwards.

My open-source project Projektor is by no means wildly popular, but I wanted to share out some of the lessons I learned while building it in the hopes it helps others who are looking to build and release their own open-source side projects.

Scratch your own itch

When creating Projektor, I had three main goals:

1. Build something that would be useful to me. In this case I focused on speeding up debugging of CI test failures
2. Get more hands-on knowledge with certain technologies I was interested in - mainly Typescript and Kotlin co-routines
3. Road construction doubled my bus commute time - find a useful way to spend that additional time sitting on the bus. Remember commuting? That feels like decades ago.

Elaborating on goal #1 - a pain point I had with headless CI systems like GitHub Actions is that I didn't have access to the full test report when a test failed in CI. This made it more time consuming to debug test failures, especially tests that passed locally and only failed in CI.

I wanted to build a system that would make it easier and quicker to diagnose and fix test failures so developers (including myself) could fix failures faster and get their code into users' hands more quickly. Repeatedly re-running tests in CI and hoping they pass is not a recipe for speedy development or happy developers.

Release early

There is a lot of talk in our industry about releasing a Minimum Viable Product (MVP), with varying interpretations of what MVP means. I like to think more in terms of Minimum Valuable Product - releasing a product as soon as it provides value beyond what users can get elsewhere.

The scope of a Minimum Valuable Product will vary depending on the product, but in Projektor's case the first value-add was the ability to easily view the standard out and err from test cases. Being able to see the standard out and err was especially helpful in debugging server integration test failures. I released the first version of Projektor as soon as the server had that capability and one publishing pattern (Gradle plugin).

Releasing soon had several benefits:

1. Personally, I could start getting benefits from my work and further dogfooding the project
2. The excitement of early adopting users using the project and getting value from my work
3. Positive feedback loop of users suggesting new capabilities and me implementing and deploying those capabilities, further growing the user base

Over time, I expanded Projektor's capabilities to include things like:

• Code coverage analysis - including graphs over time, direct links to uncovered code lines, etc.
• Performance test results - including graph of performance over time
• GitHub pull request comments - including direct links to test reports, code coverage delta, etc.
• Additional publishing integrations with a Node script and GitHub action
• Detect and report on flaky tests
• And more

And it all snowballed from that initial release of basic test results and the tests' system output.

Make adoption easy

Projektor requires a build script or plugin to collect build artifacts (test results, code coverage reports, etc.) and send them to the Projektor server for parsing and viewing. I wanted to make it as simple as possible for prospective users to add a Projektor plugin to their project, reducing the barrier to entry to start using Projektor. To that end, I focused on reducing the amount of configuration needed to just a single config value - the URL of the Projektor server. Starting with a Gradle plugin helped in that regard, as within a Gradle plugin I could automatically detect and collect the test results without any configuration required by the end user.

Over time, as the capabilities of Projektor grew - I continued to focus on ease of adoption. Projektor now has a couple dozen configuration options, but they all have sensible defaults and the server URL is still the only required configuration parameter. That way it gives users control to configure Projektor to their specific needs if needed, but getting started is as simple as possible for the majority of users.

Automate everything

Automation is key for many successful software projects, and I think especially so for this one as I only worked on it intermittently on the side. I had stretches where I worked on it pretty much every day (on the commute and on the weekends), but there were other times where I didn't work on the project for weeks in a row. So I couldn't rely on keeping the full mental model in my head of how the software worked - I needed to embed that in the project's tests. Just like in regular software projects where folks come and go, I needed to safely make changes to it without breaking existing functionality. And it's easier for external contributors to contribute to the project's codebase when they have a comprehensive test suite backing them. Also, contributors can easily augment the existing tests to add cases for the functionality they are modifying rather than having to create new testing infrastructure from scratch.

In addition to the comprehensive automated test suite, I wanted it to be as simple and hassle-free to release the software. Projektor has three main components, each deployed to a different destination location:

• Server (backend and UI bundled together) published as an executable .jar to GitHub releases
• Gradle plugin published to the Gradle plugin portal
• Node script published to the NPM registry

Each one of those destinations has its own publishing pattern. I easily could have just manually run the steps to publish each of these artifacts whenever I needed to make a release. But as you know, manually running the steps opens the door to missing a key step or running one incorrectly. I'm also going to be less likely to frequently create releases if it's a tedious manual process.

I wanted to be able to easily and safely create frequent releases to get the updates to users as timely as possible, so I invested the time to automate the release process. I used GitHub Actions' ability to run different workflows based on the naming convention of a tag to run the different release pipelines:

• Package and release the server on tags formatted v<version>, e.g. v4.10.0
• Release the Gradle plugin on tags formatted gradle_<version>, e.g. gradle_7.1.0
• Publish the Node script on tags formatted node_<version>, e.g. node_3.6.0

While it was a bit of work to automate those release workflows, that work has paid itself back many times over in the simplicity and ease of creating frequent releases.

Conclusion

Developing, releasing, and enhancing the Projektor project has been a great learning opportunity for me and a satisfying experience building a product that users get value from. Hopefully sharing these experiences helps others who want to make a similar journey!