Thank You Dan Kohn

This is a story about relationships, and how they are the currency that moves this world forward. It is a story about ii’s relationship with Dan Kohn, and how his visionary leadership and friendship changed our world, and the open source community.

Ships, Servers, and Friends #

In 2015, Hippie Hacker consulted for Chef.io, with one of his assignments involving the Maritime Telecommunications Network (MTN). Through this assignment, Hippie met Aaron Crickenburger, who at the time worked for MTN Chief innovation officer Bob Wise.

Aaron and Hippie did amazing things together, deploying hands-off, over-the-internet, network booting of servers that were then mananged by Chef software. They spent several weeks together making this happen and through this developed a friendship that, in the rippling effect of relationships, would influence the direction of Kubernetes today.

During the MTN project, Bob shipped servers to Hippie’s home in Portland, Oregon, to help with the network-booting work. When Hippie later moved to Aotearoa New Zealand, he was unable to bring the servers with them. Hippie then asked Bob if he had any money available to help relocate the servers to Aotearoa New Zealand. Bob introduced Hippie to someone who could help. This person was Dan Kohn. Bob introduced the two of them, telling Dan about Hippie’s work and values. A relationship was formed, the servers were relocated, and are still used as part of the infrastructure for Hippie’s vision.

Balena and Cross-Cloud #

Now in Aotearoa New Zealand, Hippie began doing work for Balena with Denver Williams. This technology enabled the “image pull and networking boot” in new, exciting ways. Inspired, Hippie set to use this technology with Kubernetes, by pulling down the images and installing GitLab on top of it. He showed a demo of this to Dan who, impressed by this innovative approach to using Kubernetes, asked Hippie if he’d like to join the CNCF to help demonstrate what the orginzation is capable of. Hippie jumped in, helping to create a Demo of CNCF technologies.

While the demo did not get much traction, it was foundational in exploring ideas which evolved into Hippie’s next project with CNCF. Dan had a vision to get all cloud providers actively engaged in the cloud-native experience. This vision manifested as Cross-Cloud, a way to concretely show the work of CNCF through a web frontend(cncf.ci) that showed all the projects available on participating cloud providers. Taylor Carpenter, a long time friend, continued the project with the vulk.coop team and Hippie moved on to tackle new challenges with Dan.

The start of Conformance #

In 2017 Hippie attended his first CloudNativeCon + KubeCon Europe in Berlin, Germany. It was here that Dan introduced the Kubernetes conformance standard, and the Kubernetes Certified Service Provider program.

A year later, in February 2018 Kenichi Omichi dug into the Kubernetes logs and found that Kubernetes had 481 API endpoints, only 53 of them were covered by tests. Dan knew that for the Certified Kubernetes brand to have meaning, they needed to invest to make sure test coverage was much higher. The problem was not just a lack of tests, but a lack of visibility–it required heavy, manual data mining just to calculate coverage in the first place.

To find a solution to this problem, Hippie paired with Rohan Fletcher. During a discussion with Hippie and Rohan, Dan showed them a disk usage graph for OSX and proposed they use a similar graph to visualize Conformance coverage of the Kubernetes API.

APISnoop #

Rohan started on the project to create what would be APISnoop: a visual insight into Kubernetes test coverage. This project was well-received by the community when it was introduced at Kubecon Europe 2018.

That same year, Zach Mandeville joined the team and took over development of APISnoop. At that point there was no real automation for generating Snoop’s data. Much was still done by manually looking for tests in audit event logs. Zach did a lot of the writing, rewriting, and architectural changes to APISnoop, along with driving updates of underlying Kubernetes logging so that clear coverage information could be distinguished from noise.

While APISnoop worked to show gaps in Conformance coverage, a separate effort was started to fill those gaps with tests. Unfortunately, this effort was slow going. After about a year, APISnoop showed very little movement on the graph, as the test writing efforts yielded very little results. Within character, Dan started to look for other ways to get an increase in coverage happening at a rate that satisfied his vision.

Since Hippie and his team had been looking at the Kubernetes API and all its underlying parts for almost a year, it was a logical fit for them to step up to the test-writing efforts, and work to increase the test writing velocity.

Test-Writing #

The writing of tests was a learning experience for everyone in ii, as well as the contributors in the Kubernetes community. Initially, the process was to quietly work on tests, ensuring they fit all known requirements and then, once they seemed ready, to share it with the community through a PR. These PR’s invariably sparked feedback and needed revisions, and could lead to discussions on whether the particular test meaningfully changed the coverage at all. The process for writing, rewriting, and collaborating on tests was slower than desired.

Hippie and the ii team– notably Devan Carpenter, Caleb Woodbine, and Stephen Heywood– came up with the mock test concept, using org-mode in their own flavor of Emacs (Humacs) deployed on a Kubernetes cluster. Zach decoupled the APISNoop database from the app , so that it could be deployed to the cluster and used as part of a test-writing environment. This allowed for tighter feedback loops, with test writers able to immediately see whether their mock test hit the endpoints they expected. These mock tests, along with their projected results as calculated by APISnoop, were then presented at the SIG Architecture Conformance sub-project meetings for initial approval, before creating the actual tests and pull requests.

The Kubernetes project is a complex organism, with a vast community and diverse Special Interest Groups (SIG’s). Conformance works across all these organizational levels, and reaching consensus on it is an equally complex task. The collaborative test-writing method of Hippie’s team helped increase the velocity and transparency of test writing.

Iterations and Automations #

Dan was always pushing the bar higher, continually expecting better and clearer results out of APISnoop so it would be a well-used tool throughout the Kubernetes community. This inspired Zach and ii to continually improve its functionality, changing APISnoop from a static page showing only the current status of conformance, to a multi-faceted tool with multiple ways to view not just the current data, but the historical progress of Conformance. Hippie developed a unique Kind+APISnoop combination that allowed anyone to access the querying power of APISnoop locally.

The increased conformance coverage was great news, and added the expected value to the Certified Kubernetes brand. At the same time, it was making life increasingly difficult for Taylor Waggoner at the CNCF to manage the Conformance Certification process. All pull requests for certification had to be manually verified to ensure it contained all tests, but the list of conformance tests kept growing with each release.

Dan approached Hippie and requested the process to be automated, and in 2020 the CNCF CI Bot was created by Berno Kleinhans and Rob Kielty. This allowed for the automatic checking and labeling of conformance pull requests, speeding up the process and reducing the human effort needed.

Thank you #

The Kubernetes Conformance journey up to this day is an eventful one, built around relationships and community cooperation, with many different contributors playing their part to help move this forward. All of this was made possible by the extraordinary vision and leadership of a friend dear to everyone in the Kubernetes community: Dan Kohn.