by Jarek Potiuk


During my career, I have been a software engineer, Tech Lead Manager at Google, a robotics engineer at an AI and robotics startup, and am currently the Principal Software Engineer of a software house, Polidea, which I helped grow from 6 to 60 people within 6 years as CTO. Over the past year and a half I was a user, then contributor, then committer, and now a Project Management Committee (PMC) member of Apache Airflow.


Although I took on many roles through the years, including being the main organizer of the international tech conference (MCE), deep in my heart I was always a software engineer. It took me many years to find a place where I could explore my true potential. Then I became part of the Apache community. I first learned about the Apache Software Foundation (ASF) 20 years ago when I used the Apache HTTP server at the beginning of my career. I had only made small contributions to OSS projects up to that point, and becoming involved with Apache Airflow was the first time I contributed seriously to one. As a Principal Software Engineer at Polidea, several of our customers were using Apache Airflow and wanted to contribute back to the project to help other users. Better integrations of services with Airflow would significantly improve future releases of the software. 


The needs of our customers made me and my team go from users of the project to contributors and more. We have people in our software house who understand open source, know how to follow the OSS rules, and contribute changes from customers to help other people in the community. We know how to communicate well, we can also represent a vendor-neutral point of view because we represent the view of several customers and collaborate with all the stakeholders in the project. People in our company also contribute to other OSS projects, such as Apache Beam and Flink. 


We also discovered a great model where our customers wanted us to contribute to an open-source project to make it better because they were using it and wanted to improve it for future users. This allowed us to do it full time (or even 150% of the time if you add all the out-of-hours contributions). I invite you to read about it in my blog post The evolution of Open Source - standing on the shoulders of giants.


Committing to Apache 


I found exactly what I was looking for in the Apache Software Foundation. It’s a great organization for people like me: individual contributors who are also good at working with others, the ones who don’t shy away from organizing and making things happen, who thrive when they can do meaningful work with others. 


This made me think: since the ASF is so great, how come for 20 years I was not contributing to OSS projects more? And since so many software engineers use Apache technology, why is participation not more common? I got lucky because I was in a position that allowed and supported my contributions to an open-source project. For me, it’s a dream-come-true. But what about others? There must be more people willing to contribute and get involved in the OSS community, they probably just don’t know how to go about it yet or did not like the experience.


So here I am, sharing my thoughts on what can be done to help others to get to know ASF sooner and get involved.


Apache Airflow and the initial experience 


Apache Airflow is an exciting project. It is a platform created by the community to programmatically author, schedule and monitor workflows. It started in AirBnB in 2014, was submitted to the ASF incubator in March 2016 and it graduated to a top-level project in January 2019.


When I started working with the Apache Airflow project, I quickly realized that it was hard for me to contribute to. It was not clear how to develop and debug Airflow, how to start, and how to communicate. The project had a number of channels for communication including a developer list, a Slack channel, issues and pull requests alongside the code. As a newcomer, it was not easy to understand which channel is used for what and whether it’s OK to raise certain issues using those channels. It was not clear what were the common protocols: for example how to see that one thread is a discussion and one is voting on an already discussed topic. 


Is our community welcoming enough?

At a party after a conference where I spoke about Apache Airflow, I had a long discussion with a young engineer who was new to the field, Fabian. Fabian told me that often OSS projects create some invisible barriers around communication and onboarding. I explained to him the “Apache Way” and how transparency and openness help with those barriers, fiercely protecting the fact that “we are open”. We carried on with our friendly discussion and it was really eye-opening.

That conversation stayed with me for a while. After some time, I realized that maybe our project was not as welcoming and accessible as we thought: there should be an easier way for people to contribute and join the community. I recalled my case—when I joined the community, I made a mistake by writing that something “will happen” before discussing it with the community. A long-time community member reminded me that this is not the way we should communicate at Apache Airflow. Just to note - each project is autonomous within Apache and it defines its own communication rules. It was done in a very good and friendly tone and I took it as a lesson, but some people might be put off by such a response. Not everyone has the determination, experience, thick skin and willpower to overcome all the obstacles and some people might be put off by such responses - even if they are nice and friendly. 

Could we do better to communicate the ideals of our community more straightforwardly? Without coming across as harsh? Maybe we could find a way of explaining to the future contributors how they should communicate rather than do it by trial-and-error?


Becoming a more welcoming community

A few days and emails after the discussion with Fabian I started a thread at the developer’s list of Apache Airflow “[NON-TECHNICAL] [DISCUSS] Being an even more welcoming community?” that kicked off a conversation that included people who rarely had spoken before. As a result, we managed to introduce many changes to the processes for new members. Thanks to the input of people such as Karolina Rosół (Project Manager at Polidea), we came to the conclusion that the way seasoned community members communicate at Apache Airflow is not obvious at all to newcomers. We added missing chapters to our CONTRIBUTING documentation regarding communication channels, expected response times, and more. 

What helped a lot was that we were able to improve our documentation for the development process during last year’s Google Season of Docs program. Apache Airflow was one of the first projects at the ASF to participate in the program. I was one of the mentors to Elena Fedotova, a technical writer assigned to our project. She improved and restructured the documentation, and made it more readable and easier to understand. Many people took part in reviewing and correcting the docs. Also, we took on the task of creating a new website for Apache Airflow with modern, clean design, and well thought UX addressing different personas of visitors in mind (including new contributors, users and potential partners of Apache Airflow). One of my colleagues and fellow Apache Airflow committer, Kamil Breguła, put enormous effort into both building the website and also restructuring the documentation. 

As a result of the discussion at the developer’s list, we also introduced the mentorship options and even handled (via mentorship) a few difficult cases that could have lost us valuable contributions. A great example of the improvements we’ve done as a community might be this tweet from Vanessa, a research software engineer who had no experience with the community.  Vanessa had tried to contribute support for Singularity—a popular container technology for high-performance computing - a year earlier, and came back for a second try after much of this work was done:


https://twitter.com/vsoch/status/1231523084026253312


Are we there yet?


Looking back, it’s been a long (yet satisfying) journey trying to make the Apache Airflow community more welcoming. But how do we know it works?


At the beginning of this year, we started to participate in the Outreachy and Google Summer of Code programs, where people from around the world with different backgrounds can be paid for contributing to open source projects. Together with my friend and PMC member Kaxil Naik, we became program mentors and started to receive a flood of requests from the Outreachy members. Initially, we were overwhelmed but soon realized that we have everything we need to answer the questions of the candidates (and future committers) to let them teach their lessons and easily follow the “contributing” documentation. The contribution environment was available for them to get started, and the documentation detailed how they could learn how to prepare contributions and communicate via various channels.


Just two days later, we approved a few pull requests from those people! That’s quite a difference from 1.5 years ago when it took days, if not weeks, to understand the environment and how to work with Apache Airflow. It was truly a team effort; many community members participated in the process and made the Airflow project much more welcoming to newcomers.


Despite having challenges during our experience getting started, I was never going to quit. I loved the project and people almost from day one. Realizing how hard it was initially to start contributing (other people had told me so as well), I decided that I would put a lot of effort (both professionally and also personally) into making the project easier and more open and accessible for people with different backgrounds and experiences. My experience starting as a contributor, then becoming a committer, and now a PMC member proves that this is possible.


To me, Success At Apache means making the community and the spirit of Apache Way more accessible to people around the world. With the difficult times that we are going through now with COVID-19, it’s more important than ever to build and strengthen various communities. And to strengthen the community means to be open to others and be welcoming, We hope that our experience will encourage you to take a look at your project and see if you can make your community more welcoming.


# # #


Jarek Potiuk started to work on the Apache Airflow project in September 2018. He became an Apache Airflow committer in April 2019 and a member of the Apache Airflow Project Management Committee (PMC) in October 2019. He is an Apache project mentor in Outreachy and Google Summer of Code and was a mentor in Google Season of Docs. Jarek is a Principal Software Engineer at Polidea and always keen on making it easier for people with different backgrounds to join OSS projects.


= = =

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works" https://blogs.apache.org/foundation/category/SuccessAtApache