S3Corp. Tech Blog

This post originally appeared on LinkedIn’s engineering blog.

I have often been asked by colleagues about some advice with respect to open source.  In general mostly about how we are thinking about it and our approach to it.  Here is a “short” post that attempts to share some of our learnings.

Open sourced software is no longer restricted to addressing small, low-level problems and for companies that can’t afford web application development for themselves. Today it is backed by a mature community that creates and adopts it on behalf of some of the world’s biggest technology companies. Why the shift? Why should companies focus some of their efforts on open source? Why open source?

It all starts by making your engineers better. They will grow in their craft by having their work exposed to the entire community.

They will write better software
It seems paradoxical to think that developers write better software for others than they do for themselves, but it actually makes sense. When software is written “internally” developers have a tendency to cut some corners – and I’m as guilty as anyone – especially around documenting, making code easily readable and reusable and having all the right tests in order. The Open Source community has choices and will simply lose patience in trying to figure what your code does if it is too obscure. “Internally” you may not have a choice.

With open source, developers’ names are attached to the software they create and the entire community can look at it. This puts a human face on code and reputations on the line. Once a developer open sources some software, their names will be forever associated with it. Their design choices and bugs will be visible to all. This is a huge incentive to cross their t’s and dot their i’s. A developer wants to be associated with good stuff that is well written.

Every good Software Engineer is a forever learner, they simply need to be to keep up with the latest developments in their craft. Being exposed to the developer community outside of the company where they work, will help them become better aware of new or newer trends, and help them learn how to assess the value of that community’s input. Furthermore if they are the owner of an open source project, they will grow their ‘tech lead’ skills as they have to accept or reject patches from the community. A successful project will have a lot of contributors and lots of opinions. It is a fallacy to believe that there always will be a clear better technical choice to any changes. What extensions to take or not could be their decision and that can be tough at times which gives them the ability to build new skills. Many open source message boards are animated with passionate opinions bordering on flame wars.

From a company’s perspective it helps develop your ‘engineering brand’
There are many ways to develop an engineering brand – a signature approach to development that’s like a stamp saying: “We made this.” Open source is a great way to open up code to other developers. It may not be exactly what a team runs in house, but it can definitely show some aspects of the work that went into it. It’s also a package that senior individuals can create for candidates to look at. It’s an explicit sign from an organization to prospective engineers that they’ll not only be able to develop their craft if they work there, but they will be able to reuse the stack that they like instead of having to reinvent it.

Of course, with every argument for, there’s also an argument against. Open sourcing isn’t the right path for everything and every company.

Don’t open source because you need more resources to work on a project
This is a common mistake early on. The logic usually goes that if there’s a great project that could use extra help, open sourcing it will get the entire world of developers to start contributing to it. This is a fallacy for two reasons: if the project takes off, it will end up with tons of requirements from the external community that may not fit the needs of the organization building it, which ends up requiring additional energy and time to code. If the project doesn’t take off, that’s a lot of time spent packaging it nicely for nothing. It’s also hard to predict whether the individuals or communities coding to it and submitting patches are good developers or not. It’s a wild world out there. Successful open source projects require a significant investment of time to develop, monitor and nurture in the Viet Nam software services.

Don’t open source just because you can
I often refer to this as ‘be a good citizen’ and do your work. Before starting any project, be a good citizen and investigate what are similar solutions – if any – that the community is already working on. Then do your work in evaluating if contributing to an existing project would be better. It’s often far more valuable to use and contribute to existing projects instead of reinventing the wheel.

Open sourcing bad projects will have a counter effect. People will look at it and wonder why that engineer or team would create something sub par if there is already an open source solution that is superior. This can create the exact opposite effect of all the reasons above.

How to build a successful open source program
It’s not easy, and to be honest, LinkedIn had some early stumbles. One lesson we learned early is that, in order to do open source, a team can’t just put software out into the community and not continue to innovate. Many people assume that the community will take a project forward for you once you open source, and that does happen on few rare occasions. For the most part, you have a responsibility as the creator to maintain projects as they develop in open source, and to spend as much time on it as if it was still part of your internal organization.

Another lesson we learned was that, more often than not, smaller projects should be folded into larger, existing projects. LinkedIn’s first open source projects were search components we built on top “Lucene”, our core search engine at that time. By not closely aligning our extensions with the existing developer community that already existed around Lucene, most of these extensions never gained traction and fell by the wayside – even though many of them were very good projects. Standalone open source projects are great, but if it makes sense to put a project under the umbrella of an existing Apache project, don’t hesitate. If there’s an existing community of developers eager to work on new projects it will give them a much higher chance of success.

When we built Kafka later, we learned from our past errors and judiciously invested in it and it became our golden open source project.

Lastly, if a company is going to open source something, they have to be ready to eventually fork it. The first time I forked one of my projects I felt really uncomfortable. I didn’t sleep well over the weekend after making the decision, and felt ashamed … like a traitor. Later on, however, I understood: Once a project is open sourced, it has a life on its own. And at times, it can evolve into something that you don’t want or didn’t intend, and becomes incompatible with your stack. When it happens, you’ll have no choice but to fork it. It is natural and can happen. Your baby has matured, let the adult have its own life.

That shouldn’t scare anyone off, though. Open sourcing is very much worthwhile and fun to create, manage and be a part of.

Source