I spent two years trying to do what Backstage does for free

In my final position as a technical author, I took on quite a lot of developer expertise duties, since understanding and documenting the client-facing APIs meant I had turn out to be fairly educated about how the general system (and engineering org) labored. It was a bigger engineering org that inspired autonomy, and as a author with a technical aptitude, there have been quite a lot of issues that I might apply myself to. 

To trace down the folks I wanted to speak to about particular APIs, I created a spreadsheet that listed all providers, their homeowners, code repos, docs, and extra. This grew into the doc of document for providers. I attempted to create a central hub for all of our inner instruments, onboarding docs, and data. And I helped arrange an inner documentation web site, whereas attempting to implement templates and get all the improvement groups so as to add their data to the positioning. 

Backstage, a framework for constructing developer portals, would have dealt with all of those capabilities—and extra—much better than my ad-hoc options. Spotify open-sourced their homegrown developer portal, Backstage, as a result of once they used it at conferences, builders couldn’t take note of anything: “A couple of of us went to one of many cloud conferences, and so they had been demoing some instrument from inside Backstage,” mentioned Helen Gruel, [Senior Engineering Manager]. “It wasn’t the instrument itself that acquired the eye. It was Backstage, the floor the place we’d do the demo from, and all people was like, what’s it?” My ideas precisely. 

On this article, I’ll discuss concerning the issues I confronted, how I solved them, and the way Backstage solves them higher and free of charge. I additionally reached out to a former colleague, Omar Delarosa, now a senior backend engineer at Spotify who works on their ML plugin, to see how his developer expertise modified from one position to the subsequent. 

Who owns this service?

My first undertaking when employed into my final position was to doc the API we gave to integration companions. I used to be given the record of APIs we uncovered and even had some early documentation carried out for me. However discovering details about every API (past taking part in with cURL requests) was tough. Trendy service-oriented architectures just like the one I documented are likely to fragment engineering departments, particularly when every staff is given the liberty to carry out their very own work the best way they see match. 

So once I had questions, I wanted to trace down who owned the service, the place their code lived, the place their Jira ticket undertaking was, which Slack channel they lived in, and the place within the wiki their inner documentation was. Protecting observe of this for 100+ providers was a ache, so I ended up making a spreadsheet. It seems that everybody else within the org wanted this data, so this spreadsheet that I created for myself turned the doc of document for providers. 

This will likely sound like bragging, however it isn’t. Anybody who has solved the same drawback on this manner is aware of that creating the preliminary spreadsheet is just the beginning of the work you’re doing. As extra folks discovered it and located it helpful, they added their very own fields and sheets to it in order that it turned an enormous, sprawling beast. As a result of a big engineering org has quite a lot of turnover, promotions, and cross-team motion, this spreadsheet wanted to be commonly up to date, I simply created a quarterly reminder to nag the complete engineering staff. 

Backstage was initially created to deal with this actual drawback. We spoke with two members of the Backstage staff at Spotify for the Stack Overflow Podcast. Tim Hansen, senior engineer, mentioned of Backstage’s origin: “Spotify acquired into this stage of hyper-growth and we had been simply hiring like loopy. All these new builders had been approaching and also you type of lose that institutional data. Everybody doesn’t know all of the items anymore or who owns what. So it began off simply as a catalog of possession—who owns this microservice that’s breaking prod proper now, who do I contact about this?”

That is the Software Catalog. It’s not simply providers; providers, web sites, libraries, information pipelines, and extra will be discoverable there. Every staff that owns one among these items owns the configuration of their Backstage presence. They make modifications when the API is up to date, homeowners change, or dependencies are added. After you have this data centralized with the providers, you may as effectively begin including extra. 

Delarosa verified that the distinction within the two approaches was actual. “In my earlier position, trying to find a library or service’s proprietor concerned Google Sheets or possibly asking round in Slack channels that referred me to different Slack channels. It took me generally a number of hours simply to seek out a solution to a easy query about one other staff’s code. In the meantime, trying to find a service’s proprietor on Backstage couldn’t be less complicated. So long as you already know the title of the service, simply seek for its overview web page and proper there you might have all of the related hyperlinks to tech docs, homeowners, repos, and many others. in a single place. I sometimes discover a solution to a query inside minutes.”

Docs as code

Because the lone technical author in a big engineering group, as soon as I had the public-facing API docs in fine condition, it made sense to enhance the inner ones. We had a wiki, however like many wikis, it was poorly organized and poorly maintained. So we determined to spin up a static web site in MkDocs. It could pull markdown information from every service repo and routinely add them to the docs web site. This appeared like a clearly superior course of: hold the docs alongside the code so that they’ll be up to date collectively. 

The draw back was that the doc information had been handled as code. Each change, even typos, needed to have an related ticket, be reviewed as a pull request, and undergo the construct course of. The brand new web site was a web constructive, however it was nonetheless a cumbersome course of. Even after an extended strategy of getting each staff to make use of a template for the primary service web page, I’d flip up onboarding docs, structure specs, and different documentation that was created advert hoc in no matter format groups wished them in. 

As with many giant organizations, data will be scattered throughout a variety of wikis, emails, and paperwork except there’s a deliberate effort to carry them collectively. Spotify discovered the identical points as I did. They got here up with the same answer: docs like code built on MkDocs. However they solved an issue that I couldn’t: getting it multi function place. 

One other strategy to hold data collectively is for builders to specify the Stack Overflow for Teams tags that apply once they arrange a docs web site for his or her service. If there are related questions for these tags, they’ll routinely present up on the documentation pages. Groups is a good praise to a strong inner documentation effort; Backstage simply tightly {couples} that complementary relationship with a plugin. Like Backstage, it was constructed to resolve their very own inner wants first. “As we constructed out search as a part of the Backstage expertise, we wished to incorporate these Stack Overflow for Groups questions and solutions,” mentioned Hansen 

One platform to rule all of them

One of many vital issues that I noticed at my outdated job was the huge variety of instruments, providers, and data sources out there. There was no central strategy to entry them (apart from organising bookmarks to the person websites). Bookmarking solely labored if you happen to knew that these assets existed within the first place. 

After I began at my earlier job, I acquired a doc from my supervisor with hyperlinks to a number of (however not all) of the instruments and assets that we used. This included issues just like the workplace ground plan, expense processes, wikis, and extra. However there was loads of stuff that I needed to discover by myself. This can be a fairly widespread drawback as firms scale up: groups get new instruments and processes that don’t at all times filter right down to their colleagues. The issue will get worse once you empower groups to work independently.

I ultimately discovered myself onboarding processes. There was nothing formal on the time, and groups had spun up ad-hoc onboarding docs. After I wished to know the way sure methods labored, I used to be directed to people who sat down with me in entrance of a whiteboard. However that non-public contact doesn’t scale effectively when an engineering org begins rising.  “Everybody struggles with rising their workforce, extending their groups, rising their product improvement capabilities, and sustaining the developer productiveness stage,” mentioned Greul.

Gathering the whole lot a brand new rent wants into one place—their instruments, their Stack Overflow for Groups data base, and details about the homeowners of all of the initiatives they’ll contact—provides them a roadmap to changing into a productive member of the staff. And it scales. It’s not simply helpful for brand new hires although, I discovered myself referencing documentation and looking out up instruments I’ve used a number of months in the past however have since forgotten the small print. 

Less complicated developer experiences

Anybody who has seen how giant engineering organizations function is aware of that an individual or a instrument must wrangle all of the instruments, providers, docs, and data into an area the place they are often helpful to everybody. I’ve seen extra roles seem which have “developer expertise” as a part of the title, so it’s clear that firms are beginning to put assets behind enhancing developer expertise. I’m grateful that Spotify created one thing that helps a lot free of charge. I simply want I’d had it sooner. Watch our recent webinar to see how Backstage and Stack Overflow for Groups work collectively.

Tags: developer experience, documentation

More Posts