Lessons I Learned From Building Self-Service Data Platform | by Manvik Kathuria | Jul, 2022

Is distributed knowledge possession proper on your organisation?

Picture by Kenny Eliason on Unsplash

You stroll into your nearest grocery store, seize a basket, fill it with no matter you want, and stroll to the checkout counter. You scan the barcodes of the gadgets, pay, put them in your bag and go away the place—all of it with out interacting with a single individual within the grocery store.

Think about that what you are promoting stakeholders wish to reply a selected query. They create their knowledge into a knowledge lake or a knowledge warehouse, curate it, and create a BI report for it. As soon as once more, doing this with out relying on a workforce to allow them. Welcome to the self-service knowledge platform — “A platform to allow stakeholders to ingest, curate, share and report on the information they personal with out counting on another workforce.”

Historically, knowledge platforms have been constructed, maintained, and iterated by central groups who handle each infrastructure and knowledge. This creates a dependency and a notion of a bottleneck for brand new initiatives and necessities. To unravel this problem, loads of organisations commenced on the trail of distributed knowledge possession or, in easy phrases, multitenancy.

Every knowledge proprietor is chargeable for constructing, sustaining, and working their knowledge pipelines. The platform workforce will solely be chargeable for the underlying infrastructure and ship new capabilities to make life simpler for the information homeowners. Theoretically, this can be a improbable thought the place you empower your finish customers to be self-dependent. In actuality, it’s not all roses.

Many of the challenges with self-service usually are not because of the underlying expertise that pins the platform. Let’s talk about the noteworthy ones.

Senior administration blessing

Shifting to a self-service knowledge platform is extra a mindset shift than a expertise change. Getting buy-in out of your senior administration is crucial to the success of your initiative. This helps steer the organisation and using knowledge in the fitting route.

Picture by Anton on Unsplash

The senior management is primarily chargeable for evangelising the idea of self-service analytics. Until they see worth in it, they received’t be capable of persuade their counterparts to undertake it.

Lack of related expertise

One of the prevalent complaints I hear from knowledge homeowners is that we don’t have individuals expert in knowledge engineering in our workforce. If you consider the premise of the existence of the self-service idea, you realise that we’re empowering and enabling our customers.

Picture by Isaac Smith on Unsplash

These customers have at all times been analysts or enterprise customers who don’t come from an engineering mindset. Asking them to do knowledge engineering shouldn’t be a straightforward feat to perform. Therefore these groups who employed analysts want to begin now hiring knowledge engineers. Though it’s doable, the time it should take to rent, ramp up and begin delivering remains to be a few months.

You’d assume it’s a one-time funding to rent knowledge engineers and construct a workforce that may take care of every part. Envision if in case you have tens of such teams hiring knowledge engineers after which multiply it by the price of buying these sources. We are actually tens of expert engineers scattered throughout your organisation.

Reinventing the wheel

Each workforce has methods of working that it has adopted based mostly on backlog, priorities, and aims. The identical applies to distributed knowledge engineering groups. Implementing normal ingestion, curation, sharing, and reporting patterns is arduous. More often than not, your knowledge platform will undergo from a scarcity of widespread knowledge engineering practices.

Picture by Markus Spiske on Unsplash

This can be a substantial drawback because of the interdependence of sharing and delivering knowledge to different inside and exterior groups. A number of groups implement completely different patterns in silo to resolve the identical drawback, primarily re-inventing the wheel.

Managing entry

Handing over management of information and pipelines to a number of enterprise groups possesses a brand new problem to entry and safety of information. Irrespective of what number of guidelines you apply, there’ll at all times be a danger of information proliferation. The extra entry groups have, the extra the danger of information leakage and incorrect entry throughout the organisation is amplified.

Picture by Kyle Glenn on Unsplash

There’s a high-quality line between limiting entry and enabling customers to do their job. An excessive amount of restriction and nothing would get carried out; too much less would imply knowledge is accessible greater than wanted. Having sufficient controls to lower your operational danger and allow self-service analytics is crucial.


Probably the most potent instrument in democratising your knowledge platform is getting promoters; the one method is to socialize your thought with them. Showcasing the capabilities and the profit that your platform will carry even earlier than it will get developed will begin you on the fitting foot.

Begin small-scale self-service

As a lot because it sounds nice, asking what you are promoting groups to self-service finish to finish won’t be an ideal thought. The purpose is to allow full self-service; nevertheless, breaking it down into small steps will solely assist your trigger. Possibly begin with enabling self-service reporting and let the platform construct normal frameworks and patterns for ingestion and curation. Start by handing over the tasks to not overwhelm the enterprise groups however steadiness your current with the long run route.


In case you give a pen and paper to a gaggle of individuals and ask them to attract an image, be assured no two would be the similar. The identical applies to knowledge engineering. To stop chaos, implement normal practices and patterns and prepare the engineering groups to leverage them. The motive is to comply with the identical path if it already exists and carve out a brand new one if there’s none.

Enhance collaboration

The necessity for fixed collaboration will increase with distributed groups working in the direction of a typical purpose. Coaching classes, discussion groups, and common workshops to share findings, learnings, processes, and practices will elevate the acceptance of your knowledge platform.

Knowledge platform has develop into an integral part of enterprise technique. Small to giant organisations are hungry for insights faster than ever earlier than. Investing in a knowledge platform wants cautious thought and implementation. The very last thing you need is to create one thing sluggish and unusable. No matter the best way you design your platform, make sure that to place equal precedence on using the platform, specializing in its use by stakeholders.

“Construct a platform — put together for the sudden…you’ ll know you’re profitable when the platform you’ve constructed serves you in sudden methods.” — Pierre Omidyar

More Posts