Congratulations! Whether you have been promoted from developer to a leadership position or hired to start a data team, Data Leadership is one of the most rewarding and challenging jobs in the world.
Data has historically been an area overseen by others with more hype, and it is often treated as a byproduct of organisational processes. Your first movements as a data leader will depend on the success of the data area and on changing the mindset of immediate leaders and the organisation.
In this series, we'll cover the different phases and steps to follow to succeed as a Data leader, providing you with the tools and framework needed.
Understanding Organisation
One of the common pitfalls when working in data is not being able to prove value to leadership (who very often sees the data area as a cost and not as a value provider). This can be easily solved by aligning your first initiatives with the organisation's key goals. Starting this alignment early will provide a strong foundation for proving value and gaining traction from leadership in your organisation.
Within your first days in the position, one of the main goals you have to set for yourself is to better understand your organisation, business and goals. This knowledge will come from having a series of meetings with leaders in your and other areas. Understanding each area, how they contribute to the organisation, and their goals will help you to start having a mental landscape picture and give you ideas of which initiatives are the most important in your organisation.
Once you have a better understanding of the different areas and their goals among your data leader tools, you should build your own matrix with the goals and how they are evaluated in said areas. These documents will be for yourself as a reference and not to be shared, but they will become your source of truth and reference to have a quick overview of your organisation. From this first document, you will have a quick summary of the proxy metrics that contribute to the North Star of your organisation. For your first initiative, it will be key to align it to one of these metrics to easily translate it into a contribution to the North Star of your organisation. Later in this series, we will go deep into how to choose your first data product, but for now, first lets’s create your organisation metrics matrix.
Input in this phase: Interviews with leaders in your organisation.
Output in this phase: Organisation Key Metrics Matrix.
Status Quo of Data in the Organisation
Once you have a better understanding of how success is measured in your organisation, the next step is to understand the Status Quo of data.
Starting at the source, create a list of processes in the organisation and the data they generate. Information like the technology, volume of data, historical data stored, and regulations applied will be useful for creating a good picture of how data is generated. For each one of these data sources, interview any team or area consuming that data to understand their necessities, technology used and end-use.
Now, a small reality check. You may not find information in all the areas for the next table. Our recommendation is to try to fill all of them. If any documentation or additional information is missing, create your own documentation for reference later. Having a minimal reference or document for any of the segments in the table will be extremely helpful in your position.
Data Producers (Processes)
Data Consumers/Producers (Teams)
Having those two tables for all the teams producing data and teams consuming data will come in very handy when looking at the full picture of how data is being generated and consumed in your organisation. With the interviews and conversations with different teams, you will start getting hints about pain points and bottlenecks in how data is produced and consumed. Keep an eye open, as these may be quick wins where your team can step in as a first initiative to gain leadership trust and support.
Input in this phase: Interview with Teams and Process Owners.
Output in this phase: Data Producers and Data Consumers Matrix.
Understand Your Customers
Data Engineering is like an infinite space where you can use a wide range of architectures, technologies, and programming languages. It’s a playground for engineers who seek challenges and find great joy in their work. Sometimes, we even forget that, in the end, our work has a real business purpose: to generate revenue for the company. In most cases, your team will be delivering data products or supporting other teams’ products. Therefore, before you start operating, it’s crucial to know who your customers will be, or at least who your potential customers are. In the first weeks or months in your new role, organise meetings or workshops with key members from each team. In these meetings, you can gather all the relevant information and build a knowledge base that can be utilised for your future projects.
First, identify and understand the specific business goals of each team, as these may align with the broader organisation's objectives in a unique manner. Seek to obtain metrics and KPIs related to these goals to have an idea of how the team is performing and has performed over time. Inquire about objectives that the team finds challenging to achieve and explore the reason behind these challenges. Also, review the goals that were achieved with minimal difficulties. This information can be taken from various team members, including both leaders as well as developers, to gain diverse perspectives. These discussions will uncover the primary focus of your customers, helping you to better understand their strengths and areas of difficulty where your team might need to focus additional efforts.
Once you’ve discussed the goals, engage with the leaders about the strategy and vision for both the organisation and their individual teams. Drive the discussions in a way that reveals their approach to achieving these goals. What is their methodology of work, how do they allocate resources in the projects, the steps taken to reach intermediate and final targets, and how do they prioritise tasks? Which project strategy is used within the teams, whether they use Scrum, Kanban or any other methodology? Additionally, ask about the team’s future: how they envision the team’s evolution in the coming years, which would be the most important changes they expect to happen in the coming months and years. Gathering this information will help you to adjust to their working style, better understand the expectations, and also anticipate the types of work that may be required in the future.
When discussing strategy, ask questions about who is who in the team and what is the Team’s structure. Focus specifically on key functions such as project leads, architects, and product owners - essentially, those responsible for making project decisions. Remember, collaboration is key, and it’s extremely important to start building relationships at an early stage. Therefore, do your best to organise an intro meeting with different members of the team; it could even be a quick coffee chat where you can talk about anything. Determine the scope of each role, what they are capable of doing, and whether they need help from outside the team, such as external consultants or from different internal teams. What communication channels do they use, whether they prefer emails, communicators or calls. How do they escalate any problems, and to whom? Engaging in these activities will certainly make your work more efficient by understanding the strengths and weaknesses within the team and how they communicate and handle any challenges. But, what is most important, it will build the foundation of your relationship with customers and build trust, which is essential for successful project collaboration.
After discussing the goals and strategies, it’s time to understand how they are realised and converted into actual Products or utilised in processes. First of all, you should understand what business value they bring. Also, collect information on functional/non-functional requirements, quality attributes, architecture diagrams, whether the product is for internal or external use, how and who maintains it, and who the end user is. These activities will give you more comprehensive knowledge about the product, but also, what is very important, it will give you additional information about the team’s methodology of work. Examine how they plan their work, whether they consider all relevant facts, and how they reach an agreement on the final outcomes. The next step is to explore what Technology has been used; it’s probably equally important to previous activities and perhaps the most interesting aspect for data engineers. Finally, ask about the major Pain Points currently associated with the products. This information will help you to identify areas needing improvement and where to focus more attention.
Customers characteristics matrix
Input in this phase: Workshops with multiple teams, which are the future or potential customers
Output in this phase: Customer characteristic matrix
Technology
Having a comprehensive knowledge of all the topics mentioned above is crucial for building a team that will successfully create or support various data products. Most of these items are typically located in the “business area” and would be discussed by leaders, product owners, architects, and other business-oriented roles. However, in order to materialise all the strategy and business goals into data products, we need Technology and Engineers who will use that technology to build these products. To get in-depth knowledge of the technology within the organisation, you will instead meet with architects, tech leads, and various engineers specialised in DevOps, data, or software. It’s important to determine where the solutions are located, whether on-premises or on public clouds like AWS, Azure or GCP. Once you know the deployment model used (e.g. cloud,on-premises or hybrid), review the category of services employed, such as IaaS, PaaS and SaaS. This is basic information, but it already gives you an idea of what type of skill set you need in the team. For instance, your team might require expertise with a particular cloud provider, or you might need engineers who specialize in Infrastructure/DevOps or Data, depending on the service models utilized, such as IaaS, PaaS, or SaaS.".
During conversations about the products, try to get documentation about them (e.g. solution architecture document). Ask what the reason was for choosing specific technologies and if the organisation follows any reference architecture documents. Talk through each individual component, like data storage and processing, access layers (e.g., dashboards or APIs), and monitoring. It’s also beneficial to explore whether there are any managed platforms (e.g., data & platform orchestrators, data quality, and data management frameworks) already in place that your team could leverage in your solutions.
Additionally, review their deployment processes to determine whether they are conducted manually or through Continuous Integration/Deployment (CI/CD) processes. If they are in place, inquire about the software used for the version control system (e.g., GitHub or GitLab) and CI/CD (e.g., Jenkins, Github Actions, and others).
Technology Matrix
(Note: Components which other products can use can be highlighted in specific colours)
Input in this phase: Workshops with technical stakeholders
Output in this phase: Technology matrix
The information in this section might not cover everything you need to build a complete team or create a successful product. However, it should give you a fairly good understanding of the technology used within the company, the technical skills you should look for in your team, and which components you might use in solutions developed by your team.
Summary
In this first article of our series we went through the first phase in your journey as a data leader: the discovery phase. Your focus in these first days should be directed towards getting a proper understanding of your organisation (if you were promoted) or new organisation (if you just started a new job). Use the templates provided in the article to materialise this understanding into:
Organisation Matrix
Data Producers/Consumers Matrix
Data Customers Matrix
Technology Map
These templates will help you in two ways:
As a guide to help you understand your organisation's data.
As reference notes to use in your role as data lead.
Follow us, and in the next articles of this series, we'll discuss the next phases, providing the tools and framework to ensure your success in your new role as a data lead.