Tapabrata Pal on DevOps at Fidelity: Investing in Inner Source and Engineering Excellence

Tapabrata (Topo) Pal, VP Enterprise architecture, and co-author of “Investments Limited“, presented how Fidelity improved their DevOps strategy by investing in Inner Source, Open Source & Engineering Excellence.

After having spent 10 years at Capital One, Topo joined Fidelity in March 2021, mainly because he believed in their company's Culture Code focused on “happy and engaged people delivering customer value”. In addition, Fidelity Culture Code states:

We believe in leadership at all levels and that titles don’t determine what you can and cannot do.

No roles, only role models.

At Fidelity, Topo took on the responsibility of their DevOps domain architecture and their open source program office. He saw that the company had been successful in implementing a multi public cloud and hybrid strategy. From an agile and DevOps perspective, Fidelity deployed their first application on the public cloud in 2016, and in 2017 they kicked off their organizational agility initiative, which enabled 50% of their applications to be on the cloud by 2022. With 18,000 developers across 1,600 squads, Fidelity has the goal to move 70% of their applications on the cloud by 2024.

To do this, soon after he joined the company, Topo conducted a listening tour, where he interviewed several leaders across the organization. To every leader he met, he asked the following three questions:

• What is going well today?
• What is not going well today?
• And what can we do to make it better?

Topo then sorted out and analyzed the challenges he heard, and identified four key opportunity areas:

Developer Experience:

• Onboarding is a challenge due the volume of their processes
• Documentation: they had a lot of documentation, more than they’d ever need
• Context switching exacerbated with the above and the number of tools available

The Tool Sprawl:

• Too many tools, technologies, and platforms, which is according to Topo is not just a challenge at Fidelity, but an industry problem.
• They have more tools that they can consume, leading to too many abstraction layers, frameworks and complication when upgrading.

Security, Audit Compliance:

• Shifted right leading to late feedback loops
• Time consuming audit
• Ownership. The main difficulty according to Topo is ownership. To define the future state, Topo asked: “Can the developers own the security and availability themselves, and can they decide what to do with them? Can they get feedback early enough in the development lifecycle?”

Metrics:

• Lots of metrics. Here as well, Topo mentioned that this is not just a challenge at Fidelity, but a problem that affects the industry
• Lots of different techniques used to measure
• And too many maturity models

In terms of metrics, Topo mentioned:

“We are comparing apples to oranges, and between apples and oranges it's very difficult to tell where the team is in their DevOps journey.”

With these findings, Topo met with Fidelity's CIO and together they agreed to take a collaborative approach, starting by getting everyone together to discuss and design the next steps. They formed the Fidelity DevOps Council, including participants from all functions across the organization. They then formed small groups, each tasked with identifying solutions to these challenges. They reported back to the council, presented their proposed solutions and discussed the next steps.

The DevOps Council decided to invest in engineering excellence and focused their efforts in the following areas:

• A Unified Developer Experience
• Tools standardization
• Continuous compliance
• Contextual metrics

 By “contextual metrics” the DevOps Council means “measure what matters to us, that matters to a particular set of developers, particular set of products, business and so on.” For each metric proposed they would answer Why, Who, When and How. They were cautious about the “observer effect”: they didn't measure things that might have affected the behaviors of the people, instead they promoted “measuring to improve ”. Topo proposed that every request to measure something should be stated in the following format:

“As a (role), I want to measure (this process), so that (benefit and improvement)”

And he shared the following example:

“As a Product Owner, I want to measure Team's Production Access Frequency, so that I can understand Team's Technical Maturity and help improve it.”

To support this effort without having to stand up a shared platform, the Fidelity DevOps Council invested in inner sourcing and Fidelity brought in open source to scale up. They developed an open-source framework focused on “Incubate – Fund – Govern – Manage Promote – Operate”. In 2021, they kicked off 5 new projects, among which one of them produced to date 174 features, 3500 commits, supporting 12 business units, with 50 active contributors.