Data management in the age of cloud

Mark Hermeling is the CTO of Alveo

For financial services organisations, migrating entire data management ecosystems to the cloud was once an ambitious goal. Today, it is a reality that many are embracing and see tangible benefits from. Over the past few years, businesses have been moving market and reference data to the cloud in an aim to reduce the costs of infrastructure and maintenance as well as increase scalability, elasticity and flexibility.

The latter is important as it relates to being able to effectively deal with fluctuating data volumes and adapting to changing provisioning requests from the business. This helps achieve increased agility and resilience and future-proof the infrastructure so it can withstand the challenges of tomorrow. When it comes to cost, moving market data management to the cloud helps bring down the spend through appropriate-sized infrastructure, centralised licensing and easily shared data sets.

The wider move to the cloud and its advantages is continuing today. In fact, according to the recent Market Research Future report, the financial cloud market is expected to reach the US$52 billion mark by 2028 and is projected to grow at an astounding compound annual growth rate (CAPR) of 24% between 2018 and 2028.

Another report from Allied Market Research forecasts that the global finance cloud market will reach US$90 billion by 2030. All of the above-mentioned benefits are certainly driving this market dynamic, and so is the pandemic which highlighted the need for infrastructural change and greater agility amongst financial services companies.

However, as firms seek to transform data management, they need a steer towards overcoming the hurdles along the way, minimising risks and ensuring the cloud migrations go as smoothly and successfully as possible.

Navigating data management challenges

Financial services firms have often followed a convoluted and siloed approach to provisioning market data to their businesses processes, which translates into high maintenance costs and unpredictable change cycles for the smallest of adjustments.

Past mergers and acquisitions are adding to the complexity and are reflected in an often highly complex and heterogeneous application landscape, causing a multitude of disparate data sources, databases and redundant data management.

Whilst external reporting requirements are growing and existing set-ups are impacting new product development, financial firms need a solution fast to untangle the costly web and unfold new opportunities that cloud migration can present.

A scattered and redundant architecture can not only inflate the running costs of on-premise applications but also prevent businesses from having quick and easy access to reliable information when they need it. It is often the case that departments do not have a single source of truth and encounter multiple data duplicates and validation rules which can lead to ambiguity and inconsistencies.

As well as high levels of manual data validation and verification, financial services firms are often lacking data consumption and sourcing monitoring and measuring capabilities. It is hard to create authoritative sources of market and reference data that can become an internal service bureau to make the best use of vendor-sourced data.

Very few firms can say they are managing and consistently using such data in an efficient, fault-proof way, which leads to further data discrepancies, duplications and higher costs. This could further be fuelled by regulatory requirements such as BCBS239 for banks to be able to identify and standardise the source and provenance for all data.

Demonstrating data lineage is difficult when there is a lack of transparency or controls. It can be hard for firms to pinpoint the origin of specific data points, leading to the value of data decreasing and becoming somewhat unusable and causing ineffective controls of the overall cost base.

For financial services firms, reference data, however, is not an area where mistakes can happen or can be managed in a substandard way. It is the type of data that helps organisations function smoothly and, if impacted, can have serious implications on the day-to-day operations, service providers or regulatory agencies.

As it involves the most complex transactions and numerous entities, contingencies and dependencies, the industry is pursuing a policy of standardising it. It is not a straightforward process as large data volumes make up transactions, there are different variations in data types and the rate of change in markets and their products is substantially high.

There is also the case of data and metadata that cannot be separated and should be managed cohesively as the contextual information defines its use cases. Permissions management and origins tracking is an integral part of data management and governance, making it easier to determine the sensitivity levels of pieces of data, what can be shared where and with whom and what are the permitted use cases from a business, content licensing, legal and regulatory perspective.

Considering the complexity and number of data challenges that financial services firms are facing today, it is becoming ever clearer that a market data management transformation has to accelerate, with the move to the cloud as an enabler of change.

The promises of greater mobility, flexibility and scalability when moving entire ecosystem to the cloud are not just empty ones

Moving data management and whole ecosystems into the cloud

As the need to shift market and reference data management is apparent, financial services firms are embarking on a journey to move whole data ecosystems to the cloud. This allows to improve efficiency in multiple areas and across processes, lowering the costs at the same time. Embracing a hybrid or cloud infrastructure can help eliminate a multitude of time-consuming manual processes and bring together fragmented systems that are scattered on-premise.

Supporting this move, data vendors are now starting to push their products directly onto cloud platforms such as AWS, Microsoft Azure, Oracle Cloud Infrastructure and the Google Cloud Platform. In addition to this, providers of portfolio management systems, trading solutions, risk and settlement systems and other applications are also migrating to the cloud, as they are attracted by the enhanced security and scalability, increased efficiencies and reduced costs that this deployment can bring.

Rather than financial institutions placing individual applications in the cloud or using specific software as a service provider to host their data management platforms, they are more frequently moving their entire data ecosystem. Data providers too increasingly make their data directly available on cloud platforms and cloud-based data warehouses.

The implications are that data management systems need to be both cloud agnostic and cloud native to optimally source, integrate, quality-control and distribute market data. That means systems need to be first designed and built to run in the cloud and to work effectively in that environment.

Otherwise, the migration will fail, let alone bring any operational or cost efficiencies. At the same time, systems should not rely on a single cloud provider’s proprietary service or in any way be locked into a single cloud vendor. Vendor lock-in in cloud computing could negatively impact a business as data sets will be very difficult to move once they’re set up and may require reformatting. There are always additional risks that vendor’s quality of service will decline over time, there will be a significant price increase, or worst-case scenario, a vendor goes out of businesses.

It is, therefore, important to reduce the dependence on a single provider from the start and opt in for a ‘lift and shift’ mentality to place data operations on a future-proof footing.

In addition, financial services firms need to consider the security element when moving their ecosystems to the cloud. Keeping valuable data safe should be a priority in today’s climate as cyberattacks are on the rise and they increase in sophistication. Thus, developing a robust information security strategy, implementing enhanced permissions management, monitoring usage and data quality are becoming critical.

Security in the cloud should certainly not be seen as a roadblock to migrate as providers have made strides and advancements in their technologies as well as their corresponding level of compliance.

The more a company automates to put more applications in the cloud, or simply more directly connect them, then data quality becomes extremely critical. This is because the process removes what is typically a manual step in between cloud and on-premise, which could act as a safety net to prevent mistakes escalating quickly into major problems.

There is, however, a solution that can ease the process. Through partial or full utilisation of vendor managed solutions, financial firms can experience a ’one stop shop’ for the end-to-end provision of market data from vendor feeds all the way through to the distribution to their clients.

Doing it once, doing it right

If firms focus on the delivery of achieving reusable assets that generate recurring value and develop sustainable and cost-effective solution, they will be well-positioned to meet their migration goals. However, this has to start with a comprehensive market data transformation plan first. This should focus on creating recurring value and developing collaborative and sustainable relationships among market data vendors, IT, and business units.

Transformation should be fuelled by the use of configuration-driven products, as these can be much more closely aligned to the specific business needs. This can apply both to the initial transformation and shift to the cloud as well as in supporting change afterwards. Configurable products will typically result in faster turn-around on business decisions about new datasets and process change.

In this process, firms need the ability to link different external data sets with their internal data sets. They need robust data quality management including lineage and audit to track data flows and explain data values. Monitoring consumption will help optimize data sourcing and identify underused and overused data sets to maximize Data ROI. It will also secure compliance with content licensing and other usage restrictions.

Adopting a managed services approach can help financial firms achieve the transformation goals, optimise their cloud environments and data usage to achieve the highest level of efficiency, addressing key migration challenges along the way and building an infrastructure that can deliver cost-effectiveness and scalability for years to come.

The right data management solution from an experienced and flexible vendor will make sure that users and applications are effectively supplied with the data they need to do their jobs. It is important to note though that vendor managed solutions must be cloud neutral to allow firms to interact with data on any public cloud platform and reap the most benefits.

Accelerating cloud migration

The ongoing migration of financial services market reference data to the cloud is nothing new, however, the process is certainly picking up speed. At the same time as data management solutions and processes are moving over to the cloud, data vendors are putting data on public cloud platforms.

Downstream, application providers are also doing the same to facilitate the acceleration. This not only creates a healthy competition and hunger for new customers, but also facilitates technological advancements and innovations that later on financial firms can benefit from. Thus, the time is now to overcome the cloud concerns and instead confidently move data ecosystems.

Indeed, this has started to happen as there is a greater need amongst financial service organisations to move their market and reference data to the cloud as they battle for a spot in a very competitive landscape. Choosing cloud-native solutions and opting it for a managed services approach can inject the needed level of competitiveness, agility and business resilience that financial companies are searching for post-pandemic.

As long as these solutions are also cloud-neutral and cloud-agnostic, not allowing any lock-ins, they will be able to deliver the scalability that financial firms need if they are to future-proof their operations.

The promises of greater mobility, flexibility and scalability when moving entire ecosystem to the cloud are not just empty ones. A managed services provider will enable firms to realise a full array of benefits that cloud can offer, from better data management and governance to streamlined processes and cost efficiencies. All whilst removing the laborious every-day task of data processing and platform maintenance.

As market and refence data plays a central role in business processes as well as finance and risk, it has to be migrated smoothly, without any interruptions or mistakes. With a trusted provider’s expertise and know-how, financial services organisations can – and will – successfully speed up their deployments and reap the rewards for years to come.