Using Advanced Analytics to automate processes and prevent failures: A data platform for Clearstream based on Azure

The challenge: Post-trading processes with data silos and local copies

Following a successful stock exchange trading session, it’s time for securities, bonds, and shares to change hands. As part of what’s called settlement, buyers and sellers transfer the legal ownership of what has been traded in exchange for payment. Managing all this is the job of the central securities depository. For Deutsche Börse Group this task is performed by its subsidiary Clearstream for the German, Luxembourgish and international markets. Clearstream guarantees a seamless and regulatory-compliant workflow while also ensuring that previously transacted securities are kept safe.

With a total data volume of 400 terabytes being processed every day, regular data reconciliation and ongoing maintenance are essential for making transaction decisions and ensuring efficient operations. “To access centralized data from diverse data sources for efficient analysis and to foster innovation and business agility, we use a data platform,” says Priya Sharma, Head of Client Connectivity & Data at Clearstream. “This provides our data scientists with an extremely fast way of finding and collecting data.”

In the past, however, the data platform was hosted locally, which in practice often led to the creation of silos and multiple copies of files. “What’s more, installing the hardware ate up a lot of time and money. The result was a longer time to market for our services,” says Abed Shareef, Head of Big Data & Advanced Analytics at Deutsche Börse. “And without a uniform data basis, it simply wasn’t possible to automate processes or harness artificial intelligence.” But Deutsche Börse Group had already recognized the potential and compatibility of Microsoft Azure for data platforms, advanced analytics and artificial intelligence (AI) through the cloud initiative it started some time ago. Michael Girg, Chief Cloud Officer at Deutsche Börse: “Thanks to the rich ecosystem of Azure services, we could start innovating right away. Post-trading processes and their infrastructure must be fast, high-performance, scalable, and able to support straightforward data exchange, while also offering a high level of security and protection for our sensitive data. Azure does all this for us.”

The solution: Azure speeds up processes and the development of new Advanced Analytics services

Today, more than 230 data pipelines ensure that all applications and core systems are connected to the new technology setup. Thanks to a uniform data basis and comprehensive process integration, silos are now a thing of the past. The new Databricks Lakehouse architecture employs a range of services to achieve this, with data from the core systems sent to the platform automatically at regular intervals—daily or hourly, depending on the use case. Data pipelines transport this data directly to Azure Data Lake Storage, a central repository for collecting and storing data. With the help of Azure Databricks, the data is then analyzed, standardized, and fed into tables, reports, or Power BI dashboards. Microsoft Purview helps create a comprehensive overview of the data landscape, recognizes this data automatically, and thus makes it easier to locate. Azure Kubernetes Service supports the containerization of the data, while Azure Functions is used for serverless applications.

“Thanks to the cloud and to automation, data is just a click away and processes that used to take several days to complete now take anywhere from a few minutes to a few hours, depending on the amount of data and the use case. Our data is standardized and retrievable from a central location, even by user groups without any technical knowledge,” Shareef says. “Moreover, workflows are becoming more intelligent, integrated, and agile. This means we can process data faster for enhanced insights and decision making.” Sharma adds: “In the past if we needed an overview of customers and business, we’d have to create an Excel spreadsheet. Now we can view clients' KPIs like intraday transaction volume, settlement efficiency over time, outstanding collateral pledged, lending portfolio utilization, performance monitoring or accounts information in our Omnia platform based on Power BI.” Key insights are now available at a click of a button. An additional dashboard in Omnia provides information about managed assets, average changes in inventory value, and credit assessments. “If a customer experiences a dip in volume or credit utilization, for example, we can respond right away. This makes it easier to have a proactive discussion with our clients,” Sharma says.

Clearstream’s IT team has succeeded in significantly reducing the effort it takes to operate, manage, and maintain the system. Capacity is flexible and scalable according to demand. And thanks to automation, applications can be swiftly set up, tested, and dismantled again. This all leads to significant cost savings. One of the biggest advantages is the parallel development of new services with predictive elements. “We pay very close attention to how settlement transactions could fail, as this can lead to huge penalties for the market participants,” Sharma says. “A tool that we developed in-house gives clients the probability of settlement failure two days in advance along with the potential reason—and that’s for up to 1,000 transactions within seconds. That means our clients can now prevent failures from happening in the first place.”

In the future, clients will also benefit from their own client dashboard, where they can see the efficiency of their transactions, for example. This makes it possible for companies to see how they stack up against their competition across markets, currencies, and asset types. These user-friendly tools can now provide analyses and insights like never before. And through the automated generation of reports, customers also face a lower risk of fines due to missing information.

To generate even more value for customers, as well as for Clearstream and Deutsche Börse Group, the new solution based on Microsoft Azure is being continuously expanded. This will support data recognition and data analysis in real time and the development of additional use cases in the field of Advanced Analytics in the coming months. “Azure provides a foundation from which we can develop our services faster and integrate even further Advanced Analytics elements,” Sharma concludes.