Our client, a major American music corporation, is a production house of media and entertainment data. It receives regular feeds of data on digital and streaming sales from its distribution partners Spotify, Apple, Pandora, Rhapsody, You Tube and so on. More than 50 % of its sales is obtained through online channels and other digital content in the music industry.
This data is further used by the marketing and finance departments, to analyse customer usage patterns and gain further insights from country-wise sales of albums and recording labels, downloads and shares of popular tracks and so on.
While trying to achieve its business needs, the following challenges placed an impact on the performance and scalability of the existing system:
To address the above challenges, Agile GCP Labs, a Google Cloud Partner, functioned as a suitable offshore partner for the client, to support, architect and implement an effective on-prem migration operation from MS/Oracle on-prem database to cloud-based Google Cloud Platform (GCP). The key components of the implemented solution are as follows:
The data from the distribution partners was obtained via API support and transferred to Redshift, using Amazon S3 Data Lake. From here, it was further migrated to Google Big Query and stored into GCP’s cloud-based enterprise data lake, Google Cloud Storage.
Through Google Big Query, a Unified Source of Truth was established for the users, to combine the enormous incoming data from multiple data sources and provide seamless integration onto a single platform. The ETL tool, Informatica ETL, was replaced by GCP’s Google Cloud Dataflow, to process the unbounded raw data obtained from the client’s streaming partners.
This new cloud-based system comprises a flexible and scalable data architecture, which supports powerful visual analytics and offers near real time data usage updates.
The sales data from the physical and digital world were consolidated to provide a single view of product sales. A centralized enterprise dashboard was created to handle and create over 30,000+ Enterprise level reports and 50+ dashboards into one reporting platform, while using end user information for reporting and visualization operations. With the newly established system, the client can now obtain deeper insights on music sales from over 100 classifications such as Artists, Geography and Partners, for its music consumers.
Through this On-prem to Cloud Migration (SQL Server to Google Big Query) the data loading & processing times have been reduced significantly and businesses can now share insights freely. As GCP offers Cost-Effective features such as free usage limits, up to the second billing, consumption-based charges and pay as you go pricing, the customized reporting operations on Brand interests, ad recall lifts, unique users and view through rates for global music genres have been improved.
Agile GCP Labs previously known as Agile iSS was founded in 2014 and recently rebranded in 2019 as Agile GCP Labs.