Suffyan Asad

Data Architect & Engineering Lead | Enterprise Data Platforms | AWS, Spark, Databricks, Snowflake & AI

Suffyan Asad, Data Architect & Engineering Lead

Summary

Data Architect and technical leader with 13 years of experience designing data platforms and analytical solutions across FinOps, Data Science, AI and advanced analytics. Led multiple projects including data and architecture design, stack modernization, enabling data science, AI and advanced analytics, and scaling and performance improvement of platforms processing billions of rows from multiple data sources. Built solutions with real impact: cost visibility and savings for companies including Hulu, Klarna, Roblox, Okta, Zoom, and Compass.

Certifications

Employment

CoreStack – Senior Data Software Engineer

Halifax, Canada · – Present

  • Led architecture, benchmarking, and POC development to replace MongoDB with a sharded ClickHouse warehouse for multi-tenant cost analytics, scaling up processing capability by 50% and supporting more than 1 billion rows per run.
  • Utilized Claude Code with curated, context-aware Claude Skills and hooks to run reproducible POCs, preserve evidence-backed architecture decisions, generate design documentation, and oversee software development of data processing pipelines.
  • Directed customer migration from the legacy platform with 100% retention, mapping feature parity and building single-trigger validation pipelines that reconciled cloud costs, organizational hierarchies, Kubernetes workloads, anomalies, forecasts, and recommendations against cloud-vendor source data at scale.

Tools and Technologies: ClickHouse, MongoDB, Python, SQL, Docker, Docker Compose, Kubernetes, Helm, Amazon EKS, Claude, Claude Code, Claude Skills, Claude Code Hooks, PySpark, Ray, Apache Arrow, Parquet, DuckDB, Amazon ECR, Amazon S3, AWS IAM, AWS, Amazon CloudWatch, Azure, Google Cloud, Kubeflow, Apache Airflow, Amazon ECS

Yotascale – Senior Data Engineer - Architect and Lead

Halifax, NS, Canada · –

  • Led product design, architecture, estimations, and delivery from Halifax (near-shore), collaborating with teams in Lahore and San Francisco.
  • Engineered a data processing solution that ingests FinOps FOCUS 1.1 data sources at scale, up to 2.5 billion rows per run, into a unified Total Cost view.
  • Maintained a Generative AI assistant with ReAct-based data retrieval, data summarization, and comparison capabilities on financial data.

Tools and Technologies: Apache Spark, PySpark, Python, Apache Airflow, Datadog, Amazon EMR, Amazon S3, AWS Glue, Amazon S3 Tables, TypeScript, OpenAI, ReAct, SQL, ClickHouse

Yotascale – Senior Data Engineer - Architect and Lead

Lahore, Pakistan · –

  • Led product design, architecture, estimations, and delivery from Lahore, collaborating with the product and engineering teams in San Francisco.
  • Built FinOps and cost-observability products used by enterprise companies including Hulu/Disney, Intuit, Klarna, Roblox, Compass, Jefferies, ClickUp and Okta, optimizing more than $1 billion in annualized cloud spend.
  • Extended cost processing pipelines to support Azure, GCP, container cost allocation (Amazon ECS, Amazon EKS, Google GKE, Azure AKS), and cost allocation to rule-based business hierarchy. Scaled to support up to 2.5 billion rows per run.
  • Optimized Spark clusters to handle over 2.5 billion rows by decomposition to idempotent steps with short lineage, data bucketing and partitioning, and fixing data skew, cutting runtime by 30% and cost by 15%.
  • Architected and developed container rightsize recommendations, using cost and container metrics to analyze actual and allocated resources, and compute optimized allocation with estimated cost saving.
  • Built time-series forecasting that generated 24-month forecasts at all levels of the business hierarchy, optimizing the models with cross-validation and periodic back-testing to reduce RMSE and MAPE error metrics.
  • Led pilot migration of cost processing to Databricks using Spark, Python, Lakeflow Pipelines (formerly Delta Live Tables), Jobs, Unity Catalog, and Declarative Automation Bundles (formerly Databricks Asset Bundles).
  • Led an engineering pod of up to 5 in-house and remote developers, owning onboarding, technical mentorship, and delivery across distributed engineering teams.
  • Introduced CI/CD with Bitbucket Pipelines for building and deploying Docker images, with SonarQube quality checks.
  • Led the effort to introduce automated testing in the data processing codebase, raising coverage to over 70% in one year.
  • Implemented multiple features in the REST backend.

Press release: Yotascale surpasses $1B in annualized cloud spend under management (2022)

Tools and Technologies: Python, Scala, Apache Spark, PySpark, Amazon EMR, Amazon S3, Amazon SNS, Amazon SQS, Kubeflow, Terraform, Docker, Helm, Azure, Google Cloud, Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE), Amazon ECS, Amazon EKS, Apache Airflow, AWS Glue, Amazon S3 Tables, ClickHouse, PostgreSQL, Amazon RDS, Amazon Aurora, Java, JVM, Azure Data Lake Storage (ADLS), Microsoft Entra ID, Google BigQuery, Terragrunt, Istio, Looker, Amazon ECR, Kubernetes, Prometheus, Grafana, facebook-prophet, ARIMA, LSTM, forecast (Python library), Flask, Databricks, Databricks Lakeflow Pipelines (formerly Delta Live Tables), Databricks Jobs, Databricks Unity Catalog, Databricks Declarative Automation Bundles (formerly Databricks Asset Bundles), Bitbucket Pipelines, SonarQube, pytest, CI/CD, Git, GitHub, FastAPI, REST API

Yotascale – Data Engineer, Founding Team Member

Lahore, Pakistan · –

  • Architected and developed cost data processing pipelines for AWS.
  • Developed and scaled anomaly detection using statistical analysis, functional approximation, outlier detection, and LDA to detect cost spikes across 6-8 billion rows of cost data. One detected spike helped Zoom save $600,000 in runaway cloud costs.
  • Built Reserved Instance recommendations that simulated AWS's RI application algorithm on hourly EC2 inventory and optimized convertible RI exchanges with a mixed-integer linear program. Provided near-real-time cost estimates bridging the 2–3-day lag in AWS billing data. It was a key feature leading to successfully closing the deal with Intuit.

Case study: how Zoom manages costs in the face of skyrocketing infrastructure spin-up

Tools and Technologies: Python, Scala, Apache Spark, PySpark, Amazon EMR, Amazon S3, Amazon SNS, Amazon SQS, Kubeflow, Terraform, Docker, Helm, Apache Kafka, Amazon Kinesis, Amazon Data Firehose (formerly Amazon Kinesis Data Firehose), Amazon Redshift, Amazon Athena, Java, Amazon RDS, PostgreSQL, MySQL, AWS Lambda, PuLP, pandas, Amazon EC2

Strategic Systems International – Senior Software Engineer

Lahore, Pakistan · –

  • Developed SSAS warehouse (cubes) and MDX queries to provide data to the analytics dashboards.
  • Optimized reporting and UI queries to respond in under 1 second.

Tools and Technologies: SSAS, MDX, SQL Server, SSIS, ASP.NET, ASP.NET MVC 4, SSRS, SQL, T-SQL

Strategic Systems International – Software Engineer

Lahore, Pakistan · –

  • Built business intelligence reports using Microsoft SQL Server, T-SQL, and SSRS.
  • Engineered a report generation tool using SSRS, facilitating custom report creation with user-friendly wizards and SQL capabilities for power users.
  • Customized Microsoft Master Data Services to support data-quality management for master data across key dimensions.

Tools and Technologies: SQL Server, T-SQL, SSRS, SQL, ASP.NET, ASP.NET MVC 4, C#, Microsoft Master Data Services (MDS)

GenITeam Solutions – Software Engineer

Lahore, Pakistan · –

  • Part of the iOS (iPhone and iPad) application and game development team.
  • Created and delivered a wound-tracking application for iPhone and iPad that measured wound surface area accurately and tracked healing over time.
  • Improved vehicle physics in a Unity 3D driving game, delivering better off-road handling while enforcing the game's boundaries.

Awards: Rising Star

Tools and Technologies: iOS, Objective-C, Unity 3D, C#

Skills, Tools and Technologies

Data Architecture and Engineering
Enterprise Data Architecture, Data Platform Architecture, Solution Architecture, Data Warehouse Architecture, Data Warehouse Design, Data Warehousing, Lakehouse Architecture, Dimensional Modeling, Star Schema, Snowflake Schema, Slowly Changing Dimensions, Schema Design, ETL/ELT, ETL Architecture, Data Pipeline Development, Idempotent Pipeline Design, Lineage Management, Pipeline Orchestration, Data Integration, Data Ingestion, Data Ingestion Patterns, Large-scale Data Ingestion, Large-scale Data Processing, Data Transformation, Semi-structured Data, Data Partitioning, Data Bucketing, Sharding, Distributed OLAP, Multi-tenant Analytics, Data Governance, Data Quality, Data Validation Pipelines, Data Reconciliation, Automated Validation, Master Data Management (MDM), Data Loading and Unloading, Data Security, Access Control, Data Protection, Data Collaboration, Platform Migration, Migration Planning, Feature Parity Analysis, Architecture Evaluation, Architecture Decision Records, Benchmarking, Proof of Concept, Reproducible POCs, Scalability, Performance Engineering, Query Optimization, Query Plan Analysis, Query Pattern Design, Spark Performance Tuning, Data Skew Handling, Cluster Sizing, Caching
FinOps and Cloud Cost Management
FinOps, FOCUS 1.1, Cloud Cost Observability, Cloud Cost Analytics, Cloud Cost Optimization, Cost Management, Cost Allocation, Business Hierarchy Modeling, Rule-based Allocation, Container Cost Attribution, Usage-based Allocation, Container Rightsizing, Kubernetes Cost Allocation, Resource Utilization Analysis, Container Metrics, Optimization Recommendations, Cost Savings Estimation, Cost Estimation, Cost Data Modeling, Cloud Billing Data, Reserved Instance Optimization, Convertible Reserved Instances, Spot Instances, Budgeting and Forecasting, Anomaly Detection, Root Cause Analysis, Cloud Networking Costs, Infrastructure Hygiene, Multi-cloud
Data Platforms and Tools
Apache Spark, PySpark, Apache Airflow, Apache Kafka, Kubeflow, ClickHouse, MergeTree Engines, PostgreSQL, MySQL, SQL Server, MongoDB, DuckDB, Redis, dbt, Delta Lake, Apache Arrow, Parquet, Ray, pandas, Looker
AWS
AWS, Amazon EMR, Amazon S3, Amazon S3 Tables, AWS Glue, Amazon Athena, Amazon Redshift, Amazon Kinesis, Amazon Data Firehose (formerly Amazon Kinesis Data Firehose), AWS Lambda, Amazon DynamoDB, Amazon RDS, Amazon Aurora, Amazon EC2, Amazon EKS, Amazon ECS, Amazon ECR, Amazon SQS, Amazon SNS, AWS IAM, Amazon CloudWatch, AWS CloudTrail, AWS Cost and Usage Report, AWS Billing
Azure
Azure, Azure Data Lake Storage (ADLS), Azure Blob Storage, Azure Kubernetes Service (AKS), Azure Databricks, Microsoft Entra ID, Azure Cost Data
Google Cloud
Google Cloud, Google BigQuery, Google Kubernetes Engine (GKE), Google Cloud Cost Data
Snowflake
Snowflake, Snowflake Architecture, Snowsight, SnowSQL, Virtual Warehouses, Multi-cluster Warehouses, Resource Monitors, Account Management, Serverless Compute, Snowpipe, Snowpipe Streaming, COPY INTO, Stages, Storage Integrations, Streams, Tasks, Dynamic Tables, Snowflake Tables, Materialized Views, Clustering Keys, Search Optimization Service, Query Acceleration Service, Query Profile, Time Travel, Fail-safe, Zero-copy Cloning, Replication and Failover, Secure Data Sharing, Snowflake Marketplace, Data Clean Rooms, Snowpark, Streamlit in Snowflake, Snowflake Cortex, Cortex Search, Snowflake Native Apps, Snowflake Scripting, UDFs, Stored Procedures, External Tables, Iceberg Tables, Hybrid Tables, Dynamic Data Masking, Row Access Policies, Object Tagging, Network Policies, Snowflake Kafka Connector, Snowflake Spark Connector, Connectivity
Databricks
Databricks, Azure Databricks, Databricks Unity Catalog, Databricks Lakeflow Pipelines (formerly Delta Live Tables), Databricks Lakeflow Connect, Databricks Jobs, Databricks Auto Loader, Databricks Declarative Automation Bundles (formerly Databricks Asset Bundles), Databricks Git Folders, Delta Lake, Lakehouse Architecture, Data Interoperability with Unity Catalog, ABAC Policies, Deploying Workloads, Production Deployment, Tuning Parameters, Implementing CI/CD, Configuring Lakeflow Connect
Cloud and Infrastructure
Kubernetes, Docker, Docker Compose, Helm, Istio, Terraform, Terragrunt, Prometheus, Grafana, Datadog, Microservices Architecture, Multi-cloud
Data Science, Machine Learning and Generative AI
Anomaly Detection, Statistical Analysis, Functional Approximation, Outlier Detection, LDA, Time Series Forecasting, Hierarchical Forecasting, Changepoint Detection, Seasonality, facebook-prophet, ARIMA, LSTM, forecast (Python library), Ensemble Models, ML Model Selection, Model Fitting, Model Fine-tuning, Model Evaluation, Model Testing and Validation, Cross-validation, Back-testing, Forecast Error Metrics (RMSE, MAPE), Feature Engineering, MLOps, Mathematical Optimization, Mixed-integer Linear Programming, PuLP, Algorithm Simulation, Generative AI, Generative AI Application Development, LLM Application Development, RAG, ReAct Agents, MCP, OpenAI, Data Retrieval, Summarization, Claude, Claude Code, Claude Skills, Claude Code Hooks, AI-Assisted Software Development, Agentic Coding, Agentic Software Development
Programming Languages
Python, Scala, Java, JVM, SQL, T-SQL, MDX, C#, TypeScript, Objective-C
Software Engineering and DevOps
Flask, FastAPI, REST APIs, Microservices, Backend Development, Web Services, Git, GitHub, Bitbucket, Bitbucket Pipelines, CI/CD, SonarQube, Code Quality, pytest, Automated Testing, Test Coverage, Data Pipeline Testing, Engineering Practices, Engineering Standards, DevOps, Technical Documentation, Maintainable Code, iOS Development, Unity 3D
Business Intelligence and Reporting
Business Intelligence, SSAS, SSIS, SSRS, OLAP Cubes, Analytics Dashboards, Reporting, Report Development, Report Generation Tooling, Self-service BI, Microsoft Master Data Services (MDS), ASP.NET, ASP.NET MVC 4
Leadership and Delivery
Data Engineering Leadership, Engineering Leadership, Technical Leadership, Team Leadership, Technical Mentorship, Onboarding, Remote Team Management, Distributed Teams, Near-shore Collaboration, Cross-functional Collaboration, Contractor Management, Product Design, Product Development, Delivery Management, Project Planning, Estimation, Technical Evaluation, Decision Reports, Customer Enablement, Customer Retention, Enterprise SaaS, Startup Experience, Founding Team Member, Startup Scaling

Education

National University of Computer and Emerging Sciences (FAST-NU) – BS in Computer Science

Lahore, Pakistan · –

  • GPA: 3.84 / 4.0
  • Magna Cum Laude
  • Silver Medal: second position in BS Computer Science, Batch 2008.
  • Rector's List (semester GPA 4.0) in Fall 2011, and Dean's List in the other seven semesters.
  • Teaching Assistant for Discrete Structures (MT217), Fall 2010.

Verify degree

Projects

Distributed ClickHouse Analytics Warehouse Modernization

Replacement of MongoDB with a sharded ClickHouse warehouse for multi-tenant cloud cost analytics.

  • Led architecture, benchmarking, and POC development to replace MongoDB with a sharded ClickHouse warehouse for multi-tenant cost analytics, scaling up processing capability by 50% and supporting more than 1 billion rows per run.
  • Designed table engines, sharding, sort and partition keys, dynamic-data storage, parallel ingestion, and high-throughput query patterns.
  • Utilized Claude Code with curated, context-aware Claude Skills and hooks to run reproducible POCs, preserve evidence-backed architecture decisions, generate design documentation, and oversee software development of data processing pipelines.
  • Reduced median query latency by 79%, and six-month data-load time from five days to one day (80% improvement) in POC benchmarks.

Legacy Platform Migration and Cross-Platform Validation

Migration of customers from the legacy FinOps platform to CoreStack, with automated validation of every output against cloud-vendor source data.

  • Directed customer migration from the legacy platform with 100% retention, mapping feature parity and building single-trigger validation pipelines that reconciled cloud costs, organizational hierarchies, Kubernetes workloads, anomalies, forecasts, and recommendations against cloud-vendor source data at scale.
  • Mapped feature parity between the legacy platform and CoreStack to drive the migration plan and validation scope.
  • Built single-trigger validation pipelines that reconciled application outputs against cloud-vendor source data across cloud costs, organizational hierarchies, Kubernetes workloads, anomalies, forecasts, and recommendations.

FinOps FOCUS 1.1 Ingestion and Unified Total Cost View

Ingestion of FinOps FOCUS 1.1 cost data sources at scale into a single Total Cost view across providers.

  • Engineered a data processing solution that ingests FinOps FOCUS 1.1 data sources at scale, up to 2.5 billion rows per run, into a unified Total Cost view.

Generative AI Assistant for Financial Data

LLM-based assistant that retrieves, summarizes, and compares financial and cloud-cost data on request.

  • Maintained a Generative AI assistant with ReAct-based data retrieval, data summarization, and comparison capabilities on financial data.

Migration of Core Processing to Databricks

Architectural evaluation and proof of concept for migrating cost processing from the existing stack to Databricks.

  • Led architectural evaluation, proof of concept development, evaluation of cost and engineering resources, and timelines, delivering decision-ready reports.
  • Implemented one customer's cost processing pipeline end to end on Databricks and served its output to the customer-facing application in a production environment.

Multi-cloud Cost Processing and Attribution

Cost processing that integrates cost data from AWS, Azure, and GCP, Kubernetes metrics, and rule-based business hierarchy to build a holistic view of infrastructure costs, at up to 2.5 billion rows per run.

  • Architected and developed the AWS cost data processing pipelines that became the core of the platform.
  • Extended cost processing pipelines to support Azure, GCP, container cost allocation (Amazon ECS, Amazon EKS, Google GKE, Azure AKS), and cost allocation to rule-based business hierarchy. Scaled to support up to 2.5 billion rows per run.
  • Decomposed processing into idempotent steps with short lineage, and tuned Spark worker size and memory to process over 2.5 billion rows per run.
  • Fixed data skew, applied data bucketing and partitioning, used spot instances in EMR, and other techniques to reduce processing time by 30% and costs by 15%.

Anomaly Detection

Detection of spikes in cloud infrastructure cost, along with infrastructure deltas to enable remediation, scanning over 8 billion data points.

  • Implemented and scaled anomaly detection using statistical analysis, functional approximation, standard deviation, outlier detection, and LDA across 6-8 billion rows of cost data.
  • Detected a cost spike caused by test infrastructure that was never spun down, helping Zoom save $600,000 in runaway costs.
  • Detected a $30,000 spike caused by misconfigured API calls routed over the public internet instead of AWS's internal network, enabling a quick fix.

Case study: how Zoom manages costs in the face of skyrocketing infrastructure spin-up

Budgeting and Forecasting

Up to 2-year cost forecasts for all cloud providers at all levels of business hierarchy.

  • Built the forecasting feature using facebook-prophet and Kubeflow, creating an ensemble of three models optimized using RMSE and MAPE error metrics.
  • Generated 24-month forecasts at every level of the business hierarchy, with cross-validation and periodic back-testing to reduce RMSE and MAPE error metrics.
  • Led a team of in-office and remote members, including contractors, to deliver a scalable, robust solution in less than 3 months.

Container Cost Attribution and Recommendations

Augments infrastructure cost with container workload cost breakdown, including allocation and wastage.

  • Developed a usage-based cost attribution system for containerized workloads, compatible with self-managed Kubernetes, Amazon ECS, Amazon EKS, Azure AKS, and Google GKE.
  • Architected and developed container rightsize recommendations, using cost and container metrics to analyze actual and allocated resources, and compute optimized allocation with estimated cost saving.

Reserved Instance Recommendations and Near-Real-Time Cost Estimation

Reserved Instance coverage and conversion recommendations, with cost estimates ahead of AWS billing data.

  • Simulated AWS's documented RI application algorithm on live EC2 inventory to show how reservations would apply and where coverage fell short.
  • Optimized convertible RI exchanges with a mixed-integer linear program to cover on-demand usage at minimum cost, using AWS conversion factors; 95% of the code was unchanged when the platform was retired in 2025.
  • Estimated cost for the 2–3 days before AWS Cost and Usage Report data arrives, using hourly inventory pulls (about 1 hour behind); evaluated CloudTrail as an alternative and chose inventory.
  • A key capability in closing Intuit as a customer.

CI/CD and Automated Testing for the Data Platform

Engineering practices introduced across the cost data processing codebase: pipelines for build and deploy, quality gates, and automated tests.

  • Introduced CI/CD with Bitbucket Pipelines for building and deploying Docker images, with SonarQube quality checks.
  • Led the effort to introduce automated testing in the data processing codebase, raising coverage to over 70% in one year.

Publications