Full engagement
Full consultation
Discuss your complete cloud and security strategy with the principal consultant. For comprehensive transformations and multi-quarter engagements.
Case study: Cold Bore Capital
Cold Bore Capital Modernised ML Infrastructure on Cloud-Native AWS
A manual VM-based ML stack was slowing the team. Pilotcore moved the work to AWS EKS, Fargate, and Terraform, with a data pipeline the team owns end to end.
-
Sector
Private equity, ML
-
Engagement
Cloud-native ML migration
-
Stack
AWS EKS, Fargate, Airflow, MLflow
At a glance
Engagement snapshot.
Numbers observed during the engagement window. Outcomes depend on baseline architecture, team process, and workload profile.
-
6 months
Cloud-native migration
-
40%
Faster ML workflows
-
0
Unplanned outages
Challenge
Manual ML infrastructure was slowing the data team.
Cold Bore Capital ran machine learning and data engineering workloads on hand-maintained virtual machines. The manual operating model raised risk and slowed recovery from incidents.
-
Manual VM management.
Hand-maintained virtual machines introduced inefficiencies and human error in day-to-day operations.
-
Slow recovery from failures.
The manual stack left the firm exposed to extended recovery windows when components failed.
-
Time-consuming maintenance.
Routine updates and issue resolution took disproportionate engineering time.
-
Scalability limits.
The setup struggled to keep up with growing data volumes and model lifecycle needs.
-
Distraction from data work.
The team spent more time tending infrastructure than running data-driven activities.
Approach
A cloud-native strategy built around automation.
We assessed the existing stack and designed a Kubernetes and Terraform-led approach to automate provisioning, scale workloads, and stabilise the operating model.
-
In-depth analysis.
Identified the key pain points and infrastructure bottlenecks driving operational risk.
-
Strategic planning.
Mapped a transition to cloud-native infrastructure with automation as the operating model.
-
Solution design.
Built around AWS EKS, Fargate, and custom Terraform modules for reproducible environments.
-
Data and ML pipeline.
Brought Apache Airflow and MLflow in to standardise data workflows and the model lifecycle.
-
Knowledge transfer.
Trained Cold Bore Capital's team on the new tooling so they could operate it independently.
Solution
Cloud-native foundations the team can operate independently.
Kubernetes on EKS, Terraform modules, and a data pipeline tuned for the team's actual workflow.
-
Kubernetes on AWS EKS.
Containerised workloads on EKS for scalable, repeatable environments across stages.
-
Infrastructure as code.
Standardised Terraform modules so environments could be reproduced and reviewed.
-
Serverless containers.
Used AWS Fargate to right-size resources and remove EC2 patching from the operations load.
-
Enhanced data pipeline.
Apache Airflow for workflow orchestration and MLflow for model lifecycle management.
-
Automated deployment.
CI/CD pipelines deliver infrastructure and applications with consistent guardrails.
-
Security baseline.
Tightened access controls and encryption practices across the data and ML stack.
Outcomes
Observed improvements during the engagement.
Cold Bore Capital reported the following improvements during the engagement period. Results vary based on baseline maturity, team process, and workload profile.
i. Automation
Outcome 01
Enhanced automation across workflows.
Less manual intervention, more consistency between runs of the same pipeline.
ii. Scalability
Outcome 02
Dynamic scaling for ML workloads.
The Kubernetes solution scaled workloads up and down with demand, optimising performance.
iii. Speed
Outcome 03
Faster data processing.
The new pipeline shortened end-to-end data work and accelerated model deployment.
iv. Reliability
Outcome 04
Higher system reliability.
Reduced downtime and a clearer operating model when issues did appear.
v. Consistency
Outcome 05
Reproducible environments.
Infrastructure as code locked in consistency and reduced configuration drift.
vi. Cost
Outcome 06
Better cost profile.
Serverless and right-sized resources lowered IT cost pressure on the platform.
Next step
Turn Complexity into Opportunity.
Cold Bore Capital changed how its data team operates. If your firm is on a similar path, we can talk through what worked and what we would do differently.