Build Performance Optimization
Overview
Build cache optimization can reduce Docker build times from 10+ minutes to under 3 minutes for code-only changes. This guide covers cache analytics, Dockerfile best practices, and troubleshooting slow builds.
Build Cache Optimization Flow
BUILD PERFORMANCE ISSUE
│
┌────────────┴────────────┐
│ │
Cache Hit Rate? Build Duration?
│ │
┌───────────┼───────────┐ ┌───┴───┐
│ │ │ │ │
<50% 50-70% >70% >10min 3-10min
(Poor) (Needs (Good) (Slow) (Normal)
Work) │ │ │
│ │ │ │ │
▼ ▼ │ ▼ ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ CACHE HIT RATE <50% (CRITICAL) │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. Run Detailed Analysis: │
│ ./scripts/analyze-build-cache.sh --detailed │
│ │
│ 2. Check Recent Changes: │
│ ┌────────────────────────────────────────────────────┐ │
│ │ File Changed │ Expected Impact │ │
│ ├─────────────────────┼──────────────────────────────┤ │
│ │ package.json │ Full rebuild (acceptable) │ │
│ │ Dockerfile │ Layer order broken? │ │
│ │ .dockerignore │ Context size changed? │ │
│ │ Source code only │ Should be >70% cached! │ │
│ └─────────────────────┴──────────────────────────────┘ │
│ │
│ 3. Common Fixes: │
│ A. Dockerfile Layer Order │
│ ✅ CORRECT: COPY package*.json → pnpm install → COPY src/ │
│ ❌ WRONG: COPY . → pnpm install │
│ │
│ B. .dockerignore Incomplete │
│ → Add: node_modules/, dist/, .git/, .env* │
│ → Verify: docker build context size should be <150MB │
│ │
│ C. BuildKit Not Enabled │
│ → Check: DOCKER_BUILDKIT=1 environment variable │
│ → Add to ~/.zshrc or ~/.bashrc │
│ │
│ D. Cache Cleared │
│ → Check: docker builder prune -a (did someone run this?) │
│ → Accept: First build after clear will be slow (expected) │
│ │
└──────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────┐
│ CACHE HIT RATE 50-70% (NEEDS IMPROVEMENT) │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Optimization Opportunities: │
│ │
│ 1. Review Layer Order (Dockerfile) │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Layer Frequency │ Position in Dockerfile │ │
│ ├────────────────────┼────────────────────────────┤ │
│ │ Never changes │ → Top (FROM, system deps) │ │
│ │ Rarely changes │ → Middle (package.json) │ │
│ │ Changes frequently │ → Bottom (COPY src/) │ │
│ └────────────────────┴────────────────────────────┘ │
│ │
│ 2. Use pnpm Store Cache Mount │
│ RUN --mount=type=cache,target=/root/.local/share/pnpm/store \ │
│ pnpm install --frozen-lockfile │
│ │
│ 3. Separate Build Dependencies │
│ # Install deps first (cached unless package.json changes) │
│ COPY package*.json pnpm-lock.yaml ./ │
│ RUN pnpm install │
│ # Copy source second (changes frequently) │
│ COPY src/ ./src/ │
│ │
│ 4. Multi-Stage Build Optimization │
│ # Builder stage (heavy) │
│ FROM node:22 AS builder │
│ RUN pnpm build │
│ # Runtime stage (lightweight) │
│ FROM node:22-alpine AS runtime │
│ COPY --from=builder /app/dist ./dist │
│ │
└──────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────┐
│ BUILD DURATION >10 MINUTES │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Slow Build Analysis: │
│ │
│ 1. Identify Slow Stage: │
│ docker build --progress=plain 2>&1 | grep 'DONE' │
│ → Look for stages taking >2 minutes │
│ │
│ 2. Common Slow Stages: │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Stage │ Typical Time │ If Slower │ │
│ ├─────────────────┼──────────────┼────────────────┤ │
│ │ pnpm install │ 1-2 min │ → Use cache │ │
│ │ pnpm build │ 2-3 min │ → Check CPU │ │
│ │ TypeScript emit │ 30-60s │ → tsconfig? │ │
│ │ COPY operations │ <30s │ → Reduce size │ │
│ └─────────────────┴──────────────┴────────────────┘ │
│ │
│ 3. Docker Resource Limits: │
│ Docker Desktop → Settings → Resources │
│ ├─ CPUs: Recommended 4+ cores │
│ ├─ Memory: Recommended 8GB+ │
│ └─ Disk: Recommended 60GB+ │
│ │
│ 4. Cross-Platform Build Overhead: │
│ Building AMD64 on M1 Mac adds ~40% time (emulation) │
│ ✅ Expected: 4min build → 6-7min AMD64 build │
│ ❌ Unexpected: 4min → 12min (investigate) │
│ │
└──────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────┐
│ MONITORING & ALERTS │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Set Up Monitoring: │
│ │
│ 1. Weekly Cache Health Check: │
│ ./scripts/analyze-build-cache.sh --trends │
│ → Look for degrading trends │
│ │
│ 2. Alert Thresholds: │
│ ⚠️ Cache hit rate drops below 60% for 3 consecutive builds │
│ ⚠️ Build time increases >50% from baseline │
│ ⚠️ Build time >15 minutes (extreme outlier) │
│ │
│ 3. Performance Dashboard (recommended): │
│ Create spreadsheet tracking: │
│ - Date, Git SHA, Build Duration, Cache Hit Rate │
│ - Chart trends monthly │
│ - Identify performance regressions early │
│ │
└──────────────────────────────────────────────────────────────────────────┘
QUICK WIN: If cache analytics shows <70% AND you haven't changed package.json,
check Dockerfile layer order first (most common issue).
Build Cache Analytics
Automatic Tracking
Every build automatically tracks cache performance and saves statistics to
build-cache-stats.json:
# Standard build (cache tracking automatic)
./scripts/docker-audit/build-cross-platform.sh --api --local
After each build, you'll see:
╔════════════════════════════════════════════════════════════╗
║ BUILD CACHE PERFORMANCE REPORT ║
╠════════════════════════════════════════════════════════════╣
║ Build Type: api (linux/arm64) ║
║ Total Duration: 145s ║
║ Cache Hit Rate: 72% (18/25 layers) ║
║ Time Saved: ~90s (estimated) ║
╠════════════════════════════════════════════════════════════╣
║ RECOMMENDATIONS: ║
║ • ✅ Excellent cache performance (72%) - build optimized ║
╚════════════════════════════════════════════════════════════╝
View Cache Analytics
Summary Mode (quick health check):
./scripts/analyze-build-cache.sh
Shows:
- Overall statistics (average cache hit rate, build time)
- Recent builds (last 5)
- Optimization suggestions
Detailed Mode (per-build analysis):
./scripts/analyze-build-cache.sh --detailed
Shows:
- Platform, Git SHA, duration for each build
- Cache hit rate, layers cached/rebuilt
- Time saved estimate
- Build-specific recommendations
Trends Mode (performance over time):
./scripts/analyze-build-cache.sh --trends
Shows:
- API build performance trend
- Web build performance trend
- Trend direction (improving/degrading/stable)
Performance Targets
| Metric | Target | Current Baseline | Status | | ----------------------------- | ------- | --------------------------- | ------------- | | Cache hit rate (code changes) | ≥70% | ~73% average | ✅ Good | | API build time (cached) | <3 min | ~2.5 min | ✅ Good | | Web build time (cached) | <5 min | ~4 min | ✅ Good | | Cold build (no cache) | <10 min | ~6 min (API), ~10 min (Web) | ✅ Acceptable |
When to Investigate
Investigate cache performance if:
-
Cache hit rate <50% for consecutive builds
- Indicates layer invalidation issues
- Check recent Dockerfile changes
-
Build time suddenly increases >50%
- May indicate cache corruption
- Check for context size issues
-
"CACHE" messages missing in build logs
- BuildKit may not be enabled
- Cache may have been cleared
Dockerfile Best Practices
Layer Ordering (Critical)
Correct order (from least to most frequently changing):
# 1. Base image (rarely changes)
FROM node:22.21.1-alpine AS base
# 2. System dependencies (rarely changes)
RUN apk add --no-cache curl postgresql-client
# 3. Package files (changes when dependencies update)
COPY package.json pnpm-lock.yaml ./
# 4. Install dependencies (cached unless package files change)
RUN pnpm install --frozen-lockfile
# 5. Application code (changes frequently)
COPY . .
# 6. Build step (runs when code changes)
RUN pnpm build
Why this order matters:
- Docker caches each layer separately
- Changing a layer invalidates ALL subsequent layers
- Putting frequently-changing code early breaks cache for everything after
- Putting rarely-changing dependencies first maximizes cache reuse
Package-First Pattern
✅ CORRECT: Copy package files before installing
# Copy only package files
COPY package.json pnpm-lock.yaml ./
# Install dependencies (cached unless package files change)
RUN pnpm install --frozen-lockfile
# Copy rest of code (doesn't invalidate install cache)
COPY . .
❌ WRONG: Copy all code before installing
# Copies EVERYTHING (invalidates on ANY code change)
COPY . .
# Re-installs EVERY time code changes
RUN pnpm install --frozen-lockfile
Impact: Wrong pattern causes 5-10 minute dependency reinstalls on every build!
Cache Mounts (BuildKit)
Use BuildKit cache mounts for package managers:
# Enable BuildKit syntax
# syntax=docker/dockerfile:1
# Use cache mount for pnpm
RUN --mount=type=cache,target=/root/.local/share/pnpm/store \
pnpm install --frozen-lockfile
Benefits:
- Persistent cache across builds
- Faster installs even when package.json changes
- Shared cache between multiple builds
.dockerignore Optimization
Always exclude:
# Development files
node_modules/
.next/
dist/
build/
# Git files
.git/
.gitignore
# IDE files
.vscode/
.idea/
# Documentation
*.md
docs/
# Logs
*.log
logs/
# Test files
coverage/
*.test.ts
*.spec.ts
__tests__/
# Environment files
.env*
!.env.example
Impact: Reduces context from 3.3GB → 112MB (97% reduction!)
Check context size:
./scripts/docker-audit/analyze-context-size.sh --verbose
Common Cache Issues
Issue: Cache Miss on Every Build
Symptoms:
- Cache hit rate 0% or very low (<20%)
- All layers rebuild every time
- Build logs missing "CACHE" messages
Diagnostic:
# Check cache performance
./scripts/analyze-build-cache.sh --detailed | tail -20
# Check if BuildKit enabled
docker buildx ls
Causes:
A. BuildKit Not Enabled
# Solution: BuildKit is enabled by default in modern Docker
# Verify it's working:
docker buildx inspect default
B. Dockerfile Layer Order Changed
# Check recent changes
git diff HEAD~1 HEAD -- apps/api/Dockerfile apps/web/Dockerfile
# Solution: Revert layer order changes
# Keep package.json copying before pnpm install
C. .dockerignore Missing Files
# Check context size
./scripts/docker-audit/analyze-context-size.sh --verbose | grep -i "large\|should exclude"
# Solution: Add patterns to .dockerignore
echo "*.log" >> .dockerignore
echo "coverage/" >> .dockerignore
Issue: Slow Builds Despite Good Cache
Symptoms:
- Cache hit rate >70%
- Build still takes 8+ minutes
- Specific layers take long time
Diagnostic:
# Look at build logs for slow stages
./scripts/docker-audit/build-cross-platform.sh --api --local 2>&1 | grep "DONE"
# Check layer sizes
docker history sampo-blueline-alpha-api:latest
Causes:
A. Large Dependencies
Some npm packages are large (e.g., Prisma, TypeScript):
# Solution: Split dependencies (production vs dev)
RUN pnpm install --frozen-lockfile --prod
COPY . .
RUN pnpm install --frozen-lockfile # Adds dev deps
B. Expensive Build Step
TypeScript compilation can be slow:
# Solution: Use incremental compilation
# In tsconfig.json:
{
"compilerOptions": {
"incremental": true
}
}
Issue: Cache Degradation Over Time
Symptoms:
- Cache hit rate was 70%, now 40%
- Build time increased gradually
- No obvious Dockerfile changes
Diagnostic:
# Check trends
./scripts/analyze-build-cache.sh --trends
# Compare recent builds
./scripts/analyze-build-cache.sh --detailed | head -50
Causes:
A. Package.json Changed Frequently
# Check package.json change frequency
git log --oneline -- package.json pnpm-lock.yaml | head -10
# Solution: Expected if adding dependencies
# If unintentional, review why dependencies changing
B. Context Size Growing
# Check context size trend
./scripts/docker-audit/analyze-context-size.sh
# Solution: Update .dockerignore to exclude new patterns
Optimization Strategies
Strategy 1: Multi-Stage Builds
Use separate stages for dependencies and build:
# Stage 1: Dependencies
FROM node:22.21.1-alpine AS deps
COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
# Stage 2: Build
FROM node:22.21.1-alpine AS builder
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN pnpm build
# Stage 3: Production
FROM node:22.21.1-alpine AS production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY package.json ./
CMD ["node", "dist/main.js"]
Benefits:
- Dependencies layer cached separately
- Build layer cached separately
- Production image is smaller (no build tools)
Strategy 2: Parallel Builds
Build API and Web in parallel:
# Build both simultaneously (uses all CPU cores)
./scripts/docker-audit/build-cross-platform.sh --api --local &
./scripts/docker-audit/build-cross-platform.sh --web --local &
wait
# Or use --all flag (handles parallelization automatically)
./scripts/docker-audit/build-cross-platform.sh --all --local
Time savings: 40% faster than sequential (8min → 5min)
Strategy 3: Warm Cache Maintenance
Periodically rebuild to keep cache warm:
# Weekly: Full rebuild on CI/CD
# Keeps cache layers fresh
# Prevents cache expiration
# On developer machines: Rebuild after major updates
git pull origin main
./scripts/docker-audit/build-cross-platform.sh --all --local
Monitoring & Alerts
Weekly Cache Health Check
Run every Monday:
./scripts/analyze-build-cache.sh
# Expected output:
# Average cache hit rate: ≥70%
# Recent builds: No sudden drops
If cache hit rate <50%:
- Run detailed analysis
- Check for Dockerfile changes
- Review .dockerignore coverage
- Check context size
Set Up Alerts
Monitor these metrics:
- Cache hit rate trend (should be stable 70-80%)
- Build time trend (should be stable <5min for cached builds)
- Context size (should be <150MB)
Alert if:
- Cache hit rate drops below 50% for 3 consecutive builds
- Build time increases >2x baseline
- Context size grows >50%
Performance Benchmarks
Expected Build Times
API (with good cache):
- Cold build (no cache): 6-8 minutes
- Warm build (70%+ cache): 2-3 minutes
- Code-only change: 1-2 minutes
Web (with good cache):
- Cold build (no cache): 10-12 minutes
- Warm build (70%+ cache): 4-5 minutes
- Code-only change: 2-3 minutes
Both (parallel):
- Cold build: ~10 minutes
- Warm build: ~5 minutes
Real-World Examples
Example 1: Code Change Only
Files changed: apps/api/src/modules/admin/users.service.ts
Expected cache hit rate: 80-90%
Expected build time: 2-3 minutes
Layers rebuilt: 5-7 layers (code + build stages)
Example 2: Dependency Added
Files changed: package.json, pnpm-lock.yaml
Expected cache hit rate: 40-50%
Expected build time: 5-7 minutes
Layers rebuilt: 12-15 layers (install + code + build stages)
Example 3: Dockerfile Modified
Files changed: apps/api/Dockerfile
Expected cache hit rate: 0-20%
Expected build time: 8-10 minutes
Layers rebuilt: All layers
Advanced Optimization
Cross-Compilation Optimization
Our Dockerfiles use cross-compilation to speed up builds:
# Build stage runs NATIVELY on M1 Mac (ARM64)
FROM --platform=$BUILDPLATFORM node:22.21.1-alpine AS builder
ARG TARGETPLATFORM
ARG BUILDPLATFORM
# Dependencies and build run at native speed (5-10x faster)
RUN pnpm install
RUN pnpm build
# Only runtime stage uses target platform (AMD64)
FROM --platform=$TARGETPLATFORM node:22.21.1-alpine AS production
COPY --from=builder /app/dist ./dist
Impact: 40-60% faster builds (API: 4min → 2.5min, Web: 7min → 4min)
See docs/operations/docker-cross-compilation-improvements.md for details.
BuildKit Features
Enable advanced BuildKit features:
# syntax=docker/dockerfile:1
# Cache mounts
RUN --mount=type=cache,target=/root/.local/share/pnpm/store \
pnpm install
# Secret mounts (for private registries)
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
pnpm install
# SSH mounts (for private git repos)
RUN --mount=type=ssh \
git clone git@github.com:private/repo.git
Troubleshooting Checklist
When builds are slow:
-
Check cache analytics
./scripts/analyze-build-cache.sh -
Review recent Dockerfile changes
git log --oneline -- apps/*/Dockerfile -
Check context size
./scripts/docker-audit/analyze-context-size.sh -
Verify .dockerignore coverage
cat .dockerignore -
Check for large dependencies
docker history sampo-blueline-alpha-api:latest -
Review package.json changes
git diff HEAD~5 HEAD -- package.json
Related Articles
Core Documentation
- 📘 Deployment Overview - System features and workflow
- 🎓 Deployment Quick Start - Getting started tutorial
- 🔧 Troubleshooting Guide - Issue diagnosis and resolution
Advanced Resources
- 📄 Full optimization guide:
docs/operations/build-cache-optimization.md(500+ lines) - ⚙️ Cross-compilation guide:
docs/operations/docker-cross-compilation-improvements.md - 🔧 Deployment runbook:
docs/operations/deployment-runbook.md
Key Takeaways
- ✅ Layer ordering matters: Put stable dependencies first
- ✅ Package-first pattern: Copy package.json before installing
- ✅ Use .dockerignore: Exclude unnecessary files
- ✅ Monitor cache performance: Weekly health checks
- ✅ Target 70%+ cache hit rate for code-only changes
- ✅ Investigate drops below 50% immediately
Good cache performance means faster deployments, happier developers, and more productive teams!