Storage Tier Automation: Lifecycle Policies & Archive Storage

Cloud Storage cost lớn nếu dữ liệu lạnh (cold) không được downgrade tới cheaper tiers. Một terabyte dữ liệu lạnh 5 năm tuổi ở Standard storage tốn $20/tháng. Archive storage: $0.01/tháng (tương tự dữ liệu).

Lifecycle policies tự động downgrade storage class theo age hoặc access pattern, cắt giảm 80–95% storage cost cho dữ liệu lạnh.

---

Storage Classes: Cost vs Access Trade-offs

Class       | Availability | Min Duration | Cost/GB/Month | Min Retrieval | Retrieval Cost
Standard    | 99.95%       | None         | $0.020        | Immediate     | Free
Nearline    | 99.9%        | 30 days      | $0.010        | Immediate     | $0.001/GB
Coldline    | 99.9%        | 90 days      | $0.004        | Immediate     | $0.002/GB
Archive     | 99.95%       | 365 days     | $0.0012       | 1–5 hours     | $0.005/GB

Scenario: 10 TB data not accessed in 6 months

Standard:  $0.020 × 10,000 × 6 = $1,200
Archive:   $0.0012 × 10,000 × 6 = $72
Savings:   $1,128 (94%)

When Each Class Makes Sense

Standard:  Hot data (accessed daily/weekly) or first 30 days
Nearline:  Warm data (accessed monthly, infrequent)
Coldline:  Cold data (accessed quarterly or less)
Archive:   Frozen data (accessed yearly or compliance-only)

---

Lifecycle Policies: Automatic Downgrade

Basic Lifecycle Policy

{
  "lifecycle": {
    "rule": [
      {
        "action": {
          "storageClass": "NEARLINE",
          "type": "SetStorageClass"
        },
        "condition": {
          "age": 30
        }
      },
      {
        "action": {
          "storageClass": "COLDLINE",
          "type": "SetStorageClass"
        },
        "condition": {
          "age": 90
        }
      },
      {
        "action": {
          "storageClass": "ARCHIVE",
          "type": "SetStorageClass"
        },
        "condition": {
          "age": 365
        }
      }
    ]
  }
}

Applying Lifecycle Policy

gsutil lifecycle set lifecycle.json gs://my-bucket

# Verify
gsutil lifecycle get gs://my-bucket

Advanced Rules: Delete Old Data

{
  "lifecycle": {
    "rule": [
      {
        "action": {"type": "Delete"},
        "condition": {"age": 2555}  // 7 years, then delete
      },
      {
        "action": {"storageClass": "ARCHIVE", "type": "SetStorageClass"},
        "condition": {"age": 365}
      }
    ]
  }
}

---

Early Deletion Charges: Hidden Cost

What is Early Deletion?

If object stored in Coldline/Archive, then deleted before min duration, GCP charges early deletion fee.

Coldline min duration: 90 days
Archive min duration: 365 days

Scenario 1: Upload to Coldline, delete after 45 days
  → Early deletion charge = $0.002/GB (same as retrieval cost)
  → Plus: storage cost for 45 days (prorated)
  
Scenario 2: Upload to Archive, delete after 180 days
  → Early deletion charge = $0.005/GB (retrieval cost)
  → Plus: storage cost for 180 days

Workaround: Lifecycle Policy + Autoclass

Autoclass automatically transitions objects across storage classes without triggering early deletion:

{
  "autoclass": {
    "enabled": true,
    "terminalStorageClass": "ARCHIVE"
  }
}

Behavior:

• Objects move: Standard → Nearline (30 days) → Coldline (90 days) → Archive (365 days)
• No early deletion charges (managed by Google)
• Cost: Same as manual lifecycle, but simpler

---

Autoclass: Automatic Tier Management

How Autoclass Works

1. Enable Autoclass on bucket
2. Objects automatically transition:
   - Day 0: Created (Standard)
   - Day 30: Nearline (if not accessed)
   - Day 90: Coldline (if not accessed)
   - Day 365: Archive (if not accessed)
3. Accessed object reverts to Standard
   (effectively resets timer)

Retrieval Logic

gcloud storage cp gs://my-bucket/archive-file.tar.gz .

Behind the scenes:
  1. Object in Archive
  2. User reads → automatic restore to Standard
  3. Download completes
  4. Cost: Archive retrieval fee ($0.005/GB) + Standard rate

Autoclass Configuration

gsutil buckets update --autoclass-enabled gs://my-bucket

Trade-offs

Pros:
  - Automatic cost optimization
  - No early deletion charges
  - Simpler than manual policies
  
Cons:
  - Slight overhead (object transitions)
  - May not suit specific access patterns
  - Retrieval cost if suddenly accessed

---

Cost Analysis: Lifecycle Impact

Example: Backup Bucket (100 TB, 5-year retention)

Scenario: No lifecycle (all Standard)

Cost: $0.020/GB × 100,000 GB × 12 months × 5 years = $120,000

Scenario: Lifecycle (Standard → Nearline → Coldline → Archive)

Year 1: Standard only (hot backups)      = $0.020 × 100TB × 12 = $24,000
Year 2: Nearline (warm backups)          = $0.010 × 100TB × 12 = $12,000
Year 3: Coldline (cold backups)          = $0.004 × 100TB × 12 = $4,800
Year 4–5: Archive (frozen backups)       = $0.0012 × 100TB × 12 × 2 = $2,880

Total: $43,680
Savings: $120,000 – $43,680 = $76,320 (64% reduction)

---

Practical Workflows

Workflow 1: Time-Based Transition (Common)

Logs bucket:
  - Standard (30 days): hot analysis, real-time logs
  - Nearline (30–90 days): recent analysis, compliance review
  - Archive (90+ days): compliance retention

JSON config:
{
  "rule": [
    {"action": {"storageClass": "NEARLINE", "type": "SetStorageClass"},
     "condition": {"age": 30}},
    {"action": {"storageClass": "ARCHIVE", "type": "SetStorageClass"},
     "condition": {"age": 90}}
  ]
}

Workflow 2: Access-Based Transition (Autoclass)

ML training data bucket:
  - Enable Autoclass
  - Objects auto-archive if unused 1 year
  - If accessed, auto-restore (slight latency)
  - Good for: Experimental, archived datasets

Workflow 3: Storage Class Transition + Deletion

Temp data bucket:
  - Delete after 7 days (test/scratch data)
  - No need for lifecycle (TTL simple)

{
  "rule": [
    {"action": {"type": "Delete"},
     "condition": {"age": 7}}
  ]
}

---

Monitoring & Cost Alerts

Query Storage Cost by Class

SELECT
  DATE_TRUNC(usage_start_time, MONTH) as month,
  sku.description,  -- Storage class info
  SUM(cost) as monthly_cost
FROM `project.billing_dataset.gcp_billing_export_v1`
WHERE service.description = 'Cloud Storage'
GROUP BY month, sku.description
ORDER BY month DESC;

Alert on Unused Storage

Create BigQuery scheduled query:
  - Monthly: Check storage distribution by class
  - Alert if Standard storage > 50% (may need lifecycle)
  - Alert if Coldline/Archive retrieval cost spike

---

Anti-Patterns

Anti-Pattern 1: Overly Aggressive Lifecycle

Scenario: Transition to Archive after 30 days.

Problem:

• Frequent retrieval (monthly analysis) = high retrieval cost
• Access resets timer (object stays Standard 30 more days)
• Net cost higher than Standard

Fix: Match lifecycle timing to access pattern. Use Autoclass for unknowns.

Anti-Pattern 2: Ignoring Early Deletion

Scenario: Lifecycle to Coldline at day 60, but delete at day 75.

Cost impact:

Delete charge: $0.002/GB
For 100 TB: $0.002 × 100,000 = $200 (one-time penalty)

Fix: Use Autoclass (no early deletion) or verify deletion timing before setup.

Anti-Pattern 3: No Lifecycle for Long-Lived Data

Scenario: Backup bucket, data stays 7 years, but no lifecycle.

Impact:

Cost unnecessarily high for years 2–7
Potential savings: 70–80% if downgraded

Fix: Review all long-term storage, apply lifecycle.

---

Scale: Organization-Level Storage Optimization

Scan All Buckets for Lifecycle Compliance

for bucket in $(gsutil ls | cut -d ':' -f 3-); do
  echo "Checking: $bucket"
  gsutil lifecycle get $bucket 2>/dev/null || echo "No lifecycle"
done

# Results:
# 45 buckets found
# 15 have lifecycle (33%)
# 30 missing lifecycle (67%) → potential savings

Policy Recommendation

Org Policy: Enforce lifecycle on all buckets

{
  "constraints": [
    {
      "constraint": "storage.bucketVersioningRequired",
      "enforcement": "enforce"
    },
    {
      "constraint": "storage.objectRetentionRequired",
      "enforcement": "enforce"
    }
  ]
}

---

References

• Cloud Storage Classes
• Object Lifecycle Management
• Autoclass
• Cloud Storage Pricing