Idle Resource Detection: Recommender API & Automation

Idle resources là compute tài nguyên chạy nhưng không dùng — VM không có traffic, disk không có I/O, database không có query. Chúng tốn tiền nhưng không đóng góp value.

GCP Recommender API automatically detects idle resources và suggest deletion/stoppage. Hiểu cơ chế detection là hiểu cách tự động cleanup waste.

Idle Resource Detection Mechanics

What is "Idle"?

Metric-based definition:

VM is idle if:
  - CPU utilization < 5% for 7+ days, AND
  - Network I/O < 1 Mbps for 7+ days

Persistent Disk is idle if:
  - Read/write operations < 10/day for 7+ days

Database is idle if:
  - Query rate < 1 query/day for 7+ days

Cloud Run service is idle if:
  - Request rate < 1 request/day for 7+ days

Detection Process

Metric collection (daily):
- Cloud Monitoring collects CPU, network, disk I/O metrics
- Data stored in system logs
Analysis (periodic, ~3 days):
- Recommender API analyzes 7–14 day trailing window
- Identifies resources < idle threshold
Recommendation generation:
- Recommender creates "Stop VM" or "Delete disk" recommendation
- Calculates potential monthly savings
Notification:
- Recommendation visible in Cloud Console
- Can integrate via API (Pub/Sub) for automation

Recommender API: Idle Resource Recommendations

API Usage

bash

# List all idle VM recommendations
gcloud recommender recommendations list \
  --recommender=compute.instanceIdlenessRecommender \
  --zone=us-central1-a

# Output:
# NAME: projects/.../recommendations/stop-instance-abc123
# RESOURCE: projects/.../zones/us-central1-a/instances/idle-vm-1
# DESCRIPTION: Stop unused VM instance
# ESTIMATED_COST_REDUCTION: $500/month
# PRIORITY: High

Recommendation States

ACTIVE: New recommendation (Recommender created it)
CLAIMED: Team acknowledged, working on it
SUCCEEDED: Action completed (VM stopped, disk deleted)
FAILED: Action attempted but failed
DISMISSED: Team decided not to act

False-Positive Handling

Common False Positives

False Positive 1: Batch Job Setup (Intermittent Usage)

Scenario:
  - VM booted daily at 6 AM, runs batch job, stops at 8 AM
  - Rest of day: 0% CPU, 0% network
  
Recommender sees:
  - 7-day average: ~10% utilization (6–8 AM usage)
  - Recommends: Stop VM (false positive!)

Fix:

VMs scheduled for batch should not be idle-detected
Add label: scheduled-job: true
Filter recommendations: resources NOT having label scheduled-job

False Positive 2: Standby Database Replica

Scenario:
  - Hot standby database (read-only, waits for failover)
  - No actual queries, minimal I/O
  
Recommender sees:
  - 0% query rate
  - Recommends: Delete database (disaster!)

Fix:

Add label: standby-replica: true
Exclude from idle recommendations via filter

Handling False Positives

Strategy 1: Whitelisting / Labeling

Resources with these labels excluded from idle detection:
  - production-critical: true
  - scheduled-job: true
  - standby-replica: true
  - managed-externally: true

Strategy 2: Time-Based Exemptions

New resources < 30 days old exempt from idle detection
(reason: warm-up period, may not have hit peak traffic yet)

Query:
  Recommender filters out resources with creation_time < 30 days ago

Strategy 3: Manual Review

For high-risk resources (databases, caches):
  - Recommender detects but marks CLAIMED
  - Manual human review (talk to team)
  - If confirmed idle, mark SUCCEEDED
  - If false-positive, mark DISMISSED

Automation via Pub/Sub

Architecture

[Recommender API] 
    ↓ (generates recommendations)
[Cloud Pub/Sub Topic]
    ↓ (publishes messages)
[Cloud Function]
    ↓ (processes recommendation)
    ├→ [Decision Logic] → Stop VM? Delete disk?
    └→ [Logging & Alerts] → Slack, email

Implementation

Step 1: Create Pub/Sub topic

bash

gcloud pubsub topics create recommender-notifications
gcloud pubsub subscriptions create recommender-sub \
  --topic recommender-notifications

Step 2: Configure Recommender notifications

bash

gcloud recommender recommendations list \
  --recommender=compute.instanceIdlenessRecommender \
  --filter="priority=HIGH AND estimatedMonthlyBenefit >= 100"

Then set up Cloud Function to process messages.

Step 3: Cloud Function to auto-stop/delete

python

import google.cloud.compute_v1 as compute

def process_idle_recommendation(message):
    recommendation = json.loads(message.data)
    
    resource_name = recommendation['targetResources'][0]
    instance_name = resource_name.split('/')[-1]
    zone = resource_name.split('/')[-3]
    
    # Security check: Confirm it's not tagged as critical
    instance = compute.InstancesClient().get(
        project=PROJECT_ID, zone=zone, resource=instance_name)
    
    if instance.labels.get('production-critical') == 'true':
        print(f"SKIPPED: {instance_name} is production-critical")
        return
    
    # Stop the VM
    operation = compute.InstancesClient().stop(
        project=PROJECT_ID, zone=zone, resource=instance_name)
    
    print(f"STOPPED: {instance_name}")
    
    # Alert (Slack, email)
    notify_team(f"Idle VM {instance_name} stopped, potential savings ${benefit}/month")

Step 4: Deploy Cloud Function

bash

gcloud functions deploy process-idle-recommendation \
  --runtime python39 \
  --trigger-topic recommender-notifications \
  --service-account CLOUD_FUNCTION_SA

Scale: Enterprise-Level Idle Detection

Multi-Project Scanning

bash

# Get list of all projects
projects=$(gcloud projects list --format="value(projectId)")

for project in $projects; do
  gcloud recommender recommendations list \
    --recommender=compute.instanceIdlenessRecommender \
    --project=$project \
    >> idle_recommendations.txt
done

# Aggregate and report
sort -u idle_recommendations.txt | wc -l
# Output: 45 idle VMs across all projects
# Potential savings: $50k/month

Dashboard & Metrics

Cloud Monitoring dashboard to track:

- Total idle resources detected (count)
- Potential monthly savings (sum)
- Acceptance rate (SUCCEEDED / ACTIVE)
- False-positive rate (DISMISSED / ACTIVE)

Example query:

sql

SELECT
  recommendation_type,
  COUNT(*) as count,
  SUM(estimated_monthly_benefit) as total_benefit,
  COUNTIF(status = 'SUCCEEDED') / COUNT(*) as success_rate
FROM `project.recommender_recommendations`
GROUP BY recommendation_type;

Anti-Patterns & Risks

Mistake: Auto-delete all recommendations without review.

Impact: Accidentally delete production resources (disaster).

Fix: Human gate-keeping for production resources. Auto-action only for dev/test.

Anti-Pattern 2: No Whitelist Maintenance

Mistake: Add labels to resources, then forget to update Recommender filter.

Impact: Old labels outdated, false positives persist.

Fix: Audit label consistency quarterly.

Anti-Pattern 3: Ignoring Cost of Restart

Scenario: Stop idle VM, later team restarts. 10 restart cycles/year.

Impact:

Restart cost: 5–10 minutes downtime per restart
Lost productivity: Hours of team time
vs Idle cost: $50/month

ROI: Only delete if annual cost > restart friction cost.

Practical Workflows

Workflow 1: Weekly Idle Review

Monday 9 AM:
  1. Query idle recommendations
  2. Categorize: prod-critical, dev-test, unknown
  3. Investigate "unknown" → talk to team
  4. Action: Stop dev/test, mark prod-critical as DISMISSED
  
Result: 10–15 min weekly maintenance, $5k+ monthly savings

Workflow 2: Automated Cleanup (Dev/Test Only)

Setup:
  - Auto-delete devices tagged with `environment: dev` + idle 7 days
  - Auto-delete devices tagged with `environment: test` + idle 14 days
  - Manual review required for `environment: prod`

Result: Minimal overhead, consistent test env cleanup

Workflow 3: Graduated Recommendation Actions

Severity = estimated_monthly_benefit / (project_monthly_spend)

If severity > 10% AND environment == prod:
  Action: Notify team, require approval, 48h grace period

If severity 5–10%:
  Action: Notify team, auto-action if no response in 1 week

If severity < 5%:
  Action: Auto-action immediately (small savings, low risk)

Idle Resource Detection: Recommender API & Automation

Idle Resource Detection Mechanics

What is "Idle"?

Detection Process

Recommender API: Idle Resource Recommendations

API Usage

Recommendation States

False-Positive Handling

Common False Positives

Handling False Positives

Automation via Pub/Sub

Architecture

Implementation

Scale: Enterprise-Level Idle Detection

Multi-Project Scanning

Dashboard & Metrics

Anti-Patterns & Risks

Anti-Pattern 1: Blind Automation

Anti-Pattern 2: No Whitelist Maintenance

Anti-Pattern 3: Ignoring Cost of Restart

Practical Workflows

Workflow 1: Weekly Idle Review

Workflow 2: Automated Cleanup (Dev/Test Only)

Workflow 3: Graduated Recommendation Actions

References

Idle Resource Detection: Recommender API & Automation ​

Idle Resource Detection Mechanics ​

What is "Idle"? ​

Detection Process ​

Recommender API: Idle Resource Recommendations ​

API Usage ​

Recommendation States ​

False-Positive Handling ​

Common False Positives ​

Handling False Positives ​

Automation via Pub/Sub ​

Architecture ​

Implementation ​

Scale: Enterprise-Level Idle Detection ​

Multi-Project Scanning ​

Dashboard & Metrics ​

Anti-Patterns & Risks ​

Anti-Pattern 1: Blind Automation ​

Anti-Pattern 2: No Whitelist Maintenance ​

Anti-Pattern 3: Ignoring Cost of Restart ​

Practical Workflows ​

Workflow 1: Weekly Idle Review ​

Workflow 2: Automated Cleanup (Dev/Test Only) ​

Workflow 3: Graduated Recommendation Actions ​

References ​

Idle Resource Detection: Recommender API & Automation

Idle Resource Detection Mechanics

What is "Idle"?

Detection Process

Recommender API: Idle Resource Recommendations

API Usage

Recommendation States

False-Positive Handling

Common False Positives

Handling False Positives

Automation via Pub/Sub

Architecture

Implementation

Scale: Enterprise-Level Idle Detection

Multi-Project Scanning

Dashboard & Metrics

Anti-Patterns & Risks

Anti-Pattern 1: Blind Automation

Anti-Pattern 2: No Whitelist Maintenance

Anti-Pattern 3: Ignoring Cost of Restart

Practical Workflows

Workflow 1: Weekly Idle Review

Workflow 2: Automated Cleanup (Dev/Test Only)

Workflow 3: Graduated Recommendation Actions

References