Magento Auto Scaling With Varnish: 11 Unique Functions

• What is Auto Scaling?

• Why You Need Auto Scaling: Reasons, Problems, and Consequences of Unexpected Traffic Peaks

• Advantages of AWS Auto Scaling Features

• How Magento Auto Scaling works (with AWS Auto Scaling Example)

• 3 MGT Solutions Essential for a Professional Auto Scaling

• AWS Auto Scaling Architecture: How Does it Work?

• AWS Auto Scaling Architecture Requirements

• Troubleshooting Magento Auto Scaling Common Conflicts

• Lessons Learned for Making Auto Scaling Most Efficient

• FAQs

• Summary

What is Auto Scaling?

“Auto scaling monitors cloud resources and adjusts capacity.”

AWS Autoscaling is an advanced feature that helps manage AWS resources. It optimizes application performance and reduces costs.

Key Auto Scaling characteristics:

• It monitors & adjusts resources based on the server load

• The service launches or terminates EC2 instances

• Auto Scaling Groups contain several EC2 instances together

• The groups scale upon predefined CloudWatch metrics

• Instance counts adjust between the lowest and highest thresholds

• Scaling policies define when resources increase or decrease

• Target tracking maintains specific metric values like CPU

Why You Need Auto Scaling: Reasons, Problems, and Consequences of Unexpected Traffic Peaks

Marketing campaigns create massive traffic spikes for stores. Successful marketing campaigns increase traffic peaks.

Common reasons for traffic peaks include:

• Black Friday sales overwhelm single-server architectures

• Newsletter campaigns bring thousands of simultaneous visitors online

• Television advertisements drive instant traffic to Magento stores

• Groupon deals generate immediate purchase intent and load

At worst, this marketing campaign will only burn your money. In the short and long term, direct and indirect financial losses may arise.

Consequences of inadequate infrastructure:

• Server crashes lose customer trust and revenue

• Slow page loads increase cart abandonment rates

• Competitors capture frustrated customers during downtime periods

• Search rankings drop after poor site availability metrics

• Customer lifetime value decreases after negative shopping experiences

Advantages of AWS Auto Scaling Features

AWS Auto Scaling provides three key scaling methods:

1. Dynamic Scaling: Resources adjust based on real-time demand

2. Scheduled Scaling: Capacity changes follow predictable traffic patterns

3. Predictive Scaling: Machine learning anticipates future resource needs

You can scale the capacity of the scaling group. It responds to changes in traffic. AWS Auto Scaling helps optimize resource usage.

Key benefits include:

• Cost optimization occurs through automatic resource reduction

• Performance remains consistent during unexpected traffic surges

• Health checks replace failed instances without manual intervention

• Multi-zone deployment assures high availability across regions

How Magento Auto Scaling works (with AWS Auto Scaling Example)

Suppose you have a Magento store that runs on a single server. The single server performs well when you have regular traffic. Sometimes the traffic to the Magento store increases up to five times the average load.

Auto Scaling process flow:

1. Auto Scaling monitors CPU usage through CloudWatch alarms

2. High CPU triggers the launch of added EC2 instances

3. New instances receive traffic through Elastic Load Balancer

4. Equal traffic distributes across all healthy running instances

5. Low CPU periods trigger automatic instance termination processes

6. Dynamic resources scale between one and twenty instances

The AWS autoscaling CLI creates scaling policies. CloudWatch alarms trigger at a 60% CPU usage threshold. Scale-down occurs at 25% usage after the cooldown period.

3 MGT Solutions Essential for a Professional Auto Scaling

Solution 1: MGT Code Deploy–Zero-Downtime Deployment

Imagine that you have 5 web servers located in 3 different states. You have detected that something is not working and want to fix it as soon as possible.

MGT Code Deploy makes Auto Scaling usable. Code Deploy synchronizes the source across all instances.

Key deployment features:

• The deployment process executes without a service interruption occurring

• Rolling deployments update instances in configured batch sizes

• Deployment groups target specific Auto Scaling Group instances

• Shell scripts sync code before the nginx service starts

• The appspec.yml file defines deployment lifecycle hooks

• Post-deployment validation confirms successful code synchronization completion

Solution 2: MGT Cloud Log for Easy Log Management

MGT Cloud Log makes log management easy. Centralized logging aggregates data from several instances. CloudWatch Logs stores application and system logs.

Log management capabilities:

• Separate log stream by instance ID and date

• Metric filters create alarms from specific log patterns

• Real-time monitoring detects errors across the entire fleet

• The awslogs agent pushes logs to CloudWatch

• Configuration files specify log groups and retention periods

• Developers query logs without accessing individual server instances

Solution 3: MGT WAF–Web Application Firewall for Magento

MGT WAF protects Magento shops against common web exploits. Web exploits affect availability, compromise security, or consume excessive resources. WAF rules block SQL injection and XSS attacks.

Security features include:

• Rate limiting prevents DDoS attacks from overwhelming servers

• Geographic restrictions block traffic from suspicious countries

• Custom rules protect Magento-specific endpoints and admin paths

• CloudFront integration provides edge-level security filtering capabilities

• Managed rule sets update against emerging threats

• Request logs enable forensic analysis of attack patterns

AWS Auto Scaling Architecture: How Does it Work?

The architecture combines eleven key AWS services. Each component serves specific functions within the infrastructure. Services work together to create scalable Magento hosting environments.

1. Amazon Route 53 (DNS)

Amazon Route 53 is a dependable cloud DNS web service (Domain Name System). Route 53 provides geographic routing for global audiences. Health checks scan endpoint availability across several regions.

Advanced routing capabilities:

• Weighted routing policies distribute traffic between environment versions

• Latency-based routing directs users to the nearest endpoints

• Instant DNS failover switches traffic during regional outages

• The service integrates with CloudFront for CDN routing

• ALIAS records point straight to ELB endpoints

• TTL values control DNS caching behavior for updates

2. Varnish Cache

Varnish Cache is a web accelerator, also known as an HTTP accelerator or a reverse proxy. Varnish stores the full page cache in memory.

Varnish configuration elements:

• The VCL configuration defines caching rules and backends

• ESI tags enable partial page caching for Magento

• Cache invalidation occurs through PURGE requests from Magento

• Quick backend health checks detect failed application servers

• Grace mode serves stale content during backend failures

• The varnishlog tool provides real-time request debugging capabilities

3. Elastic Load Balancing (ELB)

Elastic Load Balancing distributes incoming traffic across several Web Server instances. These instances exist in different data centers, also known as availability zones. Application Load Balancers route based on request content.

ELB features for Magento:

• Target groups contain healthy Auto Scaling Group instances

• Health checks verify application responsiveness every thirty seconds

• Sticky sessions maintain user state across several requests

• Even cross-zone load balancing distributes requests everywhere

• Connection draining helps with a graceful instance shutdown during scaling

• Access logs record all requests for security analysis

4. Auto Scaling–Web Server (NGINX)

NGINX serves PHP-FPM processes for Magento application requests. Auto Scaling Groups maintain desired instance counts. Launch configurations define instance types and startup scripts.

Auto Scaling configuration components:

• The user_data script installs dependencies and a_nd configures services

• CloudWatch monitors CPU, memory, and disk usage metrics

• Automated scaling policies trigger at predetermined threshold values

• Instance warmup periods prevent premature traffic routing decisions

• Lifecycle hooks enable custom actions during scaling events

• The AWS EC2 commands manage instances through the CLI tools

5. Database Server (MySQL)

Amazon Relational Database Service (RDS) is a reliable and scalable cloud database service. It provides the highest performance and security, and is inexpensive. RDS manages backups, patches, and failover.

RDS features for Magento:

• Multi-AZ deployment provides synchronous database replication for availability

• Read replicas offload query traffic from primary instances

• Parameter groups optimize MySQL settings for Magento workloads

• Aurora MySQL provides better performance than standard MySQL

• Automated backups enable point-in-time recovery within retention periods

• CloudWatch monitors database performance metrics and slow queries

6. Database Server (Slave)

Read replicas handle catalog browsing and search queries. Replication lag monitoring guarantees data consistency across instances. Magento configuration routes read queries to replica endpoints.

Read replica configuration:

• The split_db configuration separates read and write connections

• Connection pooling optimizes database resource usage

• Replica promotion enables quick disaster recovery procedures

• Binary log retention supports delayed replica configurations when needed

• Global cross-region replicas provide geographic disaster recovery capabilities

• Performance Insights analyzes query performance across all replicas

7. NFS Server

All media data and extra data, if required, gets stored in Amazon EFS. It is a centralized and fault-tolerant NFS service. EFS provides shared storage across several instances.

EFS implementation details:

• Automated mount targets exist in each availability zone

• Throughput scales with storage size without manual intervention

• Lifecycle policies move infrequent files to cheaper storage

• The /media directory stores product images and uploads

• The /var/import directory shares import files between instances

• Proper NFS mounting occurs through /etc/fstab configuration entries

8. ElastiCache

The Redis infrastructure has one master server managed using the Elasticache service. Magento uses the master Redis server for FPC and session data in separate databases. ElastiCache provides managed Redis clusters for caching.

Redis configuration details:

• Redis stores sessions, page cache, and backend cache

• Cluster mode enables horizontal scaling across several nodes

• Automatic failover promotes replicas during primary node failures

• The redis-cli connects to clusters for debugging operations

• Parameter groups optimize Redis settings for Magento usage

• Backup retention enables point-in-time recovery when necessary

9. Admin Server

Admin servers handle backend operations and cron jobs. These instances run outside the Auto Scaling Groups. Deployment scripts originate from the admin server locations.

Admin server responsibilities:

• The server hosts Jenkins or deployment automation tools

• Database migrations execute from admin servers during deployments

• Static content generation occurs before code synchronization begins

10. CloudFront CDN

CloudFront serves static assets from edge locations worldwide. Origin requests fetch content from S3 or ELB. Cache behaviors define TTL values per file type.

CDN implementation details:

• The base_static_url configuration points to CloudFront distributions

• Invalidations clear cached content after deployment completes

• Signed URLs protect premium content from unauthorized access attempts

• Origin request policies forward necessary headers to backends

• Compression reduces bandwidth usage for text-based assets

• Real-time logs stream to S3 for analysis purposes

11. Test Server

Test servers confirm deployments before the production release occurs. These instances mirror production configurations for accuracy. Automated tests verify functionality after each deployment cycle completes.

Testing activities include:

• Safe load testing simulates traffic spikes on the test infrastructure

• Performance profiling identifies bottlenecks before production deployment happens

• Security scanning detects vulnerabilities in code and configuration files

AWS Auto Scaling Architecture Requirements

Requirement 1: Flexible Hosting Platform

For Auto Scaling, an architecture consists of several autonomous nodes. Each node operates independent from the other without shared resources.

Platform requirements:

• Cloud platforms provide APIs for programmatic infrastructure management

• Instance metadata services enable self-configuration during startup processes

• Resource tagging organizes instances into logical groupings

Requirement 2: Multi-Server Environment

Horizontal scaling demands several server instances to run together.

Multi-server architecture elements:

• Load balancers distribute traffic across all healthy instances in equal parts

• Session storage moves to Redis or database backends

• File synchronization keeps code consistent across all servers

• Deployment automation updates all instances without manual intervention

• Monitoring tools track performance across the entire server fleet

Requirement 3: Shared Nothing

A "Shared Nothing" architecture also functions as a scalable system. The nodes have their exclusive resources, such as CPU, memory, and local storage.

Architecture principles:

• Database connections use connection pooling for efficiency improvements

• Cache layers reduce database load during traffic spikes

• Application servers remain stateless for horizontal scaling capability requirements

• Shared storage provides media files through NFS mounting points

• Configuration management maintains consistency across all running instances

Troubleshooting Magento Auto Scaling Common Conflicts

Problem 1: Sync of the latest source code

Launching new EC2 instances from an image (AMI) needs syncing the Magento source code. Do this before putting the instance into service. Code synchronization challenges arise during Auto Scaling events.

AWS CodeDeploy solution approach:

• AWS CodeDeploy automates deployment to Auto Scaling Groups

• The appspec.yml defines deployment hooks and script execution orders

• Git repositories provide source code for deployment processes

• S3 buckets store deployment artifacts for CodeDeploy access permissions

• Deployment configurations specify how many instances update together

• Health checks verify successful deployments before routing traffic begins

Problem 2: Varnish Cache Auto Scaling

For multi-server environment Varnish operations, define all backends (instances) in the config file. Static backend definitions break with dynamic infrastructure.

The open source version of Varnish doesn't have autoscaling and clustering capabilities. But Varnish Enterprise does offer this.

Varnish auto-scaling solutions:

• Service discovery updates Varnish backends during scaling operations

• The varnish-discovery service monitors Auto Scaling Group changes

• DNS-based backends resolve to several IP addresses

• Health checks remove failed backends from rotation afterwards

Problem 3: Auto Scaling Code Deployment

New instances need updated code before receiving traffic. A small shell script syncs the admin server's source files before starting Nginx. Deployment timing affects application availability during scaling events.

Deployment strategies:

• CodeDeploy lifecycle hooks pause instance registration until deployment completes

• Blue-green deployments lessen risk during major updates

• Canary deployments test changes on a subset of instances first

Problem 4: Session Management Across Instances

User sessions must persist across several server instances. Redis provides collective centralized session storage for all instances. Sticky sessions at the load balancer level create scaling limitations.

Session management implementation:

• The env.php configuration specifies Redis connection parameters throughout

• Session garbage collection runs on dedicated cron servers

• Backup strategies protect session data from unexpected Redis failures

Problem 5: Media File Synchronization

Product images must synchronize across all running instances. Using Amazon S3 buckets allows users to keep the disk size of their deployment images low. They can only include the code and static assets. S3 provides centralized media storage for all instances.

Media handling approach:

• Global CDN integration serves media files from edge locations

• The pub/media directory syncs to S3 using CLI tools

• Magento configuration points media URLs to CDN endpoints

Lessons Learned for Making Auto Scaling Most Efficient

1. Premature Scale-up

We learned to scale up early when the average CPU usage is 55% or higher. This occurs for 60 to 120 seconds. But it takes up to 2 to 3 minutes for the new instances to be operational. Early scaling prevents performance degradation during traffic increases.

Scaling timing best practices:

• CloudWatch alarms trigger before reaching critical thresholds

• Instance warmup includes code synchronization and cache priming operations

• Target tracking policies maintain performance metrics within acceptable ranges

2. Scale Down Slow

A gradual scale-down prevents aggressive cost optimization from affecting performance.

Scale-down considerations:

• Longer cooldown periods confirm that traffic patterns stabilize before termination

• Adequate, lowest instance counts maintain baseline capacity during quiet periods

• Cost analysis balances performance requirements against infrastructure spending

• Reserved instances reduce costs for baseline capacity needs

• Spot instances provide cost-effective scaling for non-critical workloads

3. Cache Warmup Strategies

New instances start with empty caches, affecting initial performance. Cache warmup scripts pre-populate critical pages before traffic arrives. Varnish grace mode serves stale content during warmup periods.

Warmup implementation methods:

• The n98-magerun2 tool provides cache warming functionality for Magento

• Sitemap URLs guide cache warming priority for important pages

• Health check delays allow the cache population before traffic routing begins

4. Monitoring and Alerting

Comprehensive monitoring detects issues before customers experience problems. CloudWatch dashboards visualize metrics across the entire Auto Scaling infrastructure. Custom metrics track Magento-specific performance indicators like checkout rates.

Monitoring components:

• SNS notifications alert teams when scaling events occur

• Log aggregation identifies patterns across several instances

• APM tools provide transaction-level visibility into application performance bottlenecks

FAQs

1. What instance types work best for Magento Auto Scaling?

C5 instances provide optimal CPU performance for Magento workloads. Memory-optimized instances suit stores with large catalogs. The c5.xlarge instance type balances cost and performance.

2. How does Varnish configuration work with Auto Scaling?

The inventory of nodes gets stored in /etc/varnish/nodes.conf. Service discovery tools update backend definitions during scaling. Varnish Enterprise provides built-in clustering capabilities for dynamic environments.

3. What happens to running cron jobs during scale-down?

Lifecycle hooks delay termination until cron jobs complete execution. Dedicated admin servers always handle long-running cron processes. SQS queues distribute short tasks across several instances.

4. How much does Auto Scaling cost for Magento stores?

AWS charges each store based on the infrastructure used in your AWS account. Costs vary based on instance types and scaling patterns. Reserved instances reduce baseline infrastructure costs by 50%.

5. Can Auto Scaling work with Magento 1 stores?

Magento 1 requires added configuration for horizontal scaling compatibility. Turpentine extension enables Varnish support for Magento 1 stores. Session management needs Redis configuration for multi-instance deployments.

Summary

Auto Scaling transforms Magento hosting from static to dynamic infrastructure. AWS Auto Scaling increases/decreases the capacity of AWS services to optimize costs. Marketing campaigns no longer threaten site availability or performance. Below are the main points from the article:

• Varnish integration requires careful configuration for dynamic backend management

• Service discovery tools maintain backend definitions during scaling events

• Enterprise Varnish provides native clustering support for production deployments

• Cost optimization occurs through automatic resource changes based on demand patterns

• Performance remains consistent regardless of random traffic volume changes

• Infrastructure complexity increases, but automation tools manage operational overhead

Consider Managed Magento hosting to guarantee good code sync and session management.

[Updated June 19, 2025]