AWS S3 for Magento Open Source: Configuration and Performance Tips
One of the core modules of the aMiSTACX S3-Titanium suite is S3R-Alpha for AWS S3 Bucket storage. Since S3R-Alpha’s original release back in June of 2018, the module has grown with an extensive set of features developed from real-life customer experience and feedback.
A glimpse of S3R-Alpha features:
- WebP Compression Support w/ Fallback
- Async media sync commands
- Custom Folder/File sync to S3 [Instead of all or none]
- Static Enable/Disable from CLI
- Verbose Logging
- Image Database Cleanup
- Resize and Sync to S3
- Increased S3 Security Options
- Multi S3 Bucket Support
- Encrypted Module
- Magento 2.1+
Initial Deployment
On designated stacks, the S3R-Alpha module for AWS S3 is pre-installed, and with a few steps ready to go.
As our stacks are designed and considered expert series, we will not go into a lengthy and exhaustive documentation of AWS services and procedures. AWS and S3 are very well documented.
Note: This module is designed to work only with aMiSTACX, and only with AWS S3 buckets.
Initial Deployment and Activation
Should you not see the aMiSTACX S3 module in Magento’s admin dashboard after a fresh CloudFormation deployment, then follow these steps:
From CLI:
cd /var/www/magento
sudo chown -R www-data:www-data /var/www/magento
sudo composer updatesudo ./amistacx.sh #helper script found in magento's doc root
Activation Section
Step 1. Set module to enabled >> Save Configuration [Clear Cache when prompted]
Step 2. Create IAM S3 Keys [Programmatic Access] with the attached AmazonS3FullAccess permission, and a custom policy you can call it anything, but we will call it: EC2DescribeRegions. Create the policy in IAM, and copy and paste the json, and then attach the policy to the new user.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ec2:DescribeRegions",
"Resource": "*"
}
]
}
Step 2b. Add the AWS IAM S3 Keys you created >> Save Configuration [Clear Cache if prompted]
Step 3. Click the “Create a new bucket button” >> Enter the S3 Bucket’s name and S3 location >> Save Configuration [Clear Cache if prompted]
Note : Normally you would enter the bucket’s location as the same as the primary EC2’s location. For example, if my EC2 is in Canada, then I’d create an S3 Bucket in Canada. Additionally, since we are using the AWS API, the bucket’s name creation must conform to AWS S3 naming conventions.
Performance Tip: You may find it useful or a possible performance advantage to separate media into several buckets. Maybe to designate content from Store A and Store B? S3R-Alpha supports Multi Buckets, but Magento may not play nice. You will want to test and load test a multi bucket architecture in QA.
Step 4. More than likely and recommended, you’ll want to use a custom domain for URL and SEO performance. Select your bucket in the drop-down menu [Image Item 1b.], set Custom Domain to “Yes”, enter the Custom Domain [Image Item 2.], and set HTTPS to “Yes”. >> Save Configuration [Clear Cache if prompted]
Content Configuration Section [Optional]
Custom Path : The Magento’s system value default is pub/media; however, you may want to use a custom image URL. Results may vary, so make sure you QA test any changes to the default.
e.g. https://s3r.awsamistacx.com/pub/media/optimized/wysiwyg/home/home-main-jpg-min.webp
Upload Static Content : Static content such as CSS and js will be served from S3. Results may vary and will depend on the quality of the theme and components you use. Performance may only slightly improve, and may not be worth the addition maintenance and support required for functionality. For consultant projects, normally aMiSTACX uses the “No” setting.
We recommend the following for static testing.
1. Leave static for last. Make sure S3 is working correctly for media.
2. Before you enable static, take a full AMI EC2 image w/ RDS snapshot.
3. After you enable static, sync with the following command:
sudo php bin/magento s3rocket:bucket:sync --static-content
!Static Rollback
If you need to rollback S3 static back to local server, then perform the following:
Option 1. Connect to the Magento database, and locate this entry in the core_config_data table:
s3/content_configuration/static_content
Set the value from 1 to 0
Option 2. Or you can try the following:
sudo php bin/magento s3rocket:settings:set static-content 0 # Disable
Then from Magento:
cd /var/www/magentosudo php bin/magento cache:cleansudo chown -R www-data:www-data /var/www/magentosudo php bin/magento setup:static-content:deploy -f
Tip! Should you still not see everything displayed correctly, run the recompile script amistacx.sh.
Compress Images: You’ll more than likely want to set this to “Yes” WebP is now supported by most major browsers, and S3R-Alpha offers image fallback when WebP is not supported by the browser client. You also have the option of compression on compression, but it is not often recommended or used. It is highly recommended you optimize all images prior to upload to Magento.
Advanced Configuration Section [Optional]
Enable S3 Logger : For development environments and for advanced troubleshooting you will want to set to “Yes”. Do not leave ON for production environments. The path to the log is in the default Magento log location under the amistacx folder.
i.e /var/www/magento/var/log/amistacx/
Sync Product Images Automatically : Normally you would set to “No”, but if you frequently upload images and find they are not syncing to S3, then you can set to “Yes”. If the image is not sync’d to S3 when the page loads, S3R-Alpha will upload the image during the first page load. The first customer to access the page will encounter a longer page load time, but each additional hit to the page will render the image without the additional overhead of the sync.
Allow from IP : Additional image security to prevent hot-linking and bots. This allows access to the S3 bucket from only the Magento server with a specific IP address. Requests from other servers, or a direct request will be blocked.
Allow from Referrer : Additional image security to prevent hot-linking and bots. This allows access to the S3 bucket from only the Magento server with a specific URL/Domain. Requests from other domains, or a direct requests will be blocked.
Image Sync to S3 Bucket
We will assume you are testing in your QA development environment. In the past, we offered an image sync button that was accessible from the Magento Admin dashboard, but because of increased complexity, we are limiting the S3 sync function to CLI.
Initial Sync
cd /var/www/magento
sudo chown -R www-data:www-data /var/www/magentosudo php bin/magento catalog:images:resizesudo php bin/magento s3rocket:bucket:sync -a -m
The image resize is recommended for a first run. Subsequent sync operations should not require it. The sync operation uses -a [async] and -m [mode = default] switches. This combination is generally the easiest to use and covers most image catalogs under 5GB.
Command Line Switches [Advanced]
sudo php bin/magento s3rocket:help
Tip! The above command will open the help menu. Use “=” for full name switch/keys, e.g., –mode=3 or short key -m 3 .
-h, --help
Displays the help menu
-q, --quiet
Do not output any messages
-V, --version
Display this application version
sudo php bin/magento s3rocket:bucket:sync [options]
--custom-folder[=CUSTOM-FOLDER]
You can sync any directory/file from the local media to the S3 bucket.
For example: --custom-folder catalog or --custom-folder catalog cache
For example: sudo php bin/magento s3rocket:bucket:sync --custom-folder /pub/media/styles.css
-a, --async Async
To save sync time, you will want to make sure you use the -a async switch.
-c, --cache
To save a lot of sync time, you can sync cache only.
-o, --offset[=OFFSET]
Start from files [default: 0]. This is useful when your sync operation gets paused or stopped. Instead of starting from the very beginning, you can resume operation from the last known file number identifier. Very handy when image syncs can potentially take hours or days.
-l, –limit[=LIMIT]
File limit. Useful when you want to batch upload/sync to S3.
-e, –exclude-folder[=EXCLUDE-FOLDER]
Exclude Folder (You can put the single folder name or comma-separated)
Some folders in your media gallery you may not want to sync, or you may have already synced these folders and don’t want to sync them again.
-m, –mode[=MODE]
0: Upload ALL content including optimized images [Default]. 1: Upload only original content. 2: Upload only optimized content 3: Upload WebP only.
Note: If mode is not specified, then the default is 0.
-q, –quiet
Do not output any messages
–ansi,
Force ANSI output
–no-ansi,
Disable ANSI output
-n, –no-interaction
Do not ask any interactive questions
-sc, –static-content
Very useful after enable static in the admin console. To avoid re-sync everything, just sync the static.
-v|vv|vvv, –verbose
Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
sudo php bin/magento s3rocket:bucket:cleanup [options]
Note: Use this command to cleanup orphaned images in the database.
Warning! Make a full backup of the stack and database before starting. It is highly recommend to clone and QA the cleanup first.
--dry-run
Will tell you the amount of images ready to be removed without making any changes.
--remote
Remove invalid images from the S3 Bucket
--remove-original
Remove original images (Only for AWS S3)
--remove-optimized
Remove optimized images (Only for AWS S3)
Example will cleanup local first, then the second line will cleanup the remote in the S3 bucket.
sudo php bin/magento s3rocket:bucket:cleanupsudo php bin/magento s3rocket:bucket:cleanup --remote
–q, --quiet
Do not output any messages
--ansi,
Force ANSI output
--no-ansi,
Disable ANSI output
-n, --no-interaction
Do not ask any interactive questions
-v|vv|vvv, --verbose
Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
sudo php bin/magento s3rocket:bucket:resize [options]
Note: This command adds additional features and performance to Magento’s image resize command.
-t, --threads[=THREADS]
Threads [default: 1]
-o, --offset[=OFFSET]
Offset [default: 0]
-r, --remote
Upload to S3 while resizing
-q, --quiet
Do not output any messages
--ansi,
Force ANSI output
--no-ansi,
Disable ANSI output
-n, --no-interaction
Do not ask any interactive questions
-v|vv|vvv, --verbose
Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
The below example command will resize images and also sync to remote using a thread count of two.
sudo php bin/magento s3rocket:images:resize -r -t=2
Troubleshooting
Some images are not loading on the frontend, or appear distorted in size.
cd /var/www/magentosudo chown -R www-data:www-data /var/www/magentosudo php bin/magento catalog:images:resize
Note: You can use S3R-Alpha’s image resize commands for better performance.
Mixed content warnings
Encountered when using custom domains, and a CDN such as Cloudflare.
Tip! You need to have Cloudflare S3 CNAME set to CDN ON [Orange Cloud]. If you do not, then you will get mixed content warnings. Also make sure the S3 domain that Cloudflare is using has the Cloudflare SSL set to FULL.
Wait about 15-30 minutes for Cloudflare to kick in before you upload any Magento media content; otherwise, you might get certificate warnings. For example, say Magento sets a path to a media image that looks like this:
But when Cloudflare CDN has not fully propagated, you will get served a URL that looks like this:
Plus, you will get a certificate warning.
Tip 2. Static Content Deployment to S3
! Not all themes or components are compatible with static deployment.
Prior to deploying static, make a full image and or database backup, also make sure you can connect to your database prior to upload.
When used in this configuration, we have removed Redis caching, and set CSS and JS settings in the Magento Console as follows:
Note: Make sure you test this on Dev environment first. Also, make sure you deploy your media files first, take a backup, and then test deployment of Static content to S3.
Additionally, when deploying static content to S3 you will need to follow the following steps to ensure success:
From CLI:
cd /var/www/magento
sudo php bin/magento cache:flush config
sudo chown -R www-data:www-data /var/www/magento
Tip! If you are also using Cloudflare, you may need to flush the cache on your domain. You can do this from A51.
Static Rollback
If you need to rollback static to local, perform the following:
Connect to the Magento database, and find this entry in the core_config_data table:
s3/content_configuration/static_content
Set the value from 1 to 0
Then from Magento:
cd /var/www/magentosudo php bin/magento cache:cleansudo chown -R www-data:www-data /var/www/magentosudo php bin/magento setup:static-content:deploy -f
Install & Update Errors
Should you receive a GIT error during install, it is because our stacks set root owner on the .git directory in vendor, while the rest of the project uses ww-data user/group. Root is set from cron in order to help prevent exploitation of GIT updates in clear text.
GIT was recently patched in April 2022 to close a vulnerability and this new patch causes a permissions check.
In future builds, we will add the exception automatically; however, you may need to explicitiy add it:
sudo git config --global --add safe.directory /var/www/magento/vendor/amistacx/
Launch
Paying attention to details and making sure you follow your aMiSTACX admin guide, and of course using your own experience, will help make sure your deployment is successful.
Video of S3R-Alpha Deployment and Configuration.
Latest S3R-Alpha Version
/var/www/magento/vendor/amistacx/{module name}/CHANGELOG.md
[1.4.0] – 2023-0514
### Added
+ Fixed S3R to handle new default bucket policy
Ref: AWS announcement
Weaponize you Business with aMiSTACX!