No image

aMiSTACX S3R-Alpha for AWS S3 Media Storage

AWS S3 for Magento Open Source: Configuration and Performance Tips

One of the core modules of the aMiSTACX S3-Titanium suite is S3R-Alpha for AWS S3 Bucket storage. Since S3R-Alpha’s original release back in June of 2018, the module has grown with an extensive set of features developed from real-life customer experience and feedback.

A glimpse of S3R-Alpha features:

  • WebP Compression Support w/ Fallback
  • Async media push commands
  • Custom Folder Pushes to S3 [Instead of all or none]
  • Verbose Logging
  • Image Database Cleanup
  • Resize and Sync to S3
  • Increased S3 Security Options
  • Multi S3 Bucket Support
  • Encrypted Module
  • Magento 2.1+

 

Initial Deployment

The S3R-Alpha module for AWS S3 is pre-installed and is equipped with auto-licensing.

As our stacks are designed and considered expert series, we will not go into a lengthy and exhaustive documentation of AWS services and procedures. AWS is very well documented.

Note: This module is designed to work only with aMiSTACX, and only with AWS S3. The module is not available for individual purchase, and is only available for use with designated aMiSTACX deployments.

Initial Deployment and Activation

Should you not see the aMiSTACX S3 module in Magento’s admin dashboard after a fresh CloudFormation deployment, then follow these steps:

From CLI:

cd /var/www/magento
sudo chown -R www-data:www-data /var/www/magento
sudo composer update
sudo ./amistacx.sh #helper script found in magento's doc root

Activation Section

s3r-alpha activation

Step 1. Set module to enabled >> Save Configuration [Clear Cache if prompted]

Step 2. Add the AWS IAM S3 Keys you created >> Save Configuration [Clear Cache if prompted]

S3R-Alpha Module Configuration


Step 3.
Click the “Create a new bucket button” >> Enter the S3 Bucket’s name and S3 location >> Save Configuration [Clear Cache if prompted]

Note : Normally you would enter the bucket’s location as the same as the primary EC2’s location. For example, if my EC2 is in Canada, then I’d create an S3 Bucket in Canada. Additionally, since we are using the AWS API, the bucket’s name creation must conform to AWS S3 naming conventions.

Performance Tip: You may find it useful or a possible performance advantage to seperate media into several buckets. Maybe to designate content from Store A and Store B? S3R-Alpha supports Multi Buckets, but Magento may not play nice. You will want to test and load test a multi bucket architeture in QA.

Step 4. More than likely and recommended, you’ll want to use a custom domain for URL and SEO performance. Select your bucket in the drop-down menu [Image Item 1b.], set Custom Domain to “Yes”, enter the Custom Domain [Image Item 2.], and set HTTPS to “Yes”. >> Save Configuration [Clear Cache if prompted]

s3r-alpha module for AWS S3. Module content configuration in Magento admin panel.

Content Configuration Section [Optional]

Custom Path : The Magento’s system value default is pub/media; however, you may want to use a custom image URL. Results may vary, so make sure you QA test any changes to the default.

e.g. https://s3r.awsamistacx.com/pub/media/optimized/wysiwyg/home/home-main-jpg-min.webp

https://s3r.awsamistacx.com/images/optimized/wysiwyg/home/home-main-jpg-min.webp

Upload Static Content : Static content such as CSS and js will be served from S3. Results may vary and will depend on the quality of the theme and components you use. Performance may only slightly improve, and may not be worth the addition maintenance and support required for functionality. For consultant projects, normally aMiSTACX uses the “No” setting.

!Static Rollback

If you need to rollback S3 static to server from local, perform the following:

Connect to the Magento database, and locate this entry in the core_config_data table:

s3/content_configuration/static_content

Set the value from 1 to 0

Then from Magento:

cd /var/www/magento
sudo php bin/magento cache:clean
sudo chown -R www-data:www-data /var/www/magento
sudo php bin/magento setup:static-content:deploy -f

Compress Images: You’ll more than likely want to set this to “Yes” WebP is no supported by most browsers, and S3R-A has offers image fallback when WebP is not supported at the client. You also have the option of compression on compression, but it is not often recommended or used. It is highly recommended you optimize all images prior to upload to Magento.

Advanced Configuration Section [Optional]

Enable S3 Logger : For development environments and for advanced troubleshooting you will want to set to “Yes”. Do not leave ON for production environments. The path to the log is in the default Magento log location under the amistacx folder.

i.e /var/www/magento/var/log/amistacx/

Sync Product Images Automatically : Normally you would set to “No”, but if you frequently upload images and find they are not syncing to S3, then you can set to “Yes”. If the image is not sync’d to S3 when the page loads, S3R-Alpha will upload the image during the first page load. The first customer to access the page will encounter a longer page load time, but each additional hit to the page will render the image without the additional overhead of the sync.

Allow from IP : Additional image security to prevent hot-linking and bots. This allows access to the S3 bucket from only the Magento server with a specific IP address. Requests from other servers, or a direct request will be blocked.

Allow from Referrer : Additional image security to prevent hot-linking and bots. This allows access to the S3 bucket from only the Magento server with a specific URL/Domain. Requests from other domains, or a direct requests will be blocked.

Image Sync to S3 Bucket

We will assume you are testing in your QA development environment. In the past, we offered an image sync button that was accessible from the Magento Admin dashboard, but because of increased complexity, we are limiting the S3 sync function to CLI.

Initial Sync

cd /var/www/magento
sudo chown -R www-data:www-data /var/www/magento
sudo php bin/magento catalog:images:resize
sudo php bin/magento s3rocket:bucket:sync -a -m

The image resize is recommended for a first run. Subsequent sync operations should not require it. The sync operation uses -a [async] and -m [mode = default] switches. This combination is generally the easiest to use and covers most image catalogs under 5GB.

Command Line Switches [Advanced]

sudo php bin/magento s3rocket:help

#Opens the switch help menu for the module

-h, –help
Displays the help menu

-q, –quiet
Do not output any messages

-V, –version
Display this application version

sudo php bin/magento s3rocket:bucket:sync [options]

–custom-folder[=CUSTOM-FOLDER]
You can sync any directory/file from the local media to the S3 bucket.

For example: –custom-folder catlog or –custom-folder catlog cache

-a, –async Async
To save sync time, you will want to make sure you use the -a async switch.

-o, –offset[=OFFSET]
Start from files [default: 0]. This is useful when your sync operation gets paused or stopped. Insead of starting from the very begining, you can resume operation from the last known file number identifer. Very handy when image syncs can potentially take hours or days.

-l, –limit[=LIMIT]
File limit. Useful when you want to batch upload/sync to S3.

-e, –exclude-folder[=EXCLUDE-FOLDER]
Exclude Folder (You can put the single folder name or comma-separated)
Some folders in your media gallery you may not want to sync, or you may have already synced these folders and don’t want to sync them again.

-m, –mode[=MODE]
0: Upload ALL content including optimized images [Default]. 1: Upload only original content. 2: Upload only optimized content 3: Upload WebP only.

Note: If mode is not specified, then the default is 0.

-q, –quiet
Do not output any messages

–ansi,
Force ANSI output

–no-ansi,
Disable ANSI output

-n, –no-interaction
Do not ask any interactive questions

-v|vv|vvv, –verbose
Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

 

sudo php bin/magento s3rocket:bucket:cleanup [options]

Note: Use this command to cleanup orphaned images in the database.

Warning! Make a full backup of the stack and database before starting. It is highly recommend to clone and QA the cleanup first.

–dry-run
Will tell you the amount of images ready to be removed without making any changes.

–remote
Remove invalid images from the S3 Bucket

–remove-original
Remove original images (Only for AWS S3)

–remove-optimized
Remove optimized images (Only for AWS S3)

Example will cleanup local first, then the second line will cleanup the remote in the S3 bucket.

sudo php bin/magento s3rocket:bucket:cleanup
sudo php bin/magento s3rocket:bucket:cleanup --remote

-q, –quiet
Do not output any messages

–ansi,
Force ANSI output

–no-ansi,
Disable ANSI output

-n, –no-interaction
Do not ask any interactive questions

-v|vv|vvv, –verbose
Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

sudo php bin/magento s3rocket:bucket:resize [options]

Note: This command adds additional features and performance to Magento’s image resize command.

-t, –threads[=THREADS]
Threads [default: 1]

-o, –offset[=OFFSET]
Offset [default: 0]

-r, –remote
Upload to S3 while resizing

-q, –quiet
Do not output any messages

–ansi,
Force ANSI output

–no-ansi,
Disable ANSI output

-n, –no-interaction
Do not ask any interactive questions

-v|vv|vvv, –verbose
Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Troubleshooting

Some images are not loading on the frontend, or appear distorted in size.

cd /var/www/magento
sudo chown -R www-data:www-data /var/www/magento
sudo php bin/magento catalog:images:resize

Magento Developer Docs Ref:

Note: You can use S3R-Alpha’s image resize commands for better performance.

Mixed content warnings

Encountered when using custom domains, and a CDN such as Cloudflare.

Tip 1. You need to have Cloudflare S3 CNAME set to CDN ON [Orange Cloud]. If you do not, then you will get mixed content warnings. Also make sure the S3 domain that Cloudflare is using has the Cloudflare SSL set to FULL.

Wait about 15-30 minutes for Cloudflare to kick in before you upload any Magento media content; otherwise, you might get certificate warnings. For example, say Magento sets a path to a media image that looks like this:

https://s3rtest.awsamistacx.com/catalog/product/6/4/640.png

But when Cloudflare CDN has not fully propagated, you will get served a URL that looks like this:

https://s3rtest.awsamistacx.com.s3-us-west-2.amazonaws.com/catalog/product/6/4/640.png

Plus, you will get a certificate warning.

Tip 2. Static Content Deployment to S3

! Not all themes or components are compatible with static deployment.

Prior to deploying static, make a full image and or database backup, also make sure you can connect to your database prior to upload.

S3Rocket Deploy CSS to S3

When used in this configuration, we have removed Redis caching, and set CSS and JS settings in the Magento Console as follows:

Magento 2.2.x Developer CSS JS Settings

Note: Make sure you test this on Dev environment first. Also, make sure you deploy your media files first, take a backup, and then test deployment of Static content to S3.

Additionally, when deploying static content to S3 you will need to follow the following steps to ensure success:

From CLI:

cd /var/www/magento

sudo chown -R www-data:www-data /var/www/magento
sudo php bin/magento setup:upgrade
sudo chown -R www-data:www-data /var/www/magento
sudo php bin/magento setup:di:compile
sudo chown -R www-data:www-data /var/www/magento
sudo php bin/magento setup:static-content:deploy -f
sudo chown -R www-data:www-data /var/www/magento
sudo php bin/magento cache:enable

*Must use the -f switch.

Static Rollback

If you need to rollback static to local, perform the following:

Connect to the Magento database, and find this entry in the core_config_data table:

s3/content_configuration/static_content

Set the value from 1 to 0

Then from Magento:

cd /var/www/magento
sudo php bin/magento cache:clean
sudo chown -R www-data:www-data /var/www/magento
sudo php bin/magento setup:static-content:deploy -f

Install & Update Errors

Should you receive a GIT error during install, it is because our stacks set root ower on the .git directory in vendor, while the rest of the project uses ww-data user/group. Root is set from cron in order to help prevent exploitation of GIT updates in clear text.

GIT was recently patched in April 2022 to close a vulnerability and this new patch causes a permissions check.

In future builds, we will add the exception automatically; however, you may need to explicitiy add it:

sudo git config --global --add safe.directory /var/www/magento/vendor/amistacx/

Launch

Paying attention to details and making sure you follow your aMiSTACX admin guide, and of course using your own experience, will help make sure your deployment is successful.

Note: Should you have interest in using S3 Titanium with AWS Auto Scale, please review this post.

Latest S3R-Alpha Version

## [1.3.5] – 2022-11-15
### Added
+ New Image Cache Creator

## [1.3.4] – 2022-11-13
### Added
+ New CLI Mode to allow WebP only sync

## [1.3.3] – 2022-11-11
### Added
+ Create and upload image cache automatically

 

Weaponize you Business with aMiSTACX!