Skip to content

Commit e29a414

Browse files
tdgroottdgroot
authored
Merge pull request #320 from ByteInternet/horizontal_autoscaling_improvements
docs: Horizontal autoscaling improvements
2 parents ff5f569 + a03b61e commit e29a414

File tree

5 files changed

+256
-105
lines changed

5 files changed

+256
-105
lines changed

‎bin/watch‎

Lines changed: 45 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
#!/usr/bin/env python3
2-
import datetime
2+
import os
33
import subprocess
4-
5-
from inotify.adapters import InotifyTree
4+
import time
5+
from pathlib import Path
6+
from time import sleep
67

78

89
def build_docs():
@@ -15,54 +16,57 @@ def refresh_assets():
1516
)
1617

1718

18-
def watch_for_changes(debug=False):
19-
i = InotifyTree("docs")
19+
def watch_for_changes():
20+
"""Watch for changes using watchdog."""
21+
from watchdog.events import FileSystemEvent, FileSystemEventHandler
22+
from watchdog.observers import Observer
23+
24+
class MyHandler(FileSystemEventHandler):
25+
def __init__(self):
26+
super().__init__()
27+
self.last_build = time.time()
2028

21-
IGNORED_EVENTS = ["IN_CLOSE_WRITE", "IN_CLOSE_NOWRITE", "IN_ACCESS", "IN_OPEN"]
22-
IGNORED_PATHS = ["docs/_build"]
29+
def on_modified(self, event: FileSystemEvent):
30+
cwd = os.getcwd()
31+
file_path = Path(event.src_path)
2332

24-
last_build = None
33+
IGNORED_PATHS = ["docs/_build"]
2534

26-
for event in i.event_gen(yield_nones=False):
27-
(_, type_names, path, filename) = event
35+
if event.is_directory:
36+
return
2837

29-
ignore = False
30-
for ignored_event_type in IGNORED_EVENTS:
31-
if ignored_event_type in type_names:
32-
ignore = True
33-
break
38+
if file_path.name.isdigit():
39+
return
3440

35-
for ignored_path in IGNORED_PATHS:
36-
if path.startswith(ignored_path):
37-
ignore = True
38-
break
41+
if (
42+
file_path.name.endswith(".swx")
43+
or file_path.name.endswith(".swp")
44+
or file_path.name.endswith("~")
45+
):
46+
return
3947

40-
if (
41-
filename.endswith(".swx")
42-
or filename.endswith(".swp")
43-
or filename.endswith("~")
44-
):
45-
ignore = True
48+
for ignored_path in IGNORED_PATHS:
49+
ignored_path = os.path.join(cwd, ignored_path)
50+
if str(file_path).startswith(ignored_path):
51+
return
4652

47-
if ignore:
48-
continue
53+
if self.last_build and time.time() - self.last_build < 0.5:
54+
return
4955

50-
if last_build:
51-
# If build has been triggered in the last 50 ms, skip
52-
delta = (datetime.datetime.now() - last_build).microseconds / 1000
53-
if delta < 50:
54-
continue
56+
build_docs()
57+
refresh_assets()
58+
self.last_build = time.time()
5559

56-
build_docs()
57-
refresh_assets()
58-
last_build = datetime.datetime.now()
60+
event_handler = MyHandler()
61+
observer = Observer()
62+
observer.schedule(event_handler, path="docs", recursive=True)
63+
observer.start()
5964

60-
if debug:
61-
print(
62-
"PATH=[{}] FILENAME=[{}] EVENT_TYPES={}".format(
63-
path, filename, type_names
64-
)
65-
)
65+
try:
66+
while True:
67+
sleep(1)
68+
except KeyboardInterrupt:
69+
observer.stop()
6670

6771

6872
if __name__ == "__main__":

‎docs/best-practices/database/how-to-run-rabbitmq-on-hypernode.md‎

Lines changed: 24 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ You can run this command to restart RabbitMQ:
5555
hypernode-servicectl restart rabbitmq-server
5656
```
5757

58-
## Accessing RabbitMQ
58+
## Accessing the RabbitMQ admin interface
5959

6060
- RabbitMQ only binds on localhost
6161
- The default admin account is username `guest` and password `guest`. You can change and add users via the admin interface.
@@ -67,22 +67,29 @@ Use your browser to go to `localhost:55672` and logon using guest/guest.
6767

6868
Another way to access the admin interface is via [hypernode-vpn](https://changelog.hypernode.com/changelog/release-6064-rabbitmq-can-be-accessed-via-the-hypernode-vpn/)
6969

70-
## RabbitMQ and Magento
71-
72-
You need to make some changes in Magento in order to use RabbitMQ. For example in `/data/web/magento2/app/etc/env.php`:
73-
74-
```php
75-
'queue' =>
76-
array (
77-
'amqp' =>
78-
array (
79-
'host' => 'rabbitmqmaster',
80-
'port' => '5672',
81-
'user' => 'guest',
82-
'password' => 'guest',
83-
'virtualhost' => '/',
84-
),
85-
),
70+
## Creating RabbitMQ users
71+
72+
You can also create your own RabbitMQ users. This can be done by in the RabbitMQ admin interface, as follows:
73+
74+
1. Go to the RabbitMQ admin interface
75+
1. Click on the `Admin` tab
76+
1. Click on `Add a user`
77+
1. Fill in the username and password
78+
1. Click on `Add user`
79+
1. Click on the user you just created
80+
1. Click on `Set permission`
81+
82+
## RabbitMQ and Magento 2
83+
84+
To configure RabbitMQ in Magento 2, you can run the following command:
85+
86+
```bash
87+
bin/magento setup:config:set \
88+
--amqp-host="rabbitmqmaster" \
89+
--amqp-port="5672" \
90+
--amqp-user="guest" \
91+
--amqp-password="guest" \
92+
--amqp-virtualhost="/"
8693
```
8794

8895
Note: Hypernode provisions a non-default user called `hypernode-admin` but you are free to create new users.

‎docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
---
2+
myst:
3+
html_meta:
4+
description: Configure remote storage for Magento 2.x. Learn how to configure
5+
Magento 2 to start storing files in your bucket using a single command.
6+
title: How to Configure Remote Storage for Magento 2.x | Hypernode
7+
---
8+
9+
# How to Configure Remote Storage for Magento 2.x
10+
11+
Magento 2.x supports remote storage for media files, import/export files, and other files.
12+
This feature allows you to store files in a remote storage location, such as an Amazon S3 bucket, instead of storing them on the server itself.
13+
14+
This can be useful for many reasons, such as:
15+
16+
- Offloading storage from your server, reducing the load on your server, and improving performance.
17+
- Allows you to make use of [horizontal scaling](../../hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work), as you can easily add more servers without having to worry about syncing files between them.
18+
- Allows for effortless storage capacity scaling, as you can easily increase the storage capacity of your remote storage location.
19+
- Serving assets from a CDN, which can improve the performance of your website.
20+
21+
## Configuring the application
22+
23+
Configuring Magento 2 to start storing files in your bucket is done using a single command.
24+
25+
**AWS S3**
26+
27+
```bash
28+
bin/magento setup:config:set \
29+
--remote-storage-driver="aws-s3" \
30+
--remote-storage-bucket="my_bucket_name" \
31+
--remote-storage-region="my-aws-region" \
32+
--remote-storage-key="abcd1234" \
33+
--remote-storage-secret="abcd1234"
34+
```
35+
36+
**Other S3 compatible providers**
37+
38+
If you're using a different provider than AWS S3, you need to specify the `--remote-storage-endpoint` option.
39+
40+
```bash
41+
bin/magento setup:config:set \
42+
--remote-storage-driver="aws-s3" \
43+
--remote-storage-bucket="my_bucket_name" \
44+
--remote-storage-region="provider-region" \
45+
--remote-storage-key="abcd1234" \
46+
--remote-storage-secret="abcd1234" \
47+
--remote-storage-endpoint="https://my-s3-compatible.endpoint.com"
48+
```
49+
50+
## Syncing the files
51+
52+
Instead of running (which is Magento's official way to do this):
53+
54+
```bash
55+
bin/magento remote-storage:sync
56+
```
57+
58+
One can run the following instead to really speed up the process:
59+
60+
```bash
61+
aws s3 sync pub/media/ s3://my_bucket_name/media/
62+
aws s3 sync var/import_export s3://my_bucket_name/import_export
63+
```
64+
65+
This is much faster than Magento's built-in sync, because `aws s3 sync` uploads files concurrently.
66+
67+
## The storage flag file in the bucket
68+
69+
Magento's S3 implementation creates a test file called `storage.flag`, which is basically created to test if the connection works. So this is not a magic file to mark anything ([source](https://github.com/magento/magento2/blob/6f4805f82bb7511f72935daa493d48ebda3d9039/app/code/Magento/AwsS3/Driver/AwsS3.php#L104)).
70+
71+
## Serving assets from your S3 bucket
72+
73+
To start serving media assets from your S3 bucket, you need to make some adjustments to your nginx configuration.
74+
75+
```nginx
76+
location /media {
77+
# ...
78+
location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ {
79+
resolver 8.8.8.8;
80+
set $bucket "<my_bucket_name>";
81+
proxy_pass https://s3.amazonaws.com/$bucket$uri;
82+
proxy_pass_request_body off;
83+
proxy_pass_request_headers off;
84+
proxy_intercept_errors on;
85+
proxy_hide_header "x-amz-id-2";
86+
proxy_hide_header "x-amz-request-id";
87+
proxy_hide_header "x-amz-storage-class";
88+
proxy_hide_header "Set-Cookie";
89+
proxy_ignore_headers "Set-Cookie";
90+
}
91+
# ...
92+
}
93+
```
94+
95+
Also make sure your S3 bucket policies are configured correctly, so that only `/media` is publicly readable. For example:
96+
97+
```json
98+
{
99+
"Version": "2012-10-17",
100+
"Statement": [
101+
{
102+
"Sid": "AllowPublicReadOnlyForMedia",
103+
"Effect": "Allow",
104+
"Principal": "*",
105+
"Action": "s3:GetObject",
106+
"Resource": "arn:aws:s3:::<your-bucket-name>/media/*"
107+
}
108+
]
109+
}
110+
```
111+
112+
## Magento remote storage documentation
113+
114+
- [Configure Remote Storage](https://experienceleague.adobe.com/en/docs/commerce-operations/configuration-guide/storage/remote-storage/remote-storage)
115+
- [Configure AWS S3 bucket for remote storage](https://experienceleague.adobe.com/en/docs/commerce-operations/configuration-guide/storage/remote-storage/remote-storage-aws-s3)

0 commit comments

Comments
 (0)