diff --git a/docs/deployment-preparation/hardware-requirements.md b/docs/deployment-preparation/hardware-requirements.md
index 5fcd2dd5..2f8972d3 100644
--- a/docs/deployment-preparation/hardware-requirements.md
+++ b/docs/deployment-preparation/hardware-requirements.md
@@ -14,11 +14,10 @@ network bandwidth, and free space on the boot disk.
| Node Type | vCPU(s) | RAM (GB) | Locally Attached Storage | Network Performance | Free Boot Disk | Number of Nodes |
|--------------|---------|------------------------|-----------------------------------|---------------------|----------------|------------------|
-| Storage Node | 8+ | 6+ DDR4 (1) | 2x dedicated NVMe (2) | 10 GBit/s | 10 GB | 3 (3) |
+| Storage Node | 8+ | 6+ DDR4 (1) | 2x dedicated NVMe (2) | 10 GBit/s | 10 GB | 3 (3) |
-1 Simplyblock highly recommends DDR5 memory on storage nodes for optimal performance.
-2 It is possible to test with only one dedicated NVMe, but this is not approved for production.
+2 Test setups require a minimum of 1 nvme. From 26.3., we support non-nvme device clusters (any SATA or SAS linux block device), this feature is still experimental. 3 The required number of nodes is only valid for erasure coding scheme 1+1.
@@ -49,16 +48,13 @@ simplyblock data plane (spdk_80xx containers) and the rest will remain under con
Simplyblock auto-detects NUMA nodes. It will configure and deploy storage nodes per NUMA node.
Each NUMA socket requires directly attached NVMe devices and NICs to deploy a storage node.
-For more information on simplyblock on NUMA, see [NUMA Considerations](numa-considerations.md).
+detected configuration. This file is later processed when adding the storage nodes to the storage host.
+Manual changes to the configuration are possible if the proposed configuration is not applicable.
-It is recommended to deploy multiple storage nodes per storage host if there are more than 32 cores available
-per socket.
+For more information on simplyblock on NUMA, see [NUMA Considerations](numa-considerations.md).
During deployment, simplyblock detects the underlying configuration and prepares a configuration file with the
recommended deployment strategy, including the recommended amount of storage nodes per storage host based on the
-detected configuration. This file is later processed when adding the storage nodes to the storage host.
-Manual changes to the configuration are possible if the proposed configuration is not applicable.
-
### Hyper-Converged Sizing Guidance
As hyper-converged deployments have to share vCPUs, it is recommended to dedicate 15%-20%, but not less than
@@ -68,125 +64,55 @@ As hyper-converged deployments have to share vCPUs, it is recommended to dedicat
### Storage Node Isolation Behavior
!!! warning
- On storage nodes, required vCPUs will be automatically isolated from the operating system. No
- kernel-space, user-space processes, or interrupt handler can be scheduled on these vCPUs. In
- Kubernetes, the CPU Manager and Topology Manager are used for this purpose.
+ On storage nodes, required vCPUs can be isolated from the operating system. No
+ kernel-space, user-space processes, or interrupt handler can be scheduled on these vCPUs.
+ On dedicated storage nodes in Non-Kubernetes deployments, the core isolation is performed
+ automatically on the host, if this option is chosen at deployment time. In
+ Kubernetes, the CPU Manager and Topology Manager are used per default, but core isolation has
+ to be opted-in and requires some additional administrator interventation on k8s and the host.
+ Core isolation can significantly improve tail latency and performance consistency.
### Storage Node Memory Sizing Formula
For RAM, it is required to define the maximum number of NVMe-oF subsystems per node. This depends on
the assigned vCPUs and networking performance of the node. For each 10 GBit/s of dedicated network bandwidth
it is recommended to use at least 3 subsystems. For each vCPU exceeding 8, it is recommended to use one additional
-subsystem. Use the lower of both values (dedicated network bandwidth, vCPUs). A hard limit of 75 subsystems per
+subsystem. Use the lower of both values (dedicated network bandwidth, vCPUs). A hard limit of max. 75 subsystems per
node applies. See [Limits](../reference/limits.md).
-For storage nodes, simplyblock highly recommends DDR5 memory for optimal performance.
-
| Unit | Memory Requirement |
|----------------------------------------------------------|--------------------|
| Fixed amount | 3 GiB |
-| Per subsystem (cluster average per node) | 25 MiB |
-| % of maximum storage capacity (cluster average per node) | 1.5 GiB / TiB |
-
-!!! info
- For disaggregated setups, it is recommended to add 50% to these numbers as a reserve. In
- a purely hyper-converged setup, stay at the requirement.
+| Per subsystem (cluster average per node) | 35 MiB |
+| Per TB of storage capacity on the host | 0.5 GiB / TiB |
## Control Plane Requirements
The simplyblock control plane has different hardware requirements depending on the deployment model.
-=== "Kubernetes"
+For the control plane, the minimum requirements per replica are 4 vcpu and 8 GiB of
+RAM as well as about 25 GiB of disk space on each of three 3 nodes.
+In Kubernetes, nodes can be workers or kubernetes control plane (OC) nodes. In Non-Kubernetes deployments,
+nodes are usually VMs. A minimum of three nodes is required for an HA setup.
- For a Kubernetes-based control plane, the minimum requirements per replica are:
+The disk space also accounts for the State Database. In addition, an S3 bucket of at least 50 GB to store
+State Database Backups is highly recommended.
- | Service | Instances | vCPU(s) | RAM (GB) | Disk (GB) |
- |------------------------------|-----------|---------|-----------|-----------|
- | Simplyblock Operator | 1 | 1 | 0.5 | 0.5 |
- | Control Plane API | 3 | 0.5 | 1 | 0.5 |
- | Meta-Database (FoundationDB) | 3 | 1 | 1 | 5 |
- | Task Runners | 11 | 0.25 | 0.1 | 0.5 |
- | CSI Driver Services | 1 | 0.5 | 0.2 | 0.5 |
- | Admin Pods | 1 | 0.25 | 0.25 | 0.5 |
- | Prometheus | 1 | 1 | 3 | 10 |
- | **Total across 3 nodes** | **3** | **10** | **10.55** | **33.5** |
-
-
- !!! important
- 3 replicas across 3 Kubernetes workers are mandatory for the Key-Value-Store. The WebAPI runs as
+!!! important
+ 3 replicas across 3 nodes are mandatory for the Key-Value-Store. The WebAPI runs as
a Daemonset on all Workers, if no taint is applied. The Observability Stack can optionally be
replicated and the sb-services run without replication.
- Additionally, a non-production observability stack can be deployed:
-
- | Service | Instances | vCPU(s) | RAM (GB) | Disk (GB) |
- |------------|-----------|---------|----------|-----------|
- | Grafana | 1 | 1 | 1 | 25 |
- | Graylog | 1 | 2 | 3 | 25 |
- | OpenSearch | 1 | 3 | 12 | 25 |
- | MongoDB | 1 | 1 | 1 | 25 |
- | Thanos | 3 | 0.25 | 2 | 25 |
-
-=== "Plain Linux"
-
- A control plane cluster of this size can manage up to 5 nodes, 1,000 logical volumes, and 2,500 snapshots. For
- larger deployments, increase the resources of the management nodes accordingly.
-
- | Node Type | vCPU(s) | RAM (GB) | Locally Attached Storage | Network Performance | Free Boot Disk | Number of Nodes |
- |---------------|---------|----------|--------------------------|---------------------|----------------|-----------------|
- | Control Plane | 4 | 16 DDR4 | - | 1 GBit/s | 35 GB | 3 |
-
-
-### Control Plane Scaling Triggers
-
-The general system requirements represent a minimal system setup with support for a limited amount of storage nodes,
-logical volumes, and log retention.
+A control plane cluster of the default size as provided below can manage up to 3 nodes and
+18,000 objects (with up to 50% of objects being volumes). For larger deployments,
+increase the resources of the management nodes accordingly. Add 1 vcpu and 2 GB of RAM as well as
+5 GB of disk space and 5 GB of backup space per managed storage node above 3.
-=== "Kubernetes"
-
- The control plane sizing is based on the minimal setup of the Simplyblock Operator. It is designed to support a
- service size of 2,000 logical volumes and 3 storage nodes. Furthermore, the assumed log storage retention is 3 days.
-
- For larger deployments, use the following tables to adjust the system requirements. The first table shows additional
- resources per 2,500 logical volumes.
-
- The second table shows additional required resources per 10 storage nodes.
-
- Additional Resources per 1,000 Logical Volumes
-
- | Service | add. vCPU | add. GB (RAM) | add. GB (Disk) |
- |------------------------------|-----------|---------------|----------------|
- | Simplyblock Operator | 0.25 | 0.5 | - |
- | Control Plane API | 0.25 | 0.5 | - |
- | Meta-Database (FoundationDB) | 0.5 | 0.25 | 2 |
- | Task Runners | 0.25 | 0.25 | - |
- | CSI Driver Services | 0.5 | 0.2 | - |
- | Admin Pods | 0.2 | 0.1 | - |
- | Prometheus | 0.5 | 0.25 | 10 |
- | **Total per node** | **2.5** | **2.5** | **12** |
-
-
-
- Additional Resources per 10 Storage Nodes
-
- | Service | add. vCPU | add. GB (RAM) | add. GB (Disk) |
- |------------------------------|-----------|---------------|----------------|
- | Simplyblock Operator | 0.5 | 0.5 | - |
- | Control Plane API | 0.1 | - | - |
- | Meta-Database (FoundationDB) | 0.5 | 0.25 | 2 |
- | Task Runners | 0.25 | 0.25 | - |
- | CSI Driver Services | - | - | - |
- | Admin Pods | 0.1 | 0.1 | - |
- | Prometheus | 0.25 | 0.25 | 10 |
- | **Total per node** | **2.35** | **1.35** | **12** |
-
-
-
-=== "Plain Linux"
-
- If more than 2,500 volumes or more than 5 storage nodes are attached to the control plane, additional RAM and vCPU
- are advised. Also, the required observability disk space must be increased, if retention of logs and statistics for
- more than 7 days is required.
+Additionally, a non-production observability stack can optionally be deployed.
+It can be distributed across OC or worker nodes with only one service (Thanos) being replicated.
+In total it requires at least 8 vCPU and 20 GB of RAM as well as 125 GB of disk space.
+The requirements for disk space will significantly increase with a custom retention period (>3 days)
+and the number of nodes (>3).
## CPU & Platform Compatibility
@@ -228,12 +154,18 @@ but performance per TiB is lower and rebalancing can take longer.
Clusters are lightweight, and it is recommended to use different clusters for different types of
hardware (NVMe, networking, compute) or with a different performance profile per TiB of raw storage.
+!!! info
+ From 26.3, Simplyblock also supports non-nvme devices (SATA/SAS linux block devices). This
+ feature is still experimental.
+
### NVMe Uniformity Recommendations
In general, all NVMe used in a single cluster should exhibit a similar performance profile per TB.
Therefore, within a single cluster, all NVMe devices are recommended to be of the same size,
but this is not a hard requirement.
+The same requirement applies to non-nvme devices, if non-nvme clusters are used.
+
### NVMe Exclusivity Requirements
Simplyblock only works with non-partitioned, exclusive NVMe devices (virtual via SRV-IO or physical) as its backing
@@ -248,6 +180,10 @@ Individual NVMe namespaces or partitions cannot be claimed by simplyblock, only
Additionally, devices will be detached from the operating system's control and will no longer show up in _lsblk_
once simplyblock's storage nodes are running.
+!!! info
+ For non-nvme clusters, block devices will remain attached under Linux. They have to be unmounted and
+ unpartitioned to be used. Partitioned devices can also be force-formatted at deployment time.
+
### NVMe Formatting Prerequisites
Simplyblock can low-level format NVMe devices with 4KB block size before deploying simplyblock. This is an optional
@@ -275,9 +211,9 @@ bond over two ports of the NIC(s) or using SRV-IO must be created.
Simplyblock implements NVMe over Fabrics (NVMe-oF), either NVMe over TCP or NVMe over RoCEv2, and works over any Ethernet
interconnect. The fabric transport layers can be mixed, like cluster internal-traffic on NVMe over RoCEv2 and client to cluster over NVMe over TCP.
-!!! recommendation
- Simplyblock highly recommends NICs with RDMA/ROCEv2 support such as NVIDIA Mellanox network adapters (ConnectX-6 or higher).
- Those network adapters are available from brands such as NVIDIA, Intel, and Broadcom.
+!!! info
+ NICs with RDMA/ROCEv2 support such as NVIDIA Mellanox network adapters (ConnectX-6 or higher) can be used to deploy ROCEv2 fabrics over standard Ethernet infrastructure.
+ The latency and tail-latency of ROCEv2 fabrics is usually significantly lower than in TCP.
### Management Traffic Network Requirements
@@ -285,8 +221,8 @@ It is recommended to use a separate physical NIC with two ports (bonded) and a h
management traffic. For management traffic, a 1 GBit/s network is sufficient and a Linux Bridge may be used.
!!! important "Highly Available Control Plane"
- When simplyblock is deployed with an HA control plane, an external load balancer is required to distribute
- requests of the storage plane to active control plane nodes. This is required to ensure that the control plane
+ When simplyblock is deployed with an HA control plane in non-Kubernetes environments, an external load balancer is required to distribute
+ requests of users or storage drivers to active control plane nodes. This is required to ensure that the control plane
is not a single point of failure when one or more management nodes are down.
For Simplyblock Operator-based deployments, the load balancer is not required, as it is already implemented as
@@ -295,11 +231,7 @@ management traffic. For management traffic, a 1 GBit/s network is sufficient and
### Layer 2 Constraints and Prohibited Topologies
!!! warning
- All storage nodes within a cluster and all hosts accessing storage shall reside within the same hardware VLAN.
-
- Any gateways, firewalls, or proxies higher than L2 on the network path must be avoided. Any of those solutions
- will heavily (and unpredictably) impact performance and latency.
-
+ Any gateways, firewalls, or proxies higher than L2 on the network path should be avoided for performance reasons.
## Additional Hardware Guidance
### PCIe Version