[转帖]MinIO系列7 - Minio性能压测

您淘淘网 2024-01-11 01:07:35 ©著作权

文章标签 sed 文章分类 Python 热度指数 0

©著作权归作者所有：来自知识热度网作者您淘淘网的原创作品，请联系作者获取转载授权，否则将追究法律责任

前言

声明：此文为本人历史笔记的整理，文章实际撰写时间为2021年2月份，所以本中所使用的相关组件版本较老。此文是通过压力测试以理解MinIO在当前硬件条件下能提供的性能，以及相关的资源消耗情况。转载请注明出处，谢谢!

前置条件：已有Kubernetes集群，并安装好了MinIO集群。

相关组件版本：

组件名	版本	说明
Kubernetes	1.19	运行环境
minio	RELEASE.2020-06-14T18-32-17Z
iozone	3-490	测试磁盘的吞吐量
s3-benchmark		测试MinIO的上传下载性能

Part 1, 测试 8Nodes/32Disks

1 集群介绍

8个节点，每节点4*1TiB的磁盘。磁盘底层为VMware的全SSD vSAN分布式存储。
minio集群的节点参数设置如下

# Number of drives attached to a node
drivesPerNode: 4
# Number of MinIO containers running
replicas: 8
# Number of expanded MinIO clusters
zones: 1

pod的资源没做限制，内存request为4G
Storage class为默认

mc admin config get  minio-test storage_class
storage_class standard= rrs=EC:2 dma=write

2 MinIO磁盘性能评估

2.1 测试单盘性能

参照官方的性能测试报告，对于单盘的性能使用dd测试
在k8s的node节点上找到挂载到minio的磁盘，如下面命令所示，minio使用的磁盘为/dev/sd{c.d.e.f}

kubectl get po -n minio -o wide | grep $(hostname)
minio-4                   1/1     Running   0          31m    10.233.102.21   prod-ops-hy-k8s-27-1-31   <none>           <none>

kubectl exec -it minio-4 -n minio -- df -Th|grep export
/dev/sde             ext4         1006.9G    214.4M    955.5G   0% /export-2
/dev/sdd             ext4         1006.9G    214.4M    955.5G   0% /export-3
/dev/sdc             ext4         1006.9G      2.1G    953.6G   0% /export-0
/dev/sdf             ext4         1006.9G     79.3M    955.6G   0% /export-1

查看本地/dev/sdc的挂载路径

df -Th|grep sdc
/dev/sdc                        ext4     1007G  2.1G  954G   1% /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7021959-4a18-42e6-b4c6-10eab427e5cc/globalmount

进入上面查找到的路径，执行dd

## 以16M块连续写入2GB数据
# dd if=/dev/zero of=./test bs=16M count=128 oflag=direct
128+0 records in
128+0 records out
2147483648 bytes (2.1 GB) copied, 2.12772 s, 1.0 GB/s

## 以16M块连续读取2GB数据
#  dd of=/dev/null if=./test bs=16M count=128 iflag=direct
128+0 records in
128+0 records out
2147483648 bytes (2.1 GB) copied, 2.01436 s, 1.1 GB/s

上面结果为，单磁盘以16M块连续写吞吐量为1GB/s；以16M块连续读吞吐量为1.1GB/s。

2.2 测试JBOD性能

下载iozone，并安装

wget http://www.iozone.org/src/current/iozone-3-490.x86_64.rpm

rpm -ivh iozone-3-490.x86_64.rpm

ls -l /opt/iozone/bin/iozone
-rwxr-xr-x 1 root root 336464 Jul 14  2020 /opt/iozone/bin/iozone

在/mnt下面创建挂载目录，并将/dev/sd{c.d.e.f}重新挂载，以方便测试

mkdir /mnt/drive{1..4}
mount /dev/sdc /mnt/drive1
mount /dev/sdd /mnt/drive2
mount /dev/sde /mnt/drive3
mount /dev/sdf /mnt/drive4

df -Th | grep /mnt/drive
/dev/sdd                        ext4     1007G  215M  956G   1% /mnt/drive2
/dev/sdc                        ext4     1007G  2.1G  954G   1% /mnt/drive1
/dev/sde                        ext4     1007G  215M  956G   1% /mnt/drive3
/dev/sdf                        ext4     1007G   80M  956G   1% /mnt/drive4

按照官方性能测试报告的推荐，执行iozone，同样使用32 thread。由于不清楚官方文档中取的读写吞吐量值是Children还是Parent，当前测试出来的取Children的话，4个盘聚合的写吞吐量可达到1.2GB/s，读吞吐量为2.3GB/s。

/opt/iozone/bin/iozone -t 32 -I -r 32K -s 256M -F /mnt/drive{1..4}/tmp{1..8}
        Iozone: Performance Test of File I/O

...

        Children see throughput for 32 initial writers  = 1290740.23 kB/sec
        Parent sees throughput for 32 initial writers   = 1030806.54 kB/sec

...

        Children see throughput for 32 readers          = 2365000.80 kB/sec
        Parent sees throughput for 32 readers           = 2362278.36 kB/sec

4. 这个与dd测试的单盘1.1GB/s没太大的差距，由于上面iozone使用的是32K块，可能存在iops的性能达到了极限，而吞吐量上不去的情况。所以改成16M的块再测试一遍，结果如下所示，写吞吐量达到了2.76GB/s，读吞吐量达到了2.74GB/s。

/opt/iozone/bin/iozone -t 32 -I -r 16M -s 256M -F /mnt/drive{1..4}/tmp{1..8}
        Iozone: Performance Test of File I/O

...

        Children see throughput for 32 initial writers  = 2895796.42 kB/sec
        Parent sees throughput for 32 initial writers   = 2450370.20 kB/sec
        Min throughput per process                      =   79270.73 kB/sec
        Max throughput per process                      =  115852.30 kB/sec
        Avg throughput per process                      =   90493.64 kB/sec
        Min xfer                                        =  180224.00 kB

...

        Children see throughput for 32 readers          = 2874974.30 kB/sec
        Parent sees throughput for 32 readers           = 2793761.68 kB/sec
        Min throughput per process                      =   81235.27 kB/sec
        Max throughput per process                      =  100890.62 kB/sec
        Avg throughput per process                      =   89842.95 kB/sec
        Min xfer                                        =  212992.00 kB

3 MinIO吞吐量Benchmark

3.1 准备客户机

从github上克隆wasabi的s3-benchmark到ansible的主控主机上

git clone https://github.com/wasabi-tech/s3-benchmark.git
Cloning into 's3-benchmark'...
remote: Enumerating objects: 40, done.
remote: Total 40 (delta 0), reused 0 (delta 0), pack-reused 40
Unpacking objects: 100% (40/40), done.

这们使用一个k8s测试集群的8个主机节点来做MinIO benchmark的客户机，在ansibie host中这些主机归属于ops-hy-k8s组

at /etc/ansible/hosts
[ops-hy-k8s]
prod-ops-hy-k8s-27-1-11
prod-ops-hy-k8s-27-1-12
prod-ops-hy-k8s-27-1-13
prod-ops-hy-k8s-27-1-14
prod-ops-hy-k8s-27-1-15
prod-ops-hy-k8s-27-1-16
prod-ops-hy-k8s-27-1-17
prod-ops-hy-k8s-27-1-18

执行如下ansible命令将s3-benchmark复制到客户机

cd s3-benchmark
ansible ops-hy-k8s -m copy -a "src=/tmp/s3-benchmark/s3-benchmark dest=/usr/local/bin/s3-benchmark mode=700"

4. 测试下是否可以批量执行

ansible ops-hy-k8s -m shell -a "/usr/local/bin/s3-benchmark -h"
prod-ops-hy-k8s-27-1-14 | FAILED | rc=2 >>
Wasabi benchmark program v2.0Usage of myflag:
  -a string
        Access key
  -b string
        Bucket for testing (default "wasabi-benchmark-bucket")
  -d int
        Duration of each test in seconds (default 60)
  -l int
        Number of times to repeat test (default 1)
  -r string
        Region for testing (default "us-east-1")
  -s string
        Secret key
  -t int
        Number of threads to run (default 1)
  -u string
        URL for host with method prefix (default "http://s3.wasabisys.com")
  -z string
        Size of objects in bytes with postfix K, M, and G (default "1M")non-zero return code

...

3.2 启动benchmark

先生成一个脚本，用来启动s3-benchmark，并复制到每个客户机。-z 2G意思是使用2G的对象进行上传下载测试，-t 32意思是客户端开启32进程

echo "/usr/local/bin/s3-benchmark -a minio -s minio123 -u https://minio-test.xxx.com -z 2G -t 32 -b s3bench-\$HOSTNAME -d 1" > /tmp/s3-benchmark.sh
ansible ops-hy-k8s -m copy -a "src=/tmp/s3-benchmark.sh dest=/root/s3-benchmark.sh"

启动s3-benchmark，请注意这里一定要带"-f 8"，不然ansible只会同样启动5个客户机上的脚本

ansible ops-hy-k8s -f 8 -m shell -a "/root/s3-benchmark.sh"

到任意客户机上查看s3-benchmark是否在运行

ps -ef|grep s3
root      4843  4815  0 22:22 pts/4    00:00:00 /bin/sh -c /root/s3-benchmark.sh
root      4844  4843 85 22:22 pts/4    00:00:02 /usr/local/bin/s3-benchmark -a minio -s minio123 -u https://minio-test.xxx.com -z 2G -t 32 -b s3bench-prod-ops-hy-k8s-27-1-11 -d 1

运行完成后返回结果，有两个客户机因为tcp连接重置而退出了

prod-ops-hy-k8s-27-1-13 | FAILED | rc=1 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-13, region=us-east-1, duration=1, threads=32, loops=1, size=2G2021/02/08 23:12:14 FATAL: Error uploading object https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-13/Object-27: Put https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-13/Object-27: net/http: HTTP/1.x transport connection broken: write tcp 10.27.1.13:40542->10.27.1.31:443: write: connection reset by peernon-zero return code

prod-ops-hy-k8s-27-1-17 | FAILED | rc=1 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-17, region=us-east-1, duration=1, threads=32, loops=1, size=2G2021/02/08 23:12:17 FATAL: Error uploading object https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-17/Object-15: Put https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-17/Object-15: net/http: HTTP/1.x transport connection broken: write tcp 10.27.1.17:35888->10.27.1.31:443: write: connection reset by peernon-zero return code

prod-ops-hy-k8s-27-1-18 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-18, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 231.4 secs, objects = 32, speed = 283.3MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 83.1 secs, objects = 32, speed = 789MB/sec, 0.4 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.2 secs, 159.8 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-12 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-12, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 366.9 secs, objects = 32, speed = 178.6MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 115.5 secs, objects = 32, speed = 567.4MB/sec, 0.3 operations/sec. Slowdowns = 0
Loop 1: DELETE time 1.2 secs, 25.9 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-11 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-11, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 429.8 secs, objects = 32, speed = 152.5MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 165.5 secs, objects = 32, speed = 396MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.2 secs, 137.0 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-16 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-16, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 450.1 secs, objects = 32, speed = 145.6MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 145.6 secs, objects = 32, speed = 450.1MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.2 secs, 160.2 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-14 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-14, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 442.7 secs, objects = 32, speed = 148MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 153.9 secs, objects = 32, speed = 425.7MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.3 secs, 110.2 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-15 | CHANGED | rc=

本文包含： sed