'NVMe storage' that's clearly cached HDD

pablowild

Member OP

Joined:: Jul 2024
Posts:: 137
From:: Buenos Aires, Argentina

1y ago

#1

¡¿Is very frustrating?! Contabo advertises "blazing NVMe storage" on their €4/mo plan. I am run fio and results smell wrong.

# fio --name=randread --ioengine=libaio --iodepth=32 \
> --rw=randread --bs=4k --direct=1 --size=4G --numjobs=4

read: IOPS=847, BW=3388KiB/s (3470kB/s)
     lat (usec): min=189, max=124832, avg=150832.23

¡¿847 IOPS?! My USB stick from 2014 does better. But then sequential write hits 400MB/s for 30 seconds, then drops to 80MB/s. Classic cache behavior, no? I am think SSD frontend + HDD backend, or very oversubscribed NVMe.

I delete long paragraph about consumer protection law. Jajaja. Better ask community first.

Provider is OVHcloud, not Contabo — Contabo I use for CDN only. Apologies for confusion in first draft.

SingaporeRep

Member

Benchmark Addict

Joined:: Jul 2024
Posts:: 227
From:: Singapore, SG

1y ago

#2

Jaja same thing on Vultr last year ok sale

My fio was showing 2000 iops then 200 iops after 10 sec no punctuation just pain ok

Their ticket say "shared storage" jaja shared with how many tenants tho sale

fio, iperf, geekbench. results or gtfo.

SiteRita

Member

data hoarder in recovery

Joined:: Sep 2024
Posts:: 9
From:: Lagos, Nigeria

1y ago

#3

I am coming with my test script now

pablowild said:

Sequential write hits 400MB/s for 30 seconds

Is it? That burst window is go go check — look at your blkio.throttle.write_bps_device inside container, is it? I am see this exact pattern when provider is throttle container not disk

Check:

cat /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device

If you see number there, you are test container policy, not physical disk at all

sync, encrypt, forget, restore

pam_blik

Member

adblock evangelist

Joined:: Dec 2024
Posts:: 25
From:: Austin, USA

1y ago

#4

Rita's onto something here

SiteRita said:

You are test container policy, not physical disk at all

I had this EXACT drama with HostHatch last year — https://hosthatch.com — Their "NVMe" was NVMe somewhere in the building, sure, but my container got HDD-tier blkio limits

Run this for the SMOKING GUN:

lsblk -d -o NAME,ROTA,TYPE,SIZE,MODEL

If ROTA=1, that's spinning rust no matter what marketing says

Also check if you're in LXC vs KVM — LXC makes this deception EASIER

ellagee91

Member

Joined:: Sep 2024
Posts:: 34
From:: Melbourne, Australia

1y ago

#5

pablowild said:

Provider is OVHcloud

Oh no. Not OVHcloud again. I tested them in 2024. Same exact pattern. 30 seconds of glory then the cliff. I thought it was ZFS arc hitting limit. But no. It was the container throttle all along. I should have checked cgroup sooner. Especially since their TOS says "fair share" which means nothing and everything. You get what they decide. Not what you expect.

My list of things to verify now.

lsblk for ROTA flag.
Cgroup blkio limits.
Ls -la /sys/block for scheduler hints.
fio with --direct=1 AND --buffered=0 just to be sure.

.never trust the dashboard...

JA

jackhrt

Member

Joined:: Sep 2024
Posts:: 2

1y ago

#6

What fio block size and runtime did you use

Doug

Member

New Jersey

Joined:: Jul 2024
Posts:: 271
From:: New Jersey, US

1y ago

#7

$4/mo "NVMe" jajaja

grabs popcorn, checks /r/drama

Hana5

Member

GPU poor

Joined:: Jun 2024
Posts:: 104
From:: Tokyo, Japan

1y ago

#8

I had the same cliff on OVHcloud, 30s then dead

8-char NTLM found in 4min 32sec

pablowild

Member OP

Joined:: Jul 2024
Posts:: 137
From:: Buenos Aires, Argentina

1y ago

#9

SiteRita said:

Look at your blkio.throttle.write_bps_device

¡Ah! I am check now and YES, there is number there! 419430400 — that is exactly 400MB/s, no? So the 400MB/s is not disk, is artificial limit. Then when cache is fill, true speed is show.

pam_blik said:

If ROTA=1, that's spinning rust

ROTA=0 on all devices, but this is LXC container so lsblk is lie to me probably. The host can present anything.

I am run inside /sys/block and check scheduler — all "none" which is typical for NVMe, but again, this is container, not sure I trust.

The real question now: I think their ToS has something about "fair share" but I am look and look, no define the actual number. 400MB/s burst, then 80MB/s? 80MB/s is HDD speed, no? So maybe is SSD cache + HDD, or maybe is just many tenant on same NVMe.

I am open ticket with OVHcloud but expect "working as designed" response. Jajaja.

ellagee91

Member

Joined:: Sep 2024
Posts:: 34
From:: Melbourne, Australia

1y ago

#10

pablowild said:

All "none" which is typical for NVMe

Careful — "none" scheduler is also what you see for dm-crypt, LVM, and plenty of virtual block devices. It proves almost nothing inside a container. The scheduler is inherited from whatever the host stacked on top of.

I ran the same test on my OVHcloud VPS just now. Sydney datacenter, €6 plan.

cat /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device
8:0 524288000

500MB/s burst. Same cliff after ~25 seconds. Then ~60MB/s sustained.

So it's definitely their platform policy, not your specific node misbehaving. This is how they all work.

The "NVMe" part is technically true — the hardware underneath is probably NVMe. What they don't advertise is the per-container throttle that makes it perform like cached HDD for sustained workloads. Classic cloud marketing.

Post a reply