Flings
Apps and tools built by our engineers that are intended to be played with and explored.

VMware Communities Roundtable Podcast

Hear VMware's own Dana Nourie and Andre Leibovici discuss the Flings process and the latest from VMware Labs!

Download episode

IOBlazer

IOBlazer

Version: 1.01 • File Size: 147 KB • 1/10/2011 • MIT License
View Terms & Download

Summary

IOBlazer is a multi-platform storage stack micro-benchmark. IOBlazer runs on Linux, Windows and OSX and it is capable of generating a highly customizable workload. Parameters like IO size and pattern, burstiness (number of outstanding IOs), burst interarrival time, read vs. write mix, buffered vs. direct IO, etc., can be configured independently. IOBlazer is also capable of playing back VSCSI traces captured using vscsiStats. The performance metrics reported are throughput (in terms of both IOPS and bytes/s) and IO latency.

IOBlazer evolved from a minimalist MS SQL Server emulator which focused solely on the IO component of said workload. The original tool had limited capabilities as it was able to generate a very specific workload based on the MS SQL Server IO model (Asynchronous, Un-buffered, Gather/Scatter). IOBlazer has now a far more generic IO model, but two limitations still remain:

  1. The alignment of memory accesses on 4 KB boundaries (i.e., a memory page)
  2. The alignment of disk accesses on 512 B boundaries (i.e., a disk sector).

Both limitations are required by the gather/scatter and un-buffered IO models.

A very useful new feature is the capability to playback VSCSI traces captured on VMware ESX through the vscsiStats utility. This allows IOBlazer to generate a synthetic workload absolutely identical to the disk activity of a Virtual Machine, ensuring 100% experiment repeatability.

System Requirements

Any 32-bit Windows or Linux system or any 64-bit Windows, Linux, or OSX system should be able to run IOBlazer. On Linux systems, libaio is required.

Instructions

Simply copy and run the appropriate binary for your OS. Alternately, IOBlazer can be built using the included source code. Further details can be found in the provided README file.

Video

Change Log

Updates in IOBlazer 1.01:

  • Added configurable IO alignment
  • Increased the robustness of the trace file parser in the face of spurious lines
  • Increased the robustness of the build process by automatically detecting target OS and arch within the Makefile
  • In the Windows version, changed the raw access mode from volume to physical drive to avoid unnecessary mount/unmount operations at every test run.

Engineers

Davide Bergamasco

Works in the Performance group.

Add Comment

61 thoughts on “IOBlazer”

  2. Chad Morgenstern

    Thanks Davide,

    It turns out the problem was with the alignment on the host from which I replayed the traces. The starting offset of the partition was on the 63 sector rather than 64th. An oversite I have corrected.

    This replay feature is very exciting for us, it allows us to create a library of workloads to reuse at later times, as well as a simple way to quickly scale up and out.

    Well Done!!!

    Reply ↓
  3. chad morgenstern

    I’m interested in using ioblazer in replay mode. I’m concerned about page size, so I set Disk IO alignment to 4096 using -A.

    Running ioblazer in playback mode, I get this:
    Warning: Trace Playback mode enabled. IO Spec parameters will be ignored.

    Though I see this:
    Test parameters: 22200
    Test duration = 180 s
    Worker threads = 1
    File/Device path(s) = /iob1,
    File/Device size = 1 MB
    Buffer size = 1 MB
    IO Alignment = 4096 B <—-
    IO access pattern = r
    IO size (Avg) = 8192 B

    I see from my storage array statistics that the I/O is not aligned.

    Is the -A ignored by ioblazer when doing replay mode? If we modified the code to not ignore the command line, would the I/O align as desired?

    Chad

    Reply ↓
    1. Davide Bergamasco

      Chad,

      When using IOBlazer in reply mode, the command line parameters that define the IO pattern are ignored because the IO process is entirely driven by the trace. If the trace contains unaligned IOs, IOBlazer will reply them verbatim on the target disk.

      Davide

      Reply ↓
  4. sfaris

    Hi, I tried running ioblazer on Mac OSX Mountain Lion, and it immediately errors out.

    In preparation, I created a file “ioblazer.db” in the /tmp directory of size 100MB and then used the run command:

    ./ioblazer-osx-x64 -a r -t 10 -r 1 -o 1 -d /tmp -f 100

    This produces the error message:

    ERROR: partial IO# 0: expected 8192 bytes, transferred 0 bytes

    Can you point me in the right direction to get it working on Mac OSX Mountain Lion? Thanks!

    Reply ↓
    1. Davide Bergamasco

      Hi Sfaris,

      Have you tried with the “-F” option? When this option is specified, the test file is created and filled before the experiment starts. Please let me know if that solved your problem.

      Thanks,

      Davide

      Reply ↓
  5. Rory

    Davide,

    First thank you for publishing this tool including source code!

    For our internal testing purposes we have been using your tool and have fixed a couple bugs in it as well as adding several enhancements to it. We would like very much to push these bug fixes and enhancements back to you; what is the correct mechanism to do so?

    -Rory

    Reply ↓
    1. Davide Bergamasco

      Hi Rory,

      Thank you for your kind words!! And thank you also for fixing bugs and contributing to improving IOBlazer. At this time there is no “formal” channel to contribute code back as no one has done that so far. I’d say that the easiest way to proceed is for you to send me a patch which I’ll apply to the code base before releasing the next version (I have a couple of fixes and improvements in the pipeline as well). I’ll make sure to mention your contribution in the change log.

      Thanks,

      Davide

      Reply ↓
  6. Joe Fagnani

    Davide,

    We’ve recorded a number of vscsiStats workloads and when we replay them with IOBlazer, the replay is 20% to 30% longer that the record time regardless of how performant the Storage is, including a local hard disk. The ioblazer.db is fully inflated and the replays tested have been on 32-bit Windows XP and 64-bit Windows Server 2003.

    We were expecting (and hoping) that replay time (and resultant rate metrics) would correspond with Storage performance. Can you shed any light as to what may be going on and what we might do to use vscsiStats replays to rate Storage in a repeatable and defined way?

    Thanks,

    Reply ↓
    1. Davide Bergamasco

      Joe,

      This problem may be due to the resolution of the Windows timer, which is around 15 ms. If two consecutive IOs are closer than 15 ms in the trace, they will be played back separated by at least 15 ms. If the trace has many of these “close” IOs, it will eventually play back slower than the original.

      You might want to try to play back the trace using a a Linux VM with the kernel recompiled with the HZ parameter (which determines the timer tick) set to 1000. This will result in a timer resolution of 1 ms.

      Cheers,

      Davide

      Reply ↓
      1. Joe Fagnani

        Davide,

        Thanks. I was afraid the answer was going to be something like that given the similar post below using an older Linux kernel below. According to a Microsoft WP, Windows 7 is also 15.6 ms. Just a thought…Microsoft allows Application Developers to change the default timer resolution for their App (mostly multimedia). Some Windows 7 documentation is found here:
        http://download.microsoft.com/download/3/0/2/3027D574-C433-412A-A8B6-5E0A75D5B237/Timer-Resolution.docx

        Any chance you’ve given a thought to modifying the IOBlazer code to set the Windows timer resolution to 1ms as well as compiling a 64-bit version?

        Thanks,

        Reply ↓
        1. Davide Bergamasco

          Joe,

          I have a few changes in the pipeline for IOBlazer and I’ll definitely consider adding to those the improvement of the Windows timer resolution, and the 64-bit binary.

          Cheers,

          Davide

          Reply ↓
          1. Joe Fagnani

            Davide,

            It turns out that Interrupt timers in our case weren’t responsible for longer replay than record times. It seems that the replay OS matters. The vscsiStats wokloads were from Windows 7 Systems which have a 1MB optimized disk I/O. Basically, IOBlazer blows out the Disk calls and each OS handles the Disk I/O based on it’s optimizations. I did vscsiStats traces of IOBlazer replays on several OSs and the partial histogram below is interesting:

            vscsiStats Trace of IOBlazer Replay
            Byte Size Replay File w2k8R2 w2k3-64 rhel6
            65536 2522 2522 2541 2497
            81920 317 317 3145 320
            131072 453 453 0 472
            262144 189 189 0 217
            524288 73 73 0 269
            1048576 46 46 0 0
            >1048576 71 71 0 0

            W2k8R2 replayed the vscsiStats workload as it was recorded, W2k3-64 bit couldn’t Read or Write anything larger than 80 KB and rhel6 defaulted to .5 MB.

            Thanks,

            Reply ↓
              1. Joe Fagnani

                Davide,

                No, I haven’t been replaying with buffered I/O. We want to achieve as close to a real world workload as possible on the replay. Would adding the -B switch to the command line help achieve that?

                Reply ↓
                1. Davide Bergamasco

                  Joe,

                  Not at all :) I actually asked that question because that would have easily explained the difference between the trace and the actual IOs. The buffer cache decouples completely the application IO pattern from the device IO pattern as the OS caches and consolidates adjacent IOs. The discrepancy you are seeing can still be explained by IO splitting and/or coalescing at the device driver level, though. I guess your best bet is to stick with W2K8R2.

                  Cheers,

                  Davide

                  Reply ↓
  7. Jeremy

    Hi,

    Thanks for this amazing tool for vSphere administration ! I have just one question : on one server (2008R2), I got an error (GetQueuedCompletionStatus failed las Error = 5) when I use IOBlazer for RAW access with a file size (-f) > 1MB.

    It works like a charm on all others servers. I can’t explain why..

    Thanks.

    Jeremy.

    Reply ↓
    1. Davide Bergamasco

      Hey Jeremy,

      Thanks for your kind words! System Error 5 is “ERROR_ACCESS_DENIED”, so it seems that the file you’re using to do your IO experiments does not have the correct access rights. It should be readable and writable by all users (or at least by the Administrator).

      Thanks,

      Davide

      Reply ↓
  8. Muthu

    Hi,
    I downloaded the ioblazer pkg and untarred. I got the following files:
    ioblazer.c IOBlazer.exe ioblazer-linux-x64 ioblazer-linux-x86 ioblazer-osx-x64 Makefile README

    Mine is a RHEL6.1 machine, I have libaio installed.. Here is that confirmation:
    #rpm -qa| grep aio
    libaio-0.3.107-10.el6.x86_64
    libsane-hpaio-3.10.9-3.el6.x86_64
    #
    #find / -name libaio*
    /usr/share/doc/libaio-0.3.107
    /lib64/libaio.so.1.0.1
    /lib64/libaio.so.1
    /lib64/libaio.so.1.0.0
    #

    When I do “make”, I get the following error:
    #cd ioblazer-1.01
    #make
    gcc -c -m64 -Wall -Werror -D__LINUX__ -D_LARGEFILE64_SOURCE ioblazer.c
    ioblazer.c:83:20: error: libaio.h: No such file or directory
    ioblazer.c:188: error: field âiocbâ has incomplete type
    ioblazer.c: In function âworkerThreadâ:
    ioblazer.c:1313: error: âio_context_tâ undeclared (first use in this function)
    ioblazer.c:1313: error: (Each undeclared identifier is reported only once
    ioblazer.c:1313: error: for each function it appears in.)
    ioblazer.c:1313: error: expected â;â before âctx_idâ
    ioblazer.c:1329: error: âctx_idâ undeclared (first use in this function)
    cc1: warnings being treated as errors
    ioblazer.c:1330: error: implicit declaration of function âio_queue_initâ
    ioblazer.c:1336: error: invalid application of âsizeofâ to incomplete type âstruct io_eventâ
    ioblazer.c:1444: error: implicit declaration of function âio_prep_preadâ
    ioblazer.c:1455: error: implicit declaration of function âio_prep_pwriteâ
    ioblazer.c:1473: error: implicit declaration of function âio_submitâ
    ioblazer.c:1495: error: implicit declaration of function âio_geteventsâ
    ioblazer.c:1515: error: invalid use of undefined type âstruct io_eventâ
    ioblazer.c:1515: error: dereferencing pointer to incomplete type
    make: *** [ioblazer.o] Error 1
    #

    Kindly help..

    Thanks and Regards,
    Muthu

    Reply ↓
    1. Davide Bergamasco

      Hi Muthu,

      Looks like you are missing the libaio.h header file which is part of libaio-devel-0.3.107-10.el6.x86_64 RPM. Once you install said RPM everything should work.

      Cheers,

      Davide

      Reply ↓
  9. Kenny

    This is really a useful tool, but I found it doesn’t work well to interpret trace file from ESX 5? Any plan to upgrade this tool?

    Reply ↓
  10. Richard

    Davide,

    I am trying to understand how the -f parameter plays in what I see as my MB/s result.

    I see the larger file size decrease my MB/s, and I would like to be clear why this is the case ?

    Would you have a recommended value for this, e.g. is default OK ?

    Thanks,
    Richard.

    Reply ↓
    1. Davide Bergamasco

      Richard,

      As the file size increases, so does the area of the physical disk(s) covered by the data. If the IO access pattern is random (default in IOBlazer) then the disk heads will have to travel farther to reach the data, resulting in larger IO latency increases and, consequently, in lower throughput.

      The default file size is very small in order to make the file fit entirely in the cache of the IO subsystem being used. This maximizes throughput, but it may or may not be realistic enough for your use case. I’d suggest to use a size as close as possible to the size of the working set of the application you’re are planning on running on your system.

      Hope this helps,

      Davide

      Reply ↓
  11. Mario Grunert

    Does it write zeros or some random data ?

    I could imagine that a zeroes load will vanish in the compression algorythm on Ontap 8 or the inline dedup in ZFS.

    Reply ↓
    1. Davide Bergamasco

      Mario,

      Each page of the memory buffer from which IOBlazer gathers the data to write to the disk/file is filled with the same random pattern, but it’s then made unique by writing the page number in the first word. Therefore, if the dedupe algorithm looks at something as large as (or larger than) a page (4 KB) then it won’t find any duplicated data. However, if it looks at something smaller, it may find data duplication depending on the alignment.

      Hope this helps,

      Davide

      Reply ↓
  12. Lance Braswell

    Hi Davide,

    Love the tool. It’s a big help. I apologize if this has already been covered elsewhere. When performing vscsiStats trace and replay via IOBlazer on RHEL 4.8 -> RHEL 5.5 and compiled from source, I noticed that the rate of IO replay was drastically slower that what was recorded. Context for this is ioblazer 1.01.

    The first hint at the source of the apparent slowdown on replay was on systems running with VMware Linux timekeeping best practices. I.e. divider=10 (so HZ=100). For any vscsiStats recording with IOps >> 100, the replay was always delivering IOps in the 100 range.

    Apparently until more recent versions of Linux kernel, nanosleep() resolution is no better than HZ so when you uSleep() -> nanosleep() on your replay, your IOs are effectively delivered no faster than HZ. I then ran with default HZ=1000 and got better results but since nanosleep sleeps for at least as long as the time requested (i.e. 1/HZ), it’s still not enough resolution to replay back with good fidelity if your IOs are recorded at a rate > HZ.

    Here is a demonstration:

    First few IOs

    Serial Number,IO Data Length,Num SG Entries,Command Type,LBN,Time Stamp (microseconds)
    2148401154,13824,4,write,7920,3890865606325
    2148401258,13824,2,read,954,3890865606335
    2148401231,13824,2,read,12388,3890865606343

    deltas in us respectively between io 1,2 and io 2,3:
    10,8

    At HZ=100 (2.6.9-89.0.11.ELlargesmp/RHEL 4.8/notsc divider=10):

    Here is an strace of the replay (via strace -f -ttt -o /tmp/out ./ioblazer -P /tmp/vmtst-ads-2-1.8329.RAW_100_1.2011_08_04_22_52_28.300s.txt -d /dev/datavg/lvol1 -R -f 100)

    28523 1312523281.074311 open(“/tmp/vmtst-ads-2-1.8329.RAW_100_1.2011_08_04_22_52_28.300s.txt”, O_RDONLY) = 4
    28523 1312523281.074329 fstat(4, {st_mode=S_IFREG|0644, st_size=21253282, …}) = 0
    28523 1312523281.074347 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a9555b000
    28523 1312523281.074358 read(4, “Vscsi Cmd Trace. (Trace Format V”…, 4096) = 4096
    28523 1312523281.074393 gettimeofday({1312523281, 74396}, NULL) = 0
    28523 1312523281.074405 io_submit(182894034944, 1, {…}) = 1
    28523 1312523281.074440 io_getevents(182894034944, 1, 1, {…}NULL) = 1
    28523 1312523281.082073 gettimeofday({1312523281, 82080}, NULL) = 0
    28523 1312523281.082105 nanosleep({0, 10000}, {222592816416, 0}) = 0
    28523 1312523281.086195 gettimeofday({1312523281, 86201}, NULL) = 0
    28523 1312523281.086219 io_submit(182894034944, 1, {…}) = 1
    28523 1312523281.086255 io_getevents(182894034944, 1, 1, {…}NULL) = 1
    28523 1312523281.086535 gettimeofday({1312523281, 86538}, NULL) = 0
    28523 1312523281.086551 nanosleep({0, 8000}, {222592816416, 0}) = 0
    28523 1312523281.096183 gettimeofday({1312523281, 96189}, NULL) = 0
    28523 1312523281.096207 io_submit(182894034944, 1, {…}) = 1

    We can see the two nanosleep requests for 10000 and 8000. But the actual time between the start of nanosleep and io_submit is:
    (086219 – 082105) = 4114 us between io 1,2 (but asked for 10 us)
    (096207 – 086551) = 9656 us between io 2,3 (but asked for 8 us)

    At HZ=1000 ((2.6.9-89.0.11.ELlargesmp/RHEL 4.8):

    6625 1312518511.432091 open(“/tmp/vmtst-ads-2-1.8329.RAW_100_1.2011_08_04_22_52_28.300s.txt”, O_RDONLY) = 4
    6625 1312518511.432113 fstat(4, {st_mode=S_IFREG|0644, st_size=21253282, …}) = 0
    6625 1312518511.432130 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a9555b000
    6625 1312518511.432169 read(4, “Vscsi Cmd Trace. (Trace Format V”…, 4096) = 4096
    6625 1312518511.432205 io_submit(182894034944, 1, {…}) = 1
    6625 1312518511.432257 io_getevents(182894034944, 1, 1, {…}NULL) = 1
    6625 1312518511.454044 nanosleep({0, 10000}, {222592816416, 0}) = 0
    6625 1312518511.455475 io_submit(182894034944, 1, {…}) = 1
    6625 1312518511.455526 io_getevents(182894034944, 1, 1, {…}NULL) = 1
    6625 1312518511.455955 nanosleep({0, 8000}, {222592816416, 0}) = 0
    6625 1312518511.457463 io_submit(182894034944, 1, {…}) = 1

    We can see the two nanosleep requests for 10000 and 8000. But the actual time between the start of nanosleep and io_submit is:

    (455475 – 454044) = 1431 us actual between io 1,2 (but asked for 10 us)
    (457463 – 455955) = 1508 us actual between io 2,3 (but asked for 8 us)

    I understand that more recent versions of mainline Linux kernel have better resolution on nanosleep but I’m not able to test with it at the moment. Just want to call this out to see if it’s the right finding.

    kind regards,
    Lance

    Reply ↓
    1. Lance Braswell

      Hi Davide,

      I sent you an email with a patch as a POC workaround to the above to further the discussion.

      Lance

      Reply ↓
      1. Davide Bergamasco

        Hi Lance,

        Thank you for pointing out this issue and your in-depth analysis. I believe you are absolutely right: According to many sources nanosleep sleeps for *at least* the requested time or a timer tick, whichever is the largest. Apparently high-resolution timers are available from 2.6.22 on, and this explains why it worked for me as I’m using Fedora 14 with a 2.6.35 kernel. I’ll take a look at your patch and test it out as soon as I can.

        Thanks!

        Davide

        Reply ↓
  13. Tomek Podlaski

    Hi Davide,

    first of all thanks for the tools finally there is an app which can compete with the infamous IOMeter and does integrate with VMware.
    I came across a problem when trying to do a vscsiStats trace using logchannellogger on the ESXi 4.0U2. My scenario is: catch a trace of a VDI VM using vscsiStats (e.g. user workstation) then replay it on 100+ clients using synchronized IOBlazer’s instances. As I did not see the logchannellogger on the ESXi i copied it from the test ESX. But got some errors:
    1) when doing, logchannellogger -l –> VSI_GetInstanceListAlloc failed: 0xbad0003/sbin #
    2) when trying, logchannellogger vscsi_cmd_trace_9394432_8219 vscsi_cmd_trace_9394432_8219.out & (with my IDs of course:) –> Error on VSIGet: Is a directory
    3) there is also slight difference in the vscsiStats output –> vscsi_cmd_trace_ID_ID

    Is it possible to make it run on the ESXi?

    Kind regards
    Tomek

    Reply ↓
    1. Davide Bergamasco

      Hi Tomek,

      Sorry for my late reply, but yesterday I was off.

      Thank you very much for your kind words. I would be so harsh on IOMeter, it’s a good tool with its own limitations, just like IOBlazer :)

      I suspect that there is some binary incompatibility between the ESX version of logchannellogger (which is basically a linux binary) and the ESXi version. I’m afraid you really have to use the ESXi logchannellogger version which is in the /sbin directory.

      Hope this helps…

      Davide

      Reply ↓
      1. Tomek Podlaski

        Davide,

        thank you for the reply. I tried with the logchannellogger from ESXi 4.1U1(348481) but got same results when running it on ESXi4.0(261974) – 1) from my prev post. So I belive it is not possible to gather SCSI trace that can be used in the IOBlazer when there is ESXi 4.0 under the hood. Correct me if I am wrong as I did not test all posibilites using different builds of ESXi. If somebody manged to make it work I would appreciate information.

        Kind regards
        Tomasz Podlaski

        Reply ↓
        1. Davide Bergamasco

          Tomek,

          I just built a fresh logchannellogger from the ESXi 4.0 source tree. If you give me your email address I’ll ship it to you. Hopefully this one should work.

          Thanks,

          Davide

          Reply ↓
                  1. Joe Fagnani

                    Davde,

                    I apologize so much but I didn’t receive it, or it wound up in junk mail and I unintentionally deleted it.

                    Also, I’ve been trying to find out what the vscsistats trace column “Num SG Entries” means, but it doesn’t seem to be documented anywhere.

                    Thanks,

                    Reply ↓
                    1. Davide Bergamasco

                      Hi Joe,

                      I just resent the file. Please let me know if you got it this time.

                      Regarding “Num SG Entries” it refers to the number of entries in the Scatter Gather list associated with an IO. This is some low level stuff handled by the HBA and its driver in order to “gather” the various pieces of data making up a write from separate memory locations, and, conversely, “scatter” the various pieces of data from a read to separate memory locations.

                      Hope this helps,

                      Davide

  14. Miroslav

    Hi Davide,

    Thanks for putting together a great tool!

    I’m trying to use IOBlazer in a Windows environment, and am having trouble expressing what I’m trying to do via the parameters. Perhaps somebody could point me in the right direction. What I’d like to do is create several worker threads to generate random I/O to the same file/disk. When I specify more than one worker, IOBlazer tries to generate load on multiple drives. For example:


    PS F:\> ioblazer -a r -A 65536 -b 1 -d J: -f 92160 -i 512 -o 32 -w 2 -t 60

    IOBlazer, the ultimate IO benchmark :D

    Test parameters:
    Test duration = 60 s
    Worker threads = 2
    File/Device path(s) = J:, K:,
    File/Device size = 92160 MB
    Buffer size = 1 MB
    IO Alignment = 65536 B
    IO access pattern = r
    IO size (Avg) = 512 B
    IO size pattern = f
    Burst size (OIOs, Avg) = 32
    Burst size pattern = f
    Inter-burst time (Avg) = 0 us
    Inter-burst time pattern = f
    Read ratio = 1.0
    Latency threshold = 5000 ms
    Buffered IO = 0
    Checksum = 0
    Raw Device Access = 0
    Fill file = 0
    Trace file =

    Running test for 60 seconds...
    Test complete.

    Test duration = 60.045 s
    Total IOs = 44640 Total Bytes = 22855680
    IOPS = 743 Byte/s = 380643 (0.4 MB/s)
    Average latency = 39114.247 us Max Latency = 166000 us
    Long IOs = 0

    Instead of aiming both workers at the J: drive, it’s putting one on the J: drive and another on the next sequential drive letter (K:). Is there a way to specify that both workers should pound the same drive? I’ve tried specifying it as “J: J:” and “J:,J:”, but that just errors out looking for “J: J:\ioblazer.db”. Is there some usage trick that I’m missing, is this a bug in how the parameters are parsed, or is this the intended behavior and I’ll need to spawn multiple instanced of IOBlazer in the background in order to get the behavior I’m looking for?

    Thanks and take care,
    Miroslav

    Reply ↓
    1. Davide Bergamasco

      Hi Miroslav,

      Thank you for your kind comments!

      One thread per test file/disk is IOBlazer’s intended behavior. The philosophy behind this design choice is to maximize the parallelism in the IOs hitting the backend storage. If multiple threads hit the same file/device, the IOs they originate may end up being serialized by the OS. If you want to increase the parallelism on the same file/device you can increase the number of outstanding IOs. However, you have to be careful not to exceed any command queue size on the IO path to avoid adding unwanted latency.

      Hope this helps!

      Davide

      Reply ↓
  15. Gregoire Laverdiere

    Last IO operation failed: Error=38
    EOF on READ. Please check file size

    Any idea anyone?

    Thanks

    Reply ↓
    1. Davide Bergamasco

      Gregoire,

      I’m assuming you’re playing back a vSCSI trace. In this case, this error may indicate that the test file you are playing back the trace on is smaller than the virtual disk you captured the trace from. Try to make the test file larger by using the “-f test_file_size_in_MB” option in conjunction withe the “-F” option to fill it up with random data before the test starts.

      Hope this helps!

      Davide

      Reply ↓
      1. Gregoire Laverdiere

        Thanks for the quick response.

        We are just trying to pump some load into the infrastructure. Also we are searching for a tool that would allow us to create comperative baseline in both the windows world and the linux world.

        We were able to pump 18000 iops with 220MB/s from a 2 vCpu win xp vm using IOmeter. Which looking at the nature of the VM seems pretty good.

        It seems that IOblazer would work on both plateform. I am trying to genereate a similar type of workload using IOblazer so I can run comparative tests.

        Reply ↓
        1. Davide Bergamasco

          Gregoire,

          Could you please send me the VM disk configuration and the command line you are using to run IOBlazer at davide at vmware dot com?

          Thanks,

          Davide

          Reply ↓
          1. Richard O Brien

            Hi,

            Was there ever a resolution to this, trying

            - ioblazer -t 30 -d e:\tmp

            on a W2003x64 VM with two drives.

            Error as follows,

            Last IO operation failed: Error=38
            EOF on READ. Please check file size.

            Any help appreciated.

            Thanks,
            Richard.

            Reply ↓
  17. Chris

    help… really like ioblazer and it’s invaluable at the moment but I am running into an issue when trying to replay a trace.

    Having followed your instructions attempting a playback i receive the error meesage – “Last IO operation failed: Error=112″

    Can anyone help?

    Reply ↓
    1. Davide Bergamasco

      Hi Chris,

      Sorry you’re running into a problem with IOBlazer. Could you please send me the trace at davide(at)vmware(dot)com?

      Thanks,

      Davide

      Reply ↓
  18. David Davis

    Hi,
    I am very interested in learning more about IOBlazer but am struggling to picture, in simple term, a couple of common use cases where a VMware Admin would use it.
    Could you give me a couple of scenarios, please?
    Also, in simple terms, can you explain the role of vscsiStats vs IOBlazer?
    Thank you for your contribution of this performance tool to the VMware community!
    David

    Reply ↓
    1. Davide Bergamasco

      Hi David,

      Thank you for your interest in IOBlazer.

      We initially developed IOBlazer as an emulator for a complex storage workload like MS SQL Server. That allowed us to do some in-depth performance studies with a (somewhat) realistic workload without having to install and run the actual app, which can be a pretty complex and time consuming task. Also, more often than not, using the real app does not allow for results comparability because of the inherent variability in the workload.

      Another important use case for us was to investigate IO performance differences between VMs running different guest OSes, i.e., Windows and Linux. Being IOBlazer a multi-platform tool, it allowed us to do a comparative performance analysis in a simple and repeatable way.

      Probably the most important use case (and what makes IOBlazer a VMware specific tool) is to replay IO traces captured from running VMs with vsciStats. IO performance issues can be “easily” debugged by replaying exactly the same workload over and over without running the actual app.

      Hope this helps!

      Davide

      Reply ↓
  19. chad morgenstern

    We would like to use logblazer to replay vscisStats. I’m not clear on how to capture the output of the vscsiStats –tracecmds command, and Im not having luck finding the answer to this on the web.

    Can you explain how to use: $ logchannellogger

    I see that the following is running:
    [root@s1d-1 ~]# /usr/sbin/logchannellogger -l
    vscsi_cmd_trace_7501_8209

    But how do I capture the data? Specifically, what is the binary trace file that is reffered to in the help command? Can you show me an example?

    Chad

    Reply ↓
    1. Davide Bergamasco

      Chad, thanks for your interest in IOBlazer. Here is an example showing how to capture a vscsiTrace (there is way more stuff than you asked for, but I thought it might be useful for others as well).

      First, figure out which VM and virtual disk you want to trace as follows:

      ~ # vscsiStats -l
      Virtual Machine worldGroupID: 9394432, Virtual Machine Display Name: dbergamasco-dev {
      Virtual SCSI Disk handleID: 8219
      }
      Virtual Machine worldGroupID: 9370301, Virtual Machine Display Name: VMware Data Recovery {
      Virtual SCSI Disk handleID: 8202
      Virtual SCSI Disk handleID: 8203
      }

      Suppose we are interested in the first VM only, so we start the trace with:

      ~ # vscsiStats -t -s -w 9394432 -i 8219
      vscsiStats: Starting Vscsi stats collection for worldGroup 9394432, handleID 8219
      vscsiStats: Starting Vscsi cmd tracing for worldGroup 9394432, handleID 8219
      vscsi_cmd_trace_9394432_8219
      Success.

      The last line mentions the name of the logChannel used by vscsiStats to dump the trace. So now start the logger on said cahnnel as follow:

      # logchannellogger vscsi_cmd_trace_9394432_8219 vscsi_cmd_trace_9394432_8219.out &

      Now the logger is collecting in background the vscsiStats binary trace into the vscsi_cmd_trace_9394432_8219.out file. Wait as long as you need to gather a trace of the workload you are interested in and then stop vscsiStats collection (logchannellogger will stop as well):

      ~ # vscsiStats -x
      vscsiStats: Stopping all Vscsi stats collection for worldGroup 9394432, handleID 8219
      Success.
      [1] + Done(1) logchannellogger vscsi_cmd_trace_9394432_8219 vscsi_cmd_trace_9394432_8219.out
      ~ #

      At this point you have the trace saved in binary format in vscsi_cmd_trace_9394432_8219.out. In order to convert it in text format and feed it to IOBlazer do the following:

      ~ # vscsiStats -e vscsi_cmd_trace_9394432_8219.out > vscsi_cmd_trace_9394432_8219.txt

      Unfortunately there is one more thing you need to do: edit the text file and remove the first line, the one with the names of the various fields of the trace. When I initially developed IOBlazer that line was not there. Since IOBlazer does not expect such line, it will terminate parsing the trace as soon as it reads it. I fixed this problem in a new version of the tool that should be available soon on this site.

      Hope this helps!

      Davide

      Reply ↓
    1. Davide Bergamasco

      J1mbo, thank you for your comments. I’ll make sure to add 4K aligned IOs, or better, a configurable alignment, to the next release.

      Reply ↓

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>