Flings
Apps and tools built by our engineers that are intended to be played with and explored.

IOBlazer

IOBlazer

Summary

IOBlazer is a multi-platform storage stack micro-benchmark. IOBlazer runs on Linux, Windows and OSX and it is capable of generating a highly customizable workload. Parameters like IO size and pattern, burstiness (number of outstanding IOs), burst interarrival time, read vs. write mix, buffered vs. direct IO, etc., can be configured independently. IOBlazer is also capable of playing back VSCSI traces captured using vscsiStats. The performance metrics reported are throughput (in terms of both IOPS and bytes/s) and IO latency.

IOBlazer evolved from a minimalist MS SQL Server emulator which focused solely on the IO component of said workload. The original tool had limited capabilities as it was able to generate a very specific workload based on the MS SQL Server IO model (Asynchronous, Un-buffered, Gather/Scatter). IOBlazer has now a far more generic IO model, but two limitations still remain:

  1. The alignment of memory accesses on 4 KB boundaries (i.e., a memory page)
  2. The alignment of disk accesses on 512 B boundaries (i.e., a disk sector).

Both limitations are required by the gather/scatter and un-buffered IO models.

A very useful new feature is the capability to playback VSCSI traces captured on VMware ESX through the vscsiStats utility. This allows IOBlazer to generate a synthetic workload absolutely identical to the disk activity of a Virtual Machine, ensuring 100% experiment repeatability.

System Requirements

Any 32-bit Windows or Linux system or any 64-bit Windows, Linux, or OSX system should be able to run IOBlazer. On Linux systems, libaio is required.

Instructions

Simply copy and run the appropriate binary for your OS. Alternately, IOBlazer can be built using the included source code. Further details can be found in the provided README file.

Video

Change Log

Updates in IOBlazer 1.01:

  • Added configurable IO alignment
  • Increased the robustness of the trace file parser in the face of spurious lines
  • Increased the robustness of the build process by automatically detecting target OS and arch within the Makefile
  • In the Windows version, changed the raw access mode from volume to physical drive to avoid unnecessary mount/unmount operations at every test run.

Engineers

Davide Bergamasco

Performance
Add Comment

82 thoughts on “IOBlazer

  1. Mike Federwisch

    I had heard that having an adjustable data duplication rate was on your feature road map and was wondering how far away this was. I have looked at changing it myself such that there was a page for each IO size and making it so that at a argument specified rate, it would send the same data or make a change to the data so it would be unique. If you have something like this coming soon, I’ll not muck with the code. What are your thoughts on this feature?

    Reply
    1. Davide Bergamasco

      Mike,

      Such a feature is getting more and more important given that all modern arrays implement some form of data dedup. However, in the last few months it’s been very hard for me to find the time to work on side projects like IOBlazer, so I cannot really give you an ETA on a new version of IOblazer which includes an adjustable data duplication feature. If you want to go ahead and implement it, I’ll be glad to add your contribution back to IOblazer when I finally find the time to update it.

      Thanks!

      Davide

      Reply
  2. David Hocking

    I can’t get the trace playback function to work – it seems to understand what is being asked, but the results are always the same – test runs for 1 sec, with 0 total IOs, 0 Bytes.

    IOBlazer.exe -P .\shcp04-1.trc

    IOBlazer, the ultimate IO benchmark :D

    Warning: Trace Playback mode enabled. IO Spec parameters will be ignored.
    Test parameters:
    Test duration = 180 s
    Worker threads = 1
    File/Device path(s) = d:\tmp,
    File/Device size = 1 MB
    Buffer size = 1 MB
    IO Alignment = 512 B
    IO access pattern = r
    IO size (Avg) = 8192 B
    IO size pattern = f
    Burst size (OIOs, Avg) = 1
    Burst size pattern = f
    Inter-burst time (Avg) = 0 us
    Inter-burst time pattern = f
    Read ratio = 1.0
    Latency threshold = 5000 ms
    Buffered IO = 0
    Checksum = 0
    Raw Device Access = 0
    Fill file = 0
    Trace file = .\shcp04-1.trc

    Playing back trace file…

    Test duration = 1.046 s
    Total IOs = 0 Total Bytes = 0
    IOPS = 0 Byte/s = 0 (0.0 MB/s)
    Average latency = -1.#IO us Max Latency = 0 us
    Long IOs = 0

    Reply
  3. chad morgenstern

    Hi David,

    ESX-6 is the name of one of the ESX server upon which the Virtual machines reside. The I/O seen in the packet capture passes From: Application -> ioblazer.db file -> EXT3 filesystem -> sd device -> VMDK -> NFS Datastore -> Storage Controller. The ESX-6 hostname is appearing in the header of the frame. The payload starts after the final @ sign shown in the trace. I found that 50% of the frames contained a payload of all 00s by filtering for nfs.data payload containing all zeroes and looking at the number of frames selected in the print screen. I was led to look for this when I noticed that the storage controller 0 block elimination code was being triggered.

    The file handle is the vmdk, dumping the contents of the block on the controller confirms what the trace showed. The truely unusual thing is that this cannot be reproduced regardless of the permutation:
    1) ioblazer -> DNFS freshly rebooted system
    2) ioblazer -> DNFS system up for some time
    3) ioblazer -> LUN presented through the Linux iscsi software initiator
    4) ioblazer -> VMDK residing on NFS Datastore
    5) ioblazer -> VMDK after initializing all non-pinned memory on the VM with zeroes.
    The issue is at this point a ghost,

    Reply
  4. chad morgenstern

    Hi David,
    I am still troubleshooting this 0 filled block condition, can you elaborate on this:

    “Each page of the memory buffer from which IOBlazer gathers the data to write to the disk/file is filled with the same random pattern,”

    How does IOBlazer confirm that the blocks of memory being read from are not zero filled, or for that matter, what the contents are? HOw hard would it be to change the pattern such that you create your own pattern? Have IOBlazer initiatlize an array filled with say 1MB of random content ( and then use that content with a page number as you are doing). This way there is a failsafe against this type of issue.

    I am not in any way implicating IOBlazer in this, in fact, I have not been able to reproduce this issue today.

    This is what I had been seeing in the traces:
    Frame 3053: 4306 bytes on wire (34448 bits), 1514 bytes captured (12112 bits)
    WTAP_ENCAP: 1
    Arrival Time: Feb 28, 2014 10:39:13.323422000 Eastern Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1393601953.323422000 seconds
    [Time delta from previous captured frame: 0.001773000 seconds]
    [Time delta from previous displayed frame: 0.001965000 seconds]
    [Time since reference or first frame: 28.005056000 seconds]
    Frame Number: 3053
    Frame Length: 4306 bytes (34448 bits)
    Capture Length: 1514 bytes (12112 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:vlan:ip:tcp:rpc:nfs]
    [Coloring Rule Name: TCP]
    [Coloring Rule String: tcp]
    Ethernet II, Src: Vmware_63:8b:8a (00:50:56:63:8b:8a), Dst: Netapp_43:50:a7 (00:a0:98:43:50:a7)
    Destination: Netapp_43:50:a7 (00:a0:98:43:50:a7)
    Address: Netapp_43:50:a7 (00:a0:98:43:50:a7)
    …. ..0. …. …. …. …. = LG bit: Globally unique address (factory default)
    …. …0 …. …. …. …. = IG bit: Individual address (unicast)
    Source: Vmware_63:8b:8a (00:50:56:63:8b:8a)
    Address: Vmware_63:8b:8a (00:50:56:63:8b:8a)
    …. ..0. …. …. …. …. = LG bit: Globally unique address (factory default)
    …. …0 …. …. …. …. = IG bit: Individual address (unicast)
    Type: 802.1Q Virtual LAN (0×8100)
    802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3081
    000. …. …. …. = Priority: Best Effort (default) (0)
    …0 …. …. …. = CFI: Canonical (0)
    …. 1100 0000 1001 = ID: 3081
    Type: IP (0×0800)
    Internet Protocol Version 4, Src: x.x.x.x), Dst: y.y.y.y)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0×00 (DSCP 0×00: Default; ECN: 0×00: Not-ECT (Not ECN-Capable Transport))
    0000 00.. = Differentiated Services Codepoint: Default (0×00)
    …. ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0×00)
    Total Length: 4288
    Identification: 0x5c52 (23634)
    Flags: 0×02 (Don’t Fragment)
    0… …. = Reserved bit: Not set
    .1.. …. = Don’t fragment: Set
    ..0. …. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (6)
    Header checksum: 0xd386 [correct]
    [Good: True]
    [Bad: False]
    Source: y.y.y.y (y.y.y.y)
    Destination: x.x.x.x (x.x.x.x)
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
    Transmission Control Protocol, Src Port: 946 (946), Dst Port: nfs (2049), Seq: 120033, Ack: 494165, Len: 4236
    Source port: 946 (946)
    Destination port: nfs (2049)
    [Stream index: 5]
    Sequence number: 120033 (relative sequence number)
    [Next sequence number: 124269 (relative sequence number)]
    Acknowledgment number: 494165 (relative ack number)
    Header length: 32 bytes
    Flags: 0×018 (PSH, ACK)
    000. …. …. = Reserved: Not set
    …0 …. …. = Nonce: Not set
    …. 0… …. = Congestion Window Reduced (CWR): Not set
    …. .0.. …. = ECN-Echo: Not set
    …. ..0. …. = Urgent: Not set
    …. …1 …. = Acknowledgment: Set
    …. …. 1… = Push: Set
    …. …. .0.. = Reset: Not set
    …. …. ..0. = Syn: Not set
    …. …. …0 = Fin: Not set
    Window size value: 512
    [Calculated window size: 512]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0x98d2 [unchecked, not all data available]
    [Good Checksum: False]
    [Bad Checksum: False]
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    No-Operation (NOP)
    Type: 1
    0… …. = Copy on fragmentation: No
    .00. …. = Class: Control (0)
    …0 0001 = Number: No-Operation (NOP) (1)
    No-Operation (NOP)
    Type: 1
    0… …. = Copy on fragmentation: No
    .00. …. = Class: Control (0)
    …0 0001 = Number: No-Operation (NOP) (1)
    Timestamps: TSval 131914063, TSecr 69830063
    Kind: Timestamp (8)
    Length: 10
    Timestamp value: 131914063
    Timestamp echo reply: 69830063
    [SEQ/ACK analysis]
    [This is an ACK to the segment in frame: 3052]
    [The RTT to ACK the segment was: 0.001773000 seconds]
    [Bytes in flight: 4236]
    Remote Procedure Call, Type:Call XID:0x1b8fa4cc
    Fragment header: Last fragment, 4232 bytes
    1… …. …. …. …. …. …. …. = Last Fragment: Yes
    .000 0000 0000 0000 0001 0000 1000 1000 = Fragment Length: 4232
    XID: 0x1b8fa4cc (462398668)
    Message Type: Call (0)
    RPC Version: 2
    Program: NFS (100003)
    Program Version: 3
    Procedure: WRITE (7)
    [The reply to this request is in frame 3054]
    Credentials
    Flavor: AUTH_UNIX (1)
    Length: 28
    Stamp: 0×00000000
    Machine Name: WE-ESX6
    length: 7
    contents: WE-ESX6
    fill bytes: opaque data
    UID: 0
    GID: 0
    Auxiliary GIDs (0)
    Verifier
    Flavor: AUTH_NULL (0)
    Length: 0
    Network File System, WRITE Call FH:0x809fdeaf Offset:163840 Len:4096 FILE_SYNC
    [Program Version: 3]
    [V3 Procedure: WRITE (7)]
    file
    length: 44
    [hash (CRC-32): 0x809fdeaf]
    decode type as: unknown
    filehandle: 0000000002040080000000008f67000089c57e0002040080…
    offset: 163840
    count: 4096
    Stable: FILE_SYNC (2)
    Data:
    length: 4096
    contents:
    [Packet size limited during capture: NFS truncated]

    0000 00 a0 98 43 50 a7 00 50 56 63 8b 8a 81 00 0c 09 …CP..PVc……
    0010 08 00 45 00 10 c0 5c 52 40 00 40 06 d3 86 ac 14 ..E…R@.@…..
    0020 51 32 ac 14 51 04 03 b2 08 01 15 c5 d0 6d 88 e9 Q2..Q……..m..
    0030 cc 61 80 18 02 00 98 d2 00 00 01 01 08 0a 07 dc .a…………..
    0040 d9 4f 04 29 85 af 80 00 10 88 1b 8f a4 cc 00 00 .O.)…………
    0050 00 00 00 00 00 02 00 01 86 a3 00 00 00 03 00 00 …………….
    0060 00 07 00 00 00 01 00 00 00 1c 00 00 00 00 00 00 …………….
    0070 00 07 57 45 2d 45 53 58 36 00 00 00 00 00 00 00 ..WE-ESX6…….
    0080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0090 00 2c 00 00 00 00 02 04 00 80 00 00 00 00 8f 67 .,………….g
    00a0 00 00 89 c5 7e 00 02 04 00 80 00 00 00 00 40 00 ….~………@.
    00b0 00 00 64 bf 18 00 40 00 00 00 00 00 00 00 00 00 ..d…@………
    00c0 00 00 00 02 80 00 00 00 10 00 00 00 00 02 00 00 …………….
    00d0 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    00e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    00f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0150 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0180 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    01a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    01b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    01c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    01d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    01e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    01f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0220 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0270 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0290 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    02a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    02b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    02c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    02d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    02e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    02f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0300 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0310 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0320 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0330 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0340 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0350 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0360 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0370 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0380 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0390 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    03a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    03b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    03c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    03d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    03e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    03f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0400 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0420 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0430 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0450 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0480 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0490 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    04a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    04b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    04c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    04d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    04e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    04f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0500 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0510 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0520 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0530 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0540 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0550 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0560 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0570 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0580 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0590 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    05a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    05b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    05c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    05d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    05e0 00 00 00 00 00 00 00 00 00 00 ……….

    Reply
    1. Davide Bergamasco

      Hi Chad,

      Once again, apologies for the delay in replying you.

      > I am still troubleshooting this 0 filled block condition, can you elaborate on this:
      >
      > “Each page of the memory buffer from which IOBlazer gathers the data to write to the disk/file is filled with the same random pattern,”
      >
      > How does IOBlazer confirm that the blocks of memory being read from are not zero filled, or for that matter, what the contents are?

      During the initialization phase, IOBlazer allocates a chunk of memory to use as a buffer and fills each page of this buffer with a random pattern generated on the fly. Then, to make each page different, writes the page number in the first two bytes of said page (I double checked the code for correctness).

      > HOw hard would it be to change the pattern such that you create your own pattern? Have IOBlazer initiatlize an array filled with say 1MB of random content ( and then use that content with a page number as you are doing). This way there is a failsafe against this type of issue.

      This wouldn’t be too hard, but it would take some time which, unfortunately, I don’t have at this moment. I’ll add this to the ever-growing IOBlazer TODO list…

      > I am not in any way implicating IOBlazer in this, in fact, I have not been able to reproduce this issue today.

      So it does not happen consistently. Have you observed if it occurs with certain file/page/buffer size combinations? Also, I noticed that the name of your host “WE-ESX6″ shows up in the NFS payload of the trace you captured. That does not look like something IOBlazer would write. Could you confirm which file filehandle 0000000002040080000000008f67000089c57e0002040080… refers to?

      Reply
  5. chad morgenstern

    David,

    I understood that Each page of the memory buffer from which IOBlazer gathers the data to write to the disk/file is filled with the same random pattern, but it’s then made unique by writing the page number in the first word.

    However, my packet capture tells another story. Im seeing a goodly number of the packets payload contains all zeros, while other contains what appears to be as you said a random pattern of “garbage”

    Any ideas?

    Reply
    1. Davide Bergamasco

      Hey Chad,

      Sorry for my late reply, but this week has been pretty insane for me. You may get lots of ’0′ pages when you are reading from a test file which has not been pre-filled. If you care about reading out ‘real’ content, you have to pre-fill the file with the -F option.

      Hope this helps…

      Reply
      1. chad morgenstern

        Thanks for your reply David,

        Your statement makes sense to me for read workload but not for write workloads. What does the content of the ioblazer.db file have to do with the content of the write operations?

        Further, please confirm my thoughts: rather than fill each ioblazer.db file, I prefill only the ranges in the file used by ioblazer when the trace files are used.

        For example:
        I use dd to create a sparse file of 16GB,
        then I take, for example, a trace file that is 50% read/50% write and (sed “s/read/write/g”).
        I then run ioblazer against the ioblazer.db file using the modified trace file above.
        This target populates the ioblazer.db file for future replay without consuming undesired (or non-available space)

        Please answer both of my questions :-D

        Reply
        1. Davide Bergamasco

          > Your statement makes sense to me for read workload but not for write workloads. What does the content of the ioblazer.db file have to do with the content of the write operations?

          Oops… Sorry, I misread your intro and I though you were talking about a read workload. I’ll take a look at the code and see if there is something wrong with it.

          > Further, please confirm my thoughts: rather than fill each ioblazer.db file, I prefill only the ranges in the file used by ioblazer when the trace files are used.
          >
          >For example: I use dd to create a sparse file of 16GB, then I take, for example, a trace file that is 50% read/50% write and (sed “s/read/write/g”). I then run ioblazer against the ioblazer.db file using the modified trace file above. This target populates the ioblazer.db file for future replay without consuming undesired (or non-available space)

          This approach works as long as you use the original trace against the pre-filled ioblazer.db files.

          > Please answer both of my questions :-D

          Why would I answer only one of them? :-)

          Reply
  6. Davide Bergamasco

    If I recall correctly, (1) is a timestamp in us and (3) is the number of elements in the scatter-gather list.

    Reply
  7. shree

    I have a trace file with the following format:

    2147483669,512,1,read,5465080,4144110262847
    2147483750,65024,16,read,1935928,4144110432766

    What do fields (1) and (3) indicate?

    field (2): io size
    field (4): io type
    field (5): offset
    field (6): timestamp

    Thank you!

    Reply
    1. Davide Bergamasco

      Oops… posted as a comment instead of a reply. Second try…

      If I recall correctly, (1) is a timestamp in us and (3) is the number of elements in the scatter-gather list.

      Reply
      1. shree

        how is timestamp in (1) different from timestamp in (6)
        and what does “elements in scatter gather list” really mean? Thanks!

        Reply
        1. Davide Bergamasco

          I’m sorry, I was wrong on (1). That’s actually the IO serial number. The number of elements in the scatter gather list is the number of buffers that were accessed to put together a write or, conversely, where the various pieces of a read ended up to.

          Reply
              1. shree

                In the case it should be a unique number?

                But in my trace file, there are ~39K entries but i only have 128 unique IO serial numbers that are not in consecutive order.

                Thanks!

                Reply
  8. chad morgenstern

    David,

    With so many arrays and caching type devices out there now that have advanced capabilities such as inline deduplication and inline compression, having the ability to specify a deduplication ratio or compression ratio target would come in very handy. Different workload have different patterns that I for one would like to match on.

    Making the content not dedulicatable defeats these technologies and limits our ability to validate.
    Just as fully compressible/deduplicatable test workload is unrealistic, having none is less than ideal these days as well.

    Consider vdbench or iozone, those generators allow you to specify the ratios of each. What do to think about adding this feature?

    Reply
    1. Davide Bergamasco

      Hi Chad,

      I completely agree with you. Lots of things have changed in the storage landscape since ioblazer was first conceived and then released as a fling, and unfortunately I haven’t found the time to keep it up-to-date. I’m adding the knob to control the amount of duplicated content in my todo list. Sooner or later I’ll get to implement it :)

      Thanks,

      Davide

      Reply
  9. Chad Morgenstern

    Thanks Davide,

    It turns out the problem was with the alignment on the host from which I replayed the traces. The starting offset of the partition was on the 63 sector rather than 64th. An oversite I have corrected.

    This replay feature is very exciting for us, it allows us to create a library of workloads to reuse at later times, as well as a simple way to quickly scale up and out.

    Well Done!!!

    Reply
  10. chad morgenstern

    I’m interested in using ioblazer in replay mode. I’m concerned about page size, so I set Disk IO alignment to 4096 using -A.

    Running ioblazer in playback mode, I get this:
    Warning: Trace Playback mode enabled. IO Spec parameters will be ignored.

    Though I see this:
    Test parameters: 22200
    Test duration = 180 s
    Worker threads = 1
    File/Device path(s) = /iob1,
    File/Device size = 1 MB
    Buffer size = 1 MB
    IO Alignment = 4096 B <—-
    IO access pattern = r
    IO size (Avg) = 8192 B

    I see from my storage array statistics that the I/O is not aligned.

    Is the -A ignored by ioblazer when doing replay mode? If we modified the code to not ignore the command line, would the I/O align as desired?

    Chad

    Reply
    1. Davide Bergamasco

      Chad,

      When using IOBlazer in reply mode, the command line parameters that define the IO pattern are ignored because the IO process is entirely driven by the trace. If the trace contains unaligned IOs, IOBlazer will reply them verbatim on the target disk.

      Davide

      Reply
  11. sfaris

    Hi, I tried running ioblazer on Mac OSX Mountain Lion, and it immediately errors out.

    In preparation, I created a file “ioblazer.db” in the /tmp directory of size 100MB and then used the run command:

    ./ioblazer-osx-x64 -a r -t 10 -r 1 -o 1 -d /tmp -f 100

    This produces the error message:

    ERROR: partial IO# 0: expected 8192 bytes, transferred 0 bytes

    Can you point me in the right direction to get it working on Mac OSX Mountain Lion? Thanks!

    Reply
    1. Davide Bergamasco

      Hi Sfaris,

      Have you tried with the “-F” option? When this option is specified, the test file is created and filled before the experiment starts. Please let me know if that solved your problem.

      Thanks,

      Davide

      Reply
  12. Rory

    Davide,

    First thank you for publishing this tool including source code!

    For our internal testing purposes we have been using your tool and have fixed a couple bugs in it as well as adding several enhancements to it. We would like very much to push these bug fixes and enhancements back to you; what is the correct mechanism to do so?

    -Rory

    Reply
    1. Davide Bergamasco

      Hi Rory,

      Thank you for your kind words!! And thank you also for fixing bugs and contributing to improving IOBlazer. At this time there is no “formal” channel to contribute code back as no one has done that so far. I’d say that the easiest way to proceed is for you to send me a patch which I’ll apply to the code base before releasing the next version (I have a couple of fixes and improvements in the pipeline as well). I’ll make sure to mention your contribution in the change log.

      Thanks,

      Davide

      Reply
  13. Joe Fagnani

    Davide,

    We’ve recorded a number of vscsiStats workloads and when we replay them with IOBlazer, the replay is 20% to 30% longer that the record time regardless of how performant the Storage is, including a local hard disk. The ioblazer.db is fully inflated and the replays tested have been on 32-bit Windows XP and 64-bit Windows Server 2003.

    We were expecting (and hoping) that replay time (and resultant rate metrics) would correspond with Storage performance. Can you shed any light as to what may be going on and what we might do to use vscsiStats replays to rate Storage in a repeatable and defined way?

    Thanks,

    Reply
    1. Davide Bergamasco

      Joe,

      This problem may be due to the resolution of the Windows timer, which is around 15 ms. If two consecutive IOs are closer than 15 ms in the trace, they will be played back separated by at least 15 ms. If the trace has many of these “close” IOs, it will eventually play back slower than the original.

      You might want to try to play back the trace using a a Linux VM with the kernel recompiled with the HZ parameter (which determines the timer tick) set to 1000. This will result in a timer resolution of 1 ms.

      Cheers,

      Davide

      Reply
      1. Joe Fagnani

        Davide,

        Thanks. I was afraid the answer was going to be something like that given the similar post below using an older Linux kernel below. According to a Microsoft WP, Windows 7 is also 15.6 ms. Just a thought…Microsoft allows Application Developers to change the default timer resolution for their App (mostly multimedia). Some Windows 7 documentation is found here:
        http://download.microsoft.com/download/3/0/2/3027D574-C433-412A-A8B6-5E0A75D5B237/Timer-Resolution.docx

        Any chance you’ve given a thought to modifying the IOBlazer code to set the Windows timer resolution to 1ms as well as compiling a 64-bit version?

        Thanks,

        Reply
        1. Davide Bergamasco

          Joe,

          I have a few changes in the pipeline for IOBlazer and I’ll definitely consider adding to those the improvement of the Windows timer resolution, and the 64-bit binary.

          Cheers,

          Davide

          Reply
          1. Joe Fagnani

            Davide,

            It turns out that Interrupt timers in our case weren’t responsible for longer replay than record times. It seems that the replay OS matters. The vscsiStats wokloads were from Windows 7 Systems which have a 1MB optimized disk I/O. Basically, IOBlazer blows out the Disk calls and each OS handles the Disk I/O based on it’s optimizations. I did vscsiStats traces of IOBlazer replays on several OSs and the partial histogram below is interesting:

            vscsiStats Trace of IOBlazer Replay
            Byte Size Replay File w2k8R2 w2k3-64 rhel6
            65536 2522 2522 2541 2497
            81920 317 317 3145 320
            131072 453 453 0 472
            262144 189 189 0 217
            524288 73 73 0 269
            1048576 46 46 0 0
            >1048576 71 71 0 0

            W2k8R2 replayed the vscsiStats workload as it was recorded, W2k3-64 bit couldn’t Read or Write anything larger than 80 KB and rhel6 defaulted to .5 MB.

            Thanks,

            Reply
              1. Joe Fagnani

                Davide,

                No, I haven’t been replaying with buffered I/O. We want to achieve as close to a real world workload as possible on the replay. Would adding the -B switch to the command line help achieve that?

                Reply
                1. Davide Bergamasco

                  Joe,

                  Not at all :) I actually asked that question because that would have easily explained the difference between the trace and the actual IOs. The buffer cache decouples completely the application IO pattern from the device IO pattern as the OS caches and consolidates adjacent IOs. The discrepancy you are seeing can still be explained by IO splitting and/or coalescing at the device driver level, though. I guess your best bet is to stick with W2K8R2.

                  Cheers,

                  Davide

                  Reply
  14. Jeremy

    Hi,

    Thanks for this amazing tool for vSphere administration ! I have just one question : on one server (2008R2), I got an error (GetQueuedCompletionStatus failed las Error = 5) when I use IOBlazer for RAW access with a file size (-f) > 1MB.

    It works like a charm on all others servers. I can’t explain why..

    Thanks.

    Jeremy.

    Reply
    1. Davide Bergamasco

      Hey Jeremy,

      Thanks for your kind words! System Error 5 is “ERROR_ACCESS_DENIED”, so it seems that the file you’re using to do your IO experiments does not have the correct access rights. It should be readable and writable by all users (or at least by the Administrator).

      Thanks,

      Davide

      Reply
  15. Muthu

    Hi,
    I downloaded the ioblazer pkg and untarred. I got the following files:
    ioblazer.c IOBlazer.exe ioblazer-linux-x64 ioblazer-linux-x86 ioblazer-osx-x64 Makefile README

    Mine is a RHEL6.1 machine, I have libaio installed.. Here is that confirmation:
    #rpm -qa| grep aio
    libaio-0.3.107-10.el6.x86_64
    libsane-hpaio-3.10.9-3.el6.x86_64
    #
    #find / -name libaio*
    /usr/share/doc/libaio-0.3.107
    /lib64/libaio.so.1.0.1
    /lib64/libaio.so.1
    /lib64/libaio.so.1.0.0
    #

    When I do “make”, I get the following error:
    #cd ioblazer-1.01
    #make
    gcc -c -m64 -Wall -Werror -D__LINUX__ -D_LARGEFILE64_SOURCE ioblazer.c
    ioblazer.c:83:20: error: libaio.h: No such file or directory
    ioblazer.c:188: error: field âiocbâ has incomplete type
    ioblazer.c: In function âworkerThreadâ:
    ioblazer.c:1313: error: âio_context_tâ undeclared (first use in this function)
    ioblazer.c:1313: error: (Each undeclared identifier is reported only once
    ioblazer.c:1313: error: for each function it appears in.)
    ioblazer.c:1313: error: expected â;â before âctx_idâ
    ioblazer.c:1329: error: âctx_idâ undeclared (first use in this function)
    cc1: warnings being treated as errors
    ioblazer.c:1330: error: implicit declaration of function âio_queue_initâ
    ioblazer.c:1336: error: invalid application of âsizeofâ to incomplete type âstruct io_eventâ
    ioblazer.c:1444: error: implicit declaration of function âio_prep_preadâ
    ioblazer.c:1455: error: implicit declaration of function âio_prep_pwriteâ
    ioblazer.c:1473: error: implicit declaration of function âio_submitâ
    ioblazer.c:1495: error: implicit declaration of function âio_geteventsâ
    ioblazer.c:1515: error: invalid use of undefined type âstruct io_eventâ
    ioblazer.c:1515: error: dereferencing pointer to incomplete type
    make: *** [ioblazer.o] Error 1
    #

    Kindly help..

    Thanks and Regards,
    Muthu

    Reply
    1. Davide Bergamasco

      Hi Muthu,

      Looks like you are missing the libaio.h header file which is part of libaio-devel-0.3.107-10.el6.x86_64 RPM. Once you install said RPM everything should work.

      Cheers,

      Davide

      Reply
  16. Kenny

    This is really a useful tool, but I found it doesn’t work well to interpret trace file from ESX 5? Any plan to upgrade this tool?

    Reply
  17. Richard

    Davide,

    I am trying to understand how the -f parameter plays in what I see as my MB/s result.

    I see the larger file size decrease my MB/s, and I would like to be clear why this is the case ?

    Would you have a recommended value for this, e.g. is default OK ?

    Thanks,
    Richard.

    Reply
    1. Davide Bergamasco

      Richard,

      As the file size increases, so does the area of the physical disk(s) covered by the data. If the IO access pattern is random (default in IOBlazer) then the disk heads will have to travel farther to reach the data, resulting in larger IO latency increases and, consequently, in lower throughput.

      The default file size is very small in order to make the file fit entirely in the cache of the IO subsystem being used. This maximizes throughput, but it may or may not be realistic enough for your use case. I’d suggest to use a size as close as possible to the size of the working set of the application you’re are planning on running on your system.

      Hope this helps,

      Davide

      Reply
  18. Mario Grunert

    Does it write zeros or some random data ?

    I could imagine that a zeroes load will vanish in the compression algorythm on Ontap 8 or the inline dedup in ZFS.

    Reply
    1. Davide Bergamasco

      Mario,

      Each page of the memory buffer from which IOBlazer gathers the data to write to the disk/file is filled with the same random pattern, but it’s then made unique by writing the page number in the first word. Therefore, if the dedupe algorithm looks at something as large as (or larger than) a page (4 KB) then it won’t find any duplicated data. However, if it looks at something smaller, it may find data duplication depending on the alignment.

      Hope this helps,

      Davide

      Reply
  19. Lance Braswell

    Hi Davide,

    Love the tool. It’s a big help. I apologize if this has already been covered elsewhere. When performing vscsiStats trace and replay via IOBlazer on RHEL 4.8 -> RHEL 5.5 and compiled from source, I noticed that the rate of IO replay was drastically slower that what was recorded. Context for this is ioblazer 1.01.

    The first hint at the source of the apparent slowdown on replay was on systems running with VMware Linux timekeeping best practices. I.e. divider=10 (so HZ=100). For any vscsiStats recording with IOps >> 100, the replay was always delivering IOps in the 100 range.

    Apparently until more recent versions of Linux kernel, nanosleep() resolution is no better than HZ so when you uSleep() -> nanosleep() on your replay, your IOs are effectively delivered no faster than HZ. I then ran with default HZ=1000 and got better results but since nanosleep sleeps for at least as long as the time requested (i.e. 1/HZ), it’s still not enough resolution to replay back with good fidelity if your IOs are recorded at a rate > HZ.

    Here is a demonstration:

    First few IOs

    Serial Number,IO Data Length,Num SG Entries,Command Type,LBN,Time Stamp (microseconds)
    2148401154,13824,4,write,7920,3890865606325
    2148401258,13824,2,read,954,3890865606335
    2148401231,13824,2,read,12388,3890865606343

    deltas in us respectively between io 1,2 and io 2,3:
    10,8

    At HZ=100 (2.6.9-89.0.11.ELlargesmp/RHEL 4.8/notsc divider=10):

    Here is an strace of the replay (via strace -f -ttt -o /tmp/out ./ioblazer -P /tmp/vmtst-ads-2-1.8329.RAW_100_1.2011_08_04_22_52_28.300s.txt -d /dev/datavg/lvol1 -R -f 100)

    28523 1312523281.074311 open(“/tmp/vmtst-ads-2-1.8329.RAW_100_1.2011_08_04_22_52_28.300s.txt”, O_RDONLY) = 4
    28523 1312523281.074329 fstat(4, {st_mode=S_IFREG|0644, st_size=21253282, …}) = 0
    28523 1312523281.074347 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a9555b000
    28523 1312523281.074358 read(4, “Vscsi Cmd Trace. (Trace Format V”…, 4096) = 4096
    28523 1312523281.074393 gettimeofday({1312523281, 74396}, NULL) = 0
    28523 1312523281.074405 io_submit(182894034944, 1, {…}) = 1
    28523 1312523281.074440 io_getevents(182894034944, 1, 1, {…}NULL) = 1
    28523 1312523281.082073 gettimeofday({1312523281, 82080}, NULL) = 0
    28523 1312523281.082105 nanosleep({0, 10000}, {222592816416, 0}) = 0
    28523 1312523281.086195 gettimeofday({1312523281, 86201}, NULL) = 0
    28523 1312523281.086219 io_submit(182894034944, 1, {…}) = 1
    28523 1312523281.086255 io_getevents(182894034944, 1, 1, {…}NULL) = 1
    28523 1312523281.086535 gettimeofday({1312523281, 86538}, NULL) = 0
    28523 1312523281.086551 nanosleep({0, 8000}, {222592816416, 0}) = 0
    28523 1312523281.096183 gettimeofday({1312523281, 96189}, NULL) = 0
    28523 1312523281.096207 io_submit(182894034944, 1, {…}) = 1

    We can see the two nanosleep requests for 10000 and 8000. But the actual time between the start of nanosleep and io_submit is:
    (086219 – 082105) = 4114 us between io 1,2 (but asked for 10 us)
    (096207 – 086551) = 9656 us between io 2,3 (but asked for 8 us)

    At HZ=1000 ((2.6.9-89.0.11.ELlargesmp/RHEL 4.8):

    6625 1312518511.432091 open(“/tmp/vmtst-ads-2-1.8329.RAW_100_1.2011_08_04_22_52_28.300s.txt”, O_RDONLY) = 4
    6625 1312518511.432113 fstat(4, {st_mode=S_IFREG|0644, st_size=21253282, …}) = 0
    6625 1312518511.432130 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a9555b000
    6625 1312518511.432169 read(4, “Vscsi Cmd Trace. (Trace Format V”…, 4096) = 4096
    6625 1312518511.432205 io_submit(182894034944, 1, {…}) = 1
    6625 1312518511.432257 io_getevents(182894034944, 1, 1, {…}NULL) = 1
    6625 1312518511.454044 nanosleep({0, 10000}, {222592816416, 0}) = 0
    6625 1312518511.455475 io_submit(182894034944, 1, {…}) = 1
    6625 1312518511.455526 io_getevents(182894034944, 1, 1, {…}NULL) = 1
    6625 1312518511.455955 nanosleep({0, 8000}, {222592816416, 0}) = 0
    6625 1312518511.457463 io_submit(182894034944, 1, {…}) = 1

    We can see the two nanosleep requests for 10000 and 8000. But the actual time between the start of nanosleep and io_submit is:

    (455475 – 454044) = 1431 us actual between io 1,2 (but asked for 10 us)
    (457463 – 455955) = 1508 us actual between io 2,3 (but asked for 8 us)

    I understand that more recent versions of mainline Linux kernel have better resolution on nanosleep but I’m not able to test with it at the moment. Just want to call this out to see if it’s the right finding.

    kind regards,
    Lance

    Reply
    1. Lance Braswell

      Hi Davide,

      I sent you an email with a patch as a POC workaround to the above to further the discussion.

      Lance

      Reply
      1. Davide Bergamasco

        Hi Lance,

        Thank you for pointing out this issue and your in-depth analysis. I believe you are absolutely right: According to many sources nanosleep sleeps for *at least* the requested time or a timer tick, whichever is the largest. Apparently high-resolution timers are available from 2.6.22 on, and this explains why it worked for me as I’m using Fedora 14 with a 2.6.35 kernel. I’ll take a look at your patch and test it out as soon as I can.

        Thanks!

        Davide

        Reply
  20. Tomek Podlaski

    Hi Davide,

    first of all thanks for the tools finally there is an app which can compete with the infamous IOMeter and does integrate with VMware.
    I came across a problem when trying to do a vscsiStats trace using logchannellogger on the ESXi 4.0U2. My scenario is: catch a trace of a VDI VM using vscsiStats (e.g. user workstation) then replay it on 100+ clients using synchronized IOBlazer’s instances. As I did not see the logchannellogger on the ESXi i copied it from the test ESX. But got some errors:
    1) when doing, logchannellogger -l –> VSI_GetInstanceListAlloc failed: 0xbad0003/sbin #
    2) when trying, logchannellogger vscsi_cmd_trace_9394432_8219 vscsi_cmd_trace_9394432_8219.out & (with my IDs of course:) –> Error on VSIGet: Is a directory
    3) there is also slight difference in the vscsiStats output –> vscsi_cmd_trace_ID_ID

    Is it possible to make it run on the ESXi?

    Kind regards
    Tomek

    Reply
    1. Davide Bergamasco

      Hi Tomek,

      Sorry for my late reply, but yesterday I was off.

      Thank you very much for your kind words. I would be so harsh on IOMeter, it’s a good tool with its own limitations, just like IOBlazer :)

      I suspect that there is some binary incompatibility between the ESX version of logchannellogger (which is basically a linux binary) and the ESXi version. I’m afraid you really have to use the ESXi logchannellogger version which is in the /sbin directory.

      Hope this helps…

      Davide

      Reply
      1. Tomek Podlaski

        Davide,

        thank you for the reply. I tried with the logchannellogger from ESXi 4.1U1(348481) but got same results when running it on ESXi4.0(261974) – 1) from my prev post. So I belive it is not possible to gather SCSI trace that can be used in the IOBlazer when there is ESXi 4.0 under the hood. Correct me if I am wrong as I did not test all posibilites using different builds of ESXi. If somebody manged to make it work I would appreciate information.

        Kind regards
        Tomasz Podlaski

        Reply
        1. Davide Bergamasco

          Tomek,

          I just built a fresh logchannellogger from the ESXi 4.0 source tree. If you give me your email address I’ll ship it to you. Hopefully this one should work.

          Thanks,

          Davide

          Reply
                  1. Joe Fagnani

                    Davde,

                    I apologize so much but I didn’t receive it, or it wound up in junk mail and I unintentionally deleted it.

                    Also, I’ve been trying to find out what the vscsistats trace column “Num SG Entries” means, but it doesn’t seem to be documented anywhere.

                    Thanks,

                    Reply
                    1. Davide Bergamasco

                      Hi Joe,

                      I just resent the file. Please let me know if you got it this time.

                      Regarding “Num SG Entries” it refers to the number of entries in the Scatter Gather list associated with an IO. This is some low level stuff handled by the HBA and its driver in order to “gather” the various pieces of data making up a write from separate memory locations, and, conversely, “scatter” the various pieces of data from a read to separate memory locations.

                      Hope this helps,

                      Davide

  21. Miroslav

    Hi Davide,

    Thanks for putting together a great tool!

    I’m trying to use IOBlazer in a Windows environment, and am having trouble expressing what I’m trying to do via the parameters. Perhaps somebody could point me in the right direction. What I’d like to do is create several worker threads to generate random I/O to the same file/disk. When I specify more than one worker, IOBlazer tries to generate load on multiple drives. For example:


    PS F:\> ioblazer -a r -A 65536 -b 1 -d J: -f 92160 -i 512 -o 32 -w 2 -t 60

    IOBlazer, the ultimate IO benchmark :D

    Test parameters:
    Test duration = 60 s
    Worker threads = 2
    File/Device path(s) = J:, K:,
    File/Device size = 92160 MB
    Buffer size = 1 MB
    IO Alignment = 65536 B
    IO access pattern = r
    IO size (Avg) = 512 B
    IO size pattern = f
    Burst size (OIOs, Avg) = 32
    Burst size pattern = f
    Inter-burst time (Avg) = 0 us
    Inter-burst time pattern = f
    Read ratio = 1.0
    Latency threshold = 5000 ms
    Buffered IO = 0
    Checksum = 0
    Raw Device Access = 0
    Fill file = 0
    Trace file =

    Running test for 60 seconds...
    Test complete.

    Test duration = 60.045 s
    Total IOs = 44640 Total Bytes = 22855680
    IOPS = 743 Byte/s = 380643 (0.4 MB/s)
    Average latency = 39114.247 us Max Latency = 166000 us
    Long IOs = 0

    Instead of aiming both workers at the J: drive, it’s putting one on the J: drive and another on the next sequential drive letter (K:). Is there a way to specify that both workers should pound the same drive? I’ve tried specifying it as “J: J:” and “J:,J:”, but that just errors out looking for “J: J:\ioblazer.db”. Is there some usage trick that I’m missing, is this a bug in how the parameters are parsed, or is this the intended behavior and I’ll need to spawn multiple instanced of IOBlazer in the background in order to get the behavior I’m looking for?

    Thanks and take care,
    Miroslav

    Reply
    1. Davide Bergamasco

      Hi Miroslav,

      Thank you for your kind comments!

      One thread per test file/disk is IOBlazer’s intended behavior. The philosophy behind this design choice is to maximize the parallelism in the IOs hitting the backend storage. If multiple threads hit the same file/device, the IOs they originate may end up being serialized by the OS. If you want to increase the parallelism on the same file/device you can increase the number of outstanding IOs. However, you have to be careful not to exceed any command queue size on the IO path to avoid adding unwanted latency.

      Hope this helps!

      Davide

      Reply
  22. Gregoire Laverdiere

    Last IO operation failed: Error=38
    EOF on READ. Please check file size

    Any idea anyone?

    Thanks

    Reply
    1. Davide Bergamasco

      Gregoire,

      I’m assuming you’re playing back a vSCSI trace. In this case, this error may indicate that the test file you are playing back the trace on is smaller than the virtual disk you captured the trace from. Try to make the test file larger by using the “-f test_file_size_in_MB” option in conjunction withe the “-F” option to fill it up with random data before the test starts.

      Hope this helps!

      Davide

      Reply
      1. Gregoire Laverdiere

        Thanks for the quick response.

        We are just trying to pump some load into the infrastructure. Also we are searching for a tool that would allow us to create comperative baseline in both the windows world and the linux world.

        We were able to pump 18000 iops with 220MB/s from a 2 vCpu win xp vm using IOmeter. Which looking at the nature of the VM seems pretty good.

        It seems that IOblazer would work on both plateform. I am trying to genereate a similar type of workload using IOblazer so I can run comparative tests.

        Reply
        1. Davide Bergamasco

          Gregoire,

          Could you please send me the VM disk configuration and the command line you are using to run IOBlazer at davide at vmware dot com?

          Thanks,

          Davide

          Reply
          1. Richard O Brien

            Hi,

            Was there ever a resolution to this, trying

            - ioblazer -t 30 -d e:\tmp

            on a W2003x64 VM with two drives.

            Error as follows,

            Last IO operation failed: Error=38
            EOF on READ. Please check file size.

            Any help appreciated.

            Thanks,
            Richard.

            Reply
  23. Chris

    help… really like ioblazer and it’s invaluable at the moment but I am running into an issue when trying to replay a trace.

    Having followed your instructions attempting a playback i receive the error meesage – “Last IO operation failed: Error=112″

    Can anyone help?

    Reply
    1. Davide Bergamasco

      Hi Chris,

      Sorry you’re running into a problem with IOBlazer. Could you please send me the trace at davide(at)vmware(dot)com?

      Thanks,

      Davide

      Reply
  24. David Davis

    Hi,
    I am very interested in learning more about IOBlazer but am struggling to picture, in simple term, a couple of common use cases where a VMware Admin would use it.
    Could you give me a couple of scenarios, please?
    Also, in simple terms, can you explain the role of vscsiStats vs IOBlazer?
    Thank you for your contribution of this performance tool to the VMware community!
    David

    Reply
    1. Davide Bergamasco

      Hi David,

      Thank you for your interest in IOBlazer.

      We initially developed IOBlazer as an emulator for a complex storage workload like MS SQL Server. That allowed us to do some in-depth performance studies with a (somewhat) realistic workload without having to install and run the actual app, which can be a pretty complex and time consuming task. Also, more often than not, using the real app does not allow for results comparability because of the inherent variability in the workload.

      Another important use case for us was to investigate IO performance differences between VMs running different guest OSes, i.e., Windows and Linux. Being IOBlazer a multi-platform tool, it allowed us to do a comparative performance analysis in a simple and repeatable way.

      Probably the most important use case (and what makes IOBlazer a VMware specific tool) is to replay IO traces captured from running VMs with vsciStats. IO performance issues can be “easily” debugged by replaying exactly the same workload over and over without running the actual app.

      Hope this helps!

      Davide

      Reply
  25. chad morgenstern

    We would like to use logblazer to replay vscisStats. I’m not clear on how to capture the output of the vscsiStats –tracecmds command, and Im not having luck finding the answer to this on the web.

    Can you explain how to use: $ logchannellogger

    I see that the following is running:
    [root@s1d-1 ~]# /usr/sbin/logchannellogger -l
    vscsi_cmd_trace_7501_8209

    But how do I capture the data? Specifically, what is the binary trace file that is reffered to in the help command? Can you show me an example?

    Chad

    Reply
    1. Davide Bergamasco

      Chad, thanks for your interest in IOBlazer. Here is an example showing how to capture a vscsiTrace (there is way more stuff than you asked for, but I thought it might be useful for others as well).

      First, figure out which VM and virtual disk you want to trace as follows:

      ~ # vscsiStats -l
      Virtual Machine worldGroupID: 9394432, Virtual Machine Display Name: dbergamasco-dev {
      Virtual SCSI Disk handleID: 8219
      }
      Virtual Machine worldGroupID: 9370301, Virtual Machine Display Name: VMware Data Recovery {
      Virtual SCSI Disk handleID: 8202
      Virtual SCSI Disk handleID: 8203
      }

      Suppose we are interested in the first VM only, so we start the trace with:

      ~ # vscsiStats -t -s -w 9394432 -i 8219
      vscsiStats: Starting Vscsi stats collection for worldGroup 9394432, handleID 8219
      vscsiStats: Starting Vscsi cmd tracing for worldGroup 9394432, handleID 8219
      vscsi_cmd_trace_9394432_8219
      Success.

      The last line mentions the name of the logChannel used by vscsiStats to dump the trace. So now start the logger on said cahnnel as follow:

      # logchannellogger vscsi_cmd_trace_9394432_8219 vscsi_cmd_trace_9394432_8219.out &

      Now the logger is collecting in background the vscsiStats binary trace into the vscsi_cmd_trace_9394432_8219.out file. Wait as long as you need to gather a trace of the workload you are interested in and then stop vscsiStats collection (logchannellogger will stop as well):

      ~ # vscsiStats -x
      vscsiStats: Stopping all Vscsi stats collection for worldGroup 9394432, handleID 8219
      Success.
      [1] + Done(1) logchannellogger vscsi_cmd_trace_9394432_8219 vscsi_cmd_trace_9394432_8219.out
      ~ #

      At this point you have the trace saved in binary format in vscsi_cmd_trace_9394432_8219.out. In order to convert it in text format and feed it to IOBlazer do the following:

      ~ # vscsiStats -e vscsi_cmd_trace_9394432_8219.out > vscsi_cmd_trace_9394432_8219.txt

      Unfortunately there is one more thing you need to do: edit the text file and remove the first line, the one with the names of the various fields of the trace. When I initially developed IOBlazer that line was not there. Since IOBlazer does not expect such line, it will terminate parsing the trace as soon as it reads it. I fixed this problem in a new version of the tool that should be available soon on this site.

      Hope this helps!

      Davide

      Reply
    1. Davide Bergamasco

      J1mbo, thank you for your comments. I’ll make sure to add 4K aligned IOs, or better, a configurable alignment, to the next release.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>