==+== FAST '09 Paper Review Form
==-== Set the paper number and fill out lettered sections A through G.
==-== DO NOT CHANGE LINES THAT START WITH ”==+==”!
==+== RAJU MUST REPLACE THIS LINE BEFORE UPLOADING
==+== Begin Review
==+== Paper #000000
==-== Replace '000000' with the actual paper number.
==+== Review Readiness
==-== Enter “Ready” here if the review is ready for others to see:
[Enter your choice here]
==+== A. Overall merit
==-== Enter a number from 1 to 5.
==-== Choices: 1. Reject
==-== 2. Weak reject
==-== 3. Weak accept
==-== 4. Accept
==-== 5. Strong accept
==+== B. Novelty
==-== Enter a number from 1 to 5.
==-== Choices: 1. Published before
==-== 2. Done before (not necessarily published)
==-== 3. Incremental improvement
==-== 4. New contribution
==-== 5. Surprisingly new contribution
==+== C. Longevity
==-== How important will this work be over time?
==-== Enter a number from 1 to 5.
==-== Choices: 1. Not important now or later
==-== 2. Low importance
==-== 3. Average importance
==-== 4. Important
==-== 5. Exciting
==+== D. Reviewer expertise
==-== Enter a number from 1 to 4.
==-== Choices: 1. No familiarity
==-== 2. Some familiarity
==-== 3. Knowledgeable
==-== 4. Expert
==+== E. Paper summary
The authors present some interesting ideas in an important, though well-established, area of I/O scheduling. Addressing fairness and performance simultaneously in storage systems is certainly becoming more important. The authors suggest using two complementary techniques – bounded concurrency and I/O batching – as a means for finely controlling the inherent tradeoffs these optimization dimensions entail. They formalize the problem, pose solutions (some of which are heuristic-based), and evaluate an actual implementation using microbencmarks (mostly) and Postmark.
==+== F. Comments for author
High-level comments:
I liked the fact that the authors are proposing new ideas in an established space. While I have not been keeping up with recent developments in fair I/O schedulers research, I thought the formulation of fairness and efficiency metrics were interesting and the tradeoffs were reasonably explored using real experiments. I think the authors have a good potential paper, but my weak-accept rating is because I think the paper needs more work given its current form.
The approach proposed is fairly straightforward at hindsight (a good thing). Two sub-ideas are proposed and both seem worth considering. The first (bounded concurrency) controls the amount of I/O requests outstanding at the storage device – a higher concurrency allows better performing I/O schedules on an average but with greater variance, thus affecting fairness. The second (variable batching) allows a round-based proportional-share scheduler to batch (and dispatch) more I/Os from a single application than its share within a single round and then starving it for subsequent rounds to make up – this allows applications with greater locality to more efficiently utilize the underlying storage device(s). Thus, both ideas allow trading-off efficiency for fairness. Finally, the authors propose a scheduler that automatically adapts to the workload (composed of multiple applications with varying and variable locality characteristics) and configures batch sizes and concurrency.
I also liked very much the experiments demonstrating the problem with current schedulers in Section 3. I suggest adding a graph illustrating the fairness metric in Fig 3 as well (similar to Fig 2). It is important.
I think the paper was somewhat lacking in motivation presented upfront. My excitement during the initial reading was fairly low, until the point where I reached actual ideas presented, which then engaged me fairly well. However, after reading through the entire paper, I have still not seen concretely what sort of applications or real-world workloads would be in a position to benefit beyond what is provided by the current state of the art? What sort of applications would need a fairness metric of say, 0.1? I think this is a very important missing piece without which readers would find it hard to appreciate your contribution.
I think the solution proposed is certainly worth considering and even perhaps has potential for use in practice. For instance, I wished the authors would have more strongly motivated the use of their work in a virtualized environment (within a VMM I/O scheduler) rather than a standalone server, because fairness in storage resource management is a substantial problem in virtualized environments. The authors' problem formulation can possibly fit that environment well, with perhaps some modifications (each VM application I/O stream is an aggregate stream of possibly more than one I/O applicaton). Also, investigating what sort of fairness metrics are appropriate in such settings would be a very valuable piece of research that in my opinion will prove a springboard for the authors work.
The second part of why I think the paper is not ready yet relates to the evaluation.
(1) The fact that most of the experiments are with synthetic workloads is somewhat of a letdown. I urge the authors to think about benchmarks for/or actual real-world applications that are fairness sensitive to conduct experiments with.
(2) It seems that existing schedulers chosen to evaluate against have configurable parameters that the authors assigned values to. For instance DRR has a configurable quantum of tokens (related to the length of the round) and SFQ has a depth parameter D which controls efficiency vs. fairness. A fair evaluation would vary these parameters to demonstrate the true scope of effectiveness of these alternative algorithms. For instance, using a depth value of D=4 with SFQ and deploying the scheduler on seven 4-disk RAID system (XP RAID) does not make much sense to me. Have only 4 outstanding I/O requests in a large multi-disk system underestimates performance and that is putting it mildly. The authors should evaluate the alternative schedulers in a more systematic and fair manner. I instinctively think their contribution would still be mostly retained, but in the absence of such an evaluation, I am nevertheless left in doubt.
Detailed comments:
- I Definition 1, there seems to be an assumption for an underlying seek-reducing scheduler. Please spell this out either way. What is the implication of such an assumption?
- Why does efficiency remain unchanged until batch size of ~ 50 ? Perhaps this is because the sequential stream is being prefetched and thus masks the overhead due to lack of batching? Please explain.
- A similar idea to variable batching has been proposed in Stream Combination (Liu et al. 2006 ACM Sigbed Review 06) for media servers. How do these compare?
==+== G. Comments for PC (hidden from authors)
[Enter your comments here]
==+== End Review