Dynamic Resource Management in a Virtualized Data Center

Participants

Project Goals

The goal of this project is to dynamically reallocate physical resources to virtual machines in a virtualized data center to maximize data center revenue.

Meetings

12/04/12: Counter-intuitively, non-work-conserving limits-based dynamic resource assignment does better than shares-based – is cache contention the culprit?
11/20/12: Alternate evaluation baselines and fine-grained analysis using time-series charts for resource allocation/usage, I/O latency, and revenue for individual VMs
10/30/12: Discussion on results from shares experiments – how do we ensure comparable performance data?
10/23/12: Updates on experiments comparing with equal/proportional allocation baselines; New direction - using ESX shares to complement use of reservations
10/02/12: Troubleshooting esxtop – solution: use vscsistats!
09/25/12: Updates, Latency range issues with reads
08/28/12: Handling performance variability – sequential ANN training
08/21/12: Summer updates from Sajib and Online Model Updating discussion
08/07/12: Monitoring overhead and Latency issue addressed; Addressing stability of observations and contention-induced prediction errors
07/10/12: 2 problems – missing low-latency regions and low-overhead VM performance monitoring infrastructure
04/24/12: Experimental setup update
04/17/12: Experimental setup change
04/05/12: Experimental setup update & VDC (customer-centric) pitch for paper
03/29/12: Experimental set up done – pilot possibilities
03/20/12: Review of experimental setup and experiment design (ref: word document in email)
03/13/12: Review of status and next steps for OSDI submission
01/17/12: Future directions
10/25/11: Discussing VM sizing section draft
10/18/11: VM sizing updates
10/11/11: Initial report on VM sizing and plans for expansion
09/13/11: Updates on paper writing
09/06/11: Many ideas for improvement of the paper writing – and task division
08/30/11: Updated related work table and Follow-up task list
08/23/11: Revisiting paper message and structure
08/16/11: Revising based on SVM contributions
08/02/11: More noise results, robustness of ann parameter tuning, initial svm results
07/26/11: Including SVM evaluation, framework discussion
07/19/11: Improving related work and re-submission venues
07/12/11: SOCC reviews and framework discussion
06/07/11: Evaluation design
05/31/11: Reviewing scope document, scenarios
05/23/11: New directions – revenue maximization or online modeling?
04/05/11: SOCC eval improvements
03/31/11: SOCC updates, eval and editing plans
03/24/11: Intro, reminders in 3&4, eval improvements
03/17/11: Review and Plans for submission to SOCC
03/15/11: SOSP paper: Detailed review of eval section and writing plans
03/10/11: SOSP paper: Reviewing comments, Section 3 message, Related work
03/03/11: Results update and writing plans for SOSP
02/24/11: Various modeling approaches, evaluating them and telling the story, plans for SOSP writing
02/10/11: Updates on accuracy of sub-modeling approach; need for formalization identified
02/03/11: Clustering findings and initial thoughts on formalizing the training process using sub-models
01/27/11: Sub-model initial findings and next steps on paper writing and eval
01/20/11: Status update, SOSP paper discussion, modeling enhancements
12/02/10: RuBiS benchmark is set up, next steps discussed with Ajay (no audio)
11/18/10: Understanding the ANN mechanisms and alternate implementations
11/04/10: Postmark initial data makes sense and more data being collected, 4-disk RAID0 for final evaluation
10/28/10: Analyzing initial data from Postmark, VMFS caching issue identified
10/19/10: Status update and planning
09/20/10: Resuming meeting — USENIX submission
05/03/10: Paper context and automating model creation
04/29/10: Sysbench accuracy improvement and modified data collection strategy
04/26/10: Modeling accuracy issues and new data collection plans
04/19/10: Initial meeting with Ajay and evaluation plan review and refinement
04/15/10: Discussion of Ajay's initial feedback
04/12/10: Paper writing steps
04/02/10: Revisiting plans for OSDI experiments
03/29/10: Plans for OSDI experiments
02/16/10: Debugging data
01/28/10: IEEE Micro magazine submission plan
12/21/09: Alternate submission venues
12/17/09: Implementation roadblock resolution
12/14/09: The Brute force algorithm for resource reallocation
12/07/09: Update on UTC paper - evaluation plan
12/03/09: Brief HPCA slides review and UTC experiment graphs
11/30/09: HPCA slides review
11/25/09: HPCA presentation, discussion of brute force algo implementation
11/19/09: USENIX paper contents, initial experiments
11/16/09: Possible directions for USENIX paper
11/12/09: 3 Possible directions for model updating
11/10/09: Last minute comments on Workshop paper - Algo modifications
11/09/09: Workshop paper content, NP completeness, mapping section, algo, etc.
11/05/09: Workshop paper structure and next steps
11/02/09: HPCA camera-ready and Workshop paper tasks
10/29/09: HPCA camera ready paper - experiments and content condensing
10/26/09: Workshop paper revised structure
10/22/09: Workshop paper discussion
10/19/09: HPCA Workshop paper contributions and structure
10/15/09: HPCA Workshop paper ideas and final conference camera-ready
10/12/09: Incremental delta-assignment algorithm issues and alternative exhaustive algorithm analysis
10/08/09: Incremental delta-assignment algorithm and discussion on optimizations
10/05/09: More discussion on resource redistribution algorithm
10/02/09: Details of algorithm and knapsack problem requirements
09/28/09: Incremental resource reallocation and mapping to knapsack
09/24/09: HPCA10 Rebuttal
09/11/09: Graphs and message, New Pt options, and Knapsack problem mapping
09/04/09: Problem formulation - feedback from Kaushik, Ming
08/24/09: Overall revenue maximization procedure - detailed discussion
08/17/09: Eurosys08 paper review, SLA curve, challenges discussion
08/03/09: Initial thoughts on online model refinement, benchmarks identified
07/30/09: Final discussions for hpca10 submission
07/27/09: Training models automatically and task assignment for paper
07/20/09: HPCA paper content outline
07/16/09: Discussion of middleware reviews and plans to address them
07/06/09: Debugging memory heatmap and structure of experiments section
07/02/09: Debugging benchmark behavior and error heatmaps
06/22/09: Results are acceptable. Next 3 steps discussion - insights on Elliot, other models, and new hybrid model?
06/18/09: Debugging ANN models and methodology
06/15/09: ANN advanced configuration and R models
06/09/09: Modeling options discussion
04/23/09: Paper tasks - experiments update, architecture figure, writing tasks
04/09/09: Designing graphs and Setting up Task-List related to Paper
04/02/09: Memory intensive benchmark requirements, graphs design, paper writing tasks
03/16/09: Sysbench output debugging. Next steps - explain sysbench output clearly, pilot other sysbench benchmarks, model oltp
03/11/09: Problems with Xen install, Benchmark identified (sysbench)
02/17/09: Middleware paper positioning and discussion of related work
02/10/09: Revisiting assumptions, dynamic model updates, enumeration of modeling techniques
02/03/09: Refining and broadening the paper direction

Design

Sajib: Fill this in.

Systems Research Laboratory

Table of Contents

Dynamic Resource Management in a Virtualized Data Center

Participants

Project Goals

Meetings

Design