====== Dynamic Resource Management in a Virtualized Data Center ====== ===== Participants ===== * [[http://www.cs.fiu.edu/~skund001/|Sajib Kundu]] * [[http://www.cs.fiu.edu/~raju/|Raju Rangaswami]] * [[|Ajay Gulati]] * [[http://www.fiu.edu/~duttak/|Kaushik Dutta]] * [[http://www.cs.fiu.edu/~zhaom/|Ming Zhao]] ===== Project Goals ===== The goal of this project is to dynamically reallocate physical resources to virtual machines in a virtualized data center to maximize data center revenue. ===== Meetings ===== * 12/04/12: Counter-intuitively, non-work-conserving limits-based dynamic resource assignment does better than shares-based -- is cache contention the culprit? {{{{internal:projects:res-mgmt-virt:sajib120412.mp3|mp3}} * 11/20/12: Alternate evaluation baselines and fine-grained analysis using time-series charts for resource allocation/usage, I/O latency, and revenue for individual VMs {{{{internal:projects:res-mgmt-virt:sajib112012.mp3|mp3}} * 10/30/12: Discussion on results from shares experiments -- how do we ensure comparable performance data? {{{{internal:projects:res-mgmt-virt:sajib103012.mp3|mp3}} * 10/23/12: Updates on experiments comparing with equal/proportional allocation baselines; New direction - using ESX shares to complement use of reservations {{{{internal:projects:res-mgmt-virt:sajib102312.mp3|mp3}} * 10/02/12: Troubleshooting esxtop -- solution: use vscsistats! {{{{internal:projects:res-mgmt-virt:sajib100212.mp3|mp3}} * 09/25/12: Updates, Latency range issues with reads {{{{internal:projects:res-mgmt-virt:sajib092512.mp3|mp3}} * 08/28/12: Handling performance variability -- sequential ANN training {{{{internal:projects:res-mgmt-virt:sajib082812.mp3|mp3}} * 08/21/12: Summer updates from Sajib and Online Model Updating discussion {{{{internal:projects:res-mgmt-virt:sajib082112.mp3|mp3}} * 08/07/12: Monitoring overhead and Latency issue addressed; Addressing stability of observations and contention-induced prediction errors {{{{internal:projects:res-mgmt-virt:sajib080712.mp3|mp3}} * 07/10/12: 2 problems -- missing low-latency regions and low-overhead VM performance monitoring infrastructure {{{{internal:projects:res-mgmt-virt:sajib071012.mp3|mp3}} * 04/24/12: Experimental setup update {{{{internal:projects:res-mgmt-virt:sajib042412.mp3|mp3}} * 04/17/12: Experimental setup change {{{{internal:projects:res-mgmt-virt:sajib041712.mp3|mp3}} * 04/05/12: Experimental setup update & VDC (customer-centric) pitch for paper {{{{internal:projects:res-mgmt-virt:sajib040512.mp3|mp3}} * 03/29/12: Experimental set up done -- pilot possibilities {{{{internal:projects:res-mgmt-virt:sajib032912.mp3|mp3}} * 03/20/12: Review of experimental setup and experiment design (ref: word document in email) {{{{internal:projects:res-mgmt-virt:sajib032012.mp3|mp3}} * 03/13/12: Review of status and next steps for OSDI submission {{{{internal:projects:res-mgmt-virt:sajib031312.mp3|mp3}} * 01/17/12: Future directions {{{{internal:projects:res-mgmt-virt:sajib011712.mp3|mp3}} * 10/25/11: Discussing VM sizing section draft {{{{internal:projects:res-mgmt-virt:sajib102511.mp3|mp3}} * 10/18/11: VM sizing updates {{{{internal:projects:res-mgmt-virt:sajib101811.mp3|mp3}} * 10/11/11: Initial report on VM sizing and plans for expansion {{{{internal:projects:res-mgmt-virt:sajib101111.mp3|mp3}} * 09/13/11: Updates on paper writing {{{{internal:projects:res-mgmt-virt:sajib091311.mp3|mp3}} * 09/06/11: Many ideas for improvement of the paper writing -- and task division {{{{internal:projects:res-mgmt-virt:sajib090611.mp3|mp3}} * 08/30/11: Updated related work table and Follow-up task list {{{{internal:projects:res-mgmt-virt:sajib083011.mp3|mp3}} * 08/23/11: Revisiting paper message and structure {{{{internal:projects:res-mgmt-virt:sajib082311.mp3|mp3}} * 08/16/11: Revising based on SVM contributions {{{{internal:projects:res-mgmt-virt:sajib081611.mp3|mp3}} * 08/02/11: More noise results, robustness of ann parameter tuning, initial svm results {{{{internal:projects:res-mgmt-virt:sajib080211.mp3|mp3}} * 07/26/11: Including SVM evaluation, framework discussion {{{{internal:projects:res-mgmt-virt:sajib072611.mp3|mp3}} * 07/19/11: Improving related work and re-submission venues {{{{internal:projects:res-mgmt-virt:sajib071911.mp3|mp3}} * 07/12/11: SOCC reviews and framework discussion {{{{internal:projects:res-mgmt-virt:sajib071211.mp3|mp3}} * 06/07/11: Evaluation design {{{{internal:projects:res-mgmt-virt:sajib060711.mp3|mp3}} * 05/31/11: Reviewing scope document, scenarios {{{{internal:projects:res-mgmt-virt:sajib053111.mp3|mp3}} * 05/23/11: New directions -- revenue maximization or online modeling? {{{{internal:projects:res-mgmt-virt:sajib052311.mp3|mp3}} * 04/05/11: SOCC eval improvements {{{{internal:projects:res-mgmt-virt:sajib040511.mp3|mp3}} * 03/31/11: SOCC updates, eval and editing plans {{{{internal:projects:res-mgmt-virt:sajib033111.mp3|mp3}} * 03/24/11: Intro, reminders in 3&4, eval improvements {{{{internal:projects:res-mgmt-virt:sajib032411.mp3|mp3}} * 03/17/11: Review and Plans for submission to SOCC {{{{internal:projects:res-mgmt-virt:sajib031711.mp3|mp3}} * 03/15/11: SOSP paper: Detailed review of eval section and writing plans {{{{internal:projects:res-mgmt-virt:sajib031511.mp3|mp3}} * 03/10/11: SOSP paper: Reviewing comments, Section 3 message, Related work {{{{internal:projects:res-mgmt-virt:sajib031011.mp3|mp3}} * 03/03/11: Results update and writing plans for SOSP {{{{internal:projects:res-mgmt-virt:sajib030311.mp3|mp3}} * 02/24/11: Various modeling approaches, evaluating them and telling the story, plans for SOSP writing {{{{internal:projects:res-mgmt-virt:sajib022411.mp3|mp3}} * 02/10/11: Updates on accuracy of sub-modeling approach; need for formalization identified {{{{internal:projects:res-mgmt-virt:sajib021011.mp3|mp3}} * 02/03/11: Clustering findings and initial thoughts on formalizing the training process using sub-models {{{{internal:projects:res-mgmt-virt:sajib020311.mp3|mp3}} * 01/27/11: Sub-model initial findings and next steps on paper writing and eval {{{{internal:projects:res-mgmt-virt:sajib012711.mp3|mp3}} * 01/20/11: Status update, SOSP paper discussion, modeling enhancements {{{{internal:projects:res-mgmt-virt:sajib012011.mp3|mp3}} * 12/02/10: RuBiS benchmark is set up, next steps discussed with Ajay (no audio) {{{{internal:projects:res-mgmt-virt:sajib120210.mp3|mp3}} * 11/18/10: Understanding the ANN mechanisms and alternate implementations {{{{internal:projects:res-mgmt-virt:sajib111810.mp3|mp3}} * 11/04/10: Postmark initial data makes sense and more data being collected, 4-disk RAID0 for final evaluation {{{{internal:projects:res-mgmt-virt:sajib110410.mp3|mp3}} * 10/28/10: Analyzing initial data from Postmark, VMFS caching issue identified {{{{internal:projects:res-mgmt-virt:sajib102810.mp3|mp3}} * 10/19/10: Status update and planning {{{{internal:projects:res-mgmt-virt:sajib101910.mp3|mp3}} * 09/20/10: Resuming meeting --- USENIX submission {{{{internal:projects:res-mgmt-virt:sajib092010.mp3|mp3}} * 05/03/10: Paper context and automating model creation {{{{internal:projects:res-mgmt-virt:sajib050310.mp3|mp3}} * 04/29/10: Sysbench accuracy improvement and modified data collection strategy {{{{internal:projects:res-mgmt-virt:sajib042910.mp3|mp3}} * 04/26/10: Modeling accuracy issues and new data collection plans {{{{internal:projects:res-mgmt-virt:sajib042610.mp3|mp3}} * 04/19/10: Initial meeting with Ajay and evaluation plan review and refinement {{{{internal:projects:res-mgmt-virt:sajib041910.mp3|mp3}} * 04/15/10: Discussion of Ajay's initial feedback {{{{internal:projects:res-mgmt-virt:sajib041510.mp3|mp3}} * 04/12/10: Paper writing steps {{{{internal:projects:res-mgmt-virt:sajib041210.mp3|mp3}} * 04/02/10: Revisiting plans for OSDI experiments {{{{internal:projects:res-mgmt-virt:sajib040210.mp3|mp3}} * 03/29/10: Plans for OSDI experiments {{{{internal:projects:res-mgmt-virt:sajib032910.mp3|mp3}} * 02/16/10: Debugging data {{{{internal:projects:res-mgmt-virt:sajib021610.mp3|mp3}} * 01/28/10: IEEE Micro magazine submission plan {{{{internal:projects:res-mgmt-virt:sajib012810.mp3|mp3}} * 12/21/09: Alternate submission venues {{{{internal:projects:res-mgmt-virt:sajib122109.mp3|mp3}} * 12/17/09: Implementation roadblock resolution {{{{internal:projects:res-mgmt-virt:sajib121709.mp3|mp3}} * 12/14/09: The Brute force algorithm for resource reallocation {{{{internal:projects:res-mgmt-virt:sajib121409.mp3|mp3}} * 12/07/09: Update on UTC paper - evaluation plan {{{{internal:projects:res-mgmt-virt:sajib120709.mp3|mp3}} * 12/03/09: Brief HPCA slides review and UTC experiment graphs {{{{internal:projects:res-mgmt-virt:sajib113009.mp3|mp3}} * 11/30/09: HPCA slides review {{{{internal:projects:res-mgmt-virt:sajib113009.mp3|mp3}} * 11/25/09: HPCA presentation, discussion of brute force algo implementation {{{{internal:projects:res-mgmt-virt:sajib112509.mp3|mp3}} * 11/19/09: USENIX paper contents, initial experiments {{{{internal:projects:res-mgmt-virt:sajib111909.mp3|mp3}} * 11/16/09: Possible directions for USENIX paper {{{{internal:projects:res-mgmt-virt:sajib111609.mp3|mp3}} * 11/12/09: 3 Possible directions for model updating {{{{internal:projects:res-mgmt-virt:sajib111209.mp3|mp3}} * 11/10/09: Last minute comments on Workshop paper - Algo modifications {{{{internal:projects:res-mgmt-virt:sajib111009.mp3|mp3}} * 11/09/09: Workshop paper content, NP completeness, mapping section, algo, etc. {{{{internal:projects:res-mgmt-virt:sajib110909.mp3|mp3}} * 11/05/09: Workshop paper structure and next steps {{{{internal:projects:res-mgmt-virt:sajib110509.mp3|mp3}} * 11/02/09: HPCA camera-ready and Workshop paper tasks {{{{internal:projects:res-mgmt-virt:sajib110209.mp3|mp3}} * 10/29/09: HPCA camera ready paper - experiments and content condensing {{{{internal:projects:res-mgmt-virt:sajib102909.mp3|mp3}} * 10/26/09: Workshop paper revised structure {{{{internal:projects:res-mgmt-virt:sajib102609.mp3|mp3}} * 10/22/09: Workshop paper discussion {{{{internal:projects:res-mgmt-virt:sajib102209.mp3|mp3}} * 10/19/09: HPCA Workshop paper contributions and structure {{{{internal:projects:res-mgmt-virt:sajib101909.mp3|mp3}} * 10/15/09: HPCA Workshop paper ideas and final conference camera-ready {{{{internal:projects:res-mgmt-virt:sajib101509.mp3|mp3}} * 10/12/09: Incremental delta-assignment algorithm issues and alternative exhaustive algorithm analysis{{{{internal:projects:res-mgmt-virt:sajib101209.mp3|mp3}} * 10/08/09: Incremental delta-assignment algorithm and discussion on optimizations {{{{internal:projects:res-mgmt-virt:sajib100809.mp3|mp3}} * 10/05/09: More discussion on resource redistribution algorithm {{{{internal:projects:res-mgmt-virt:sajib100509.mp3|mp3}} * 10/02/09: Details of algorithm and knapsack problem requirements {{{{internal:projects:res-mgmt-virt:sajib100209.mp3|mp3}} * 09/28/09: Incremental resource reallocation and mapping to knapsack {{{{internal:projects:res-mgmt-virt:sajib092809.mp3|mp3}} * 09/24/09: HPCA10 Rebuttal {{{{internal:projects:res-mgmt-virt:sajib092409.mp3|mp3}} * 09/11/09: Graphs and message, New Pt options, and Knapsack problem mapping {{{{internal:projects:res-mgmt-virt:sajib091109.mp3|mp3}} * 09/04/09: Problem formulation - feedback from Kaushik, Ming {{{{internal:projects:res-mgmt-virt:sajib090409.mp3|mp3}} * 08/24/09: Overall revenue maximization procedure - detailed discussion {{{{internal:projects:res-mgmt-virt:sajib082409.mp3|mp3}} * 08/17/09: Eurosys08 paper review, SLA curve, challenges discussion {{{{internal:projects:res-mgmt-virt:sajib081709.mp3|mp3}} * 08/03/09: Initial thoughts on online model refinement, benchmarks identified {{{{internal:projects:res-mgmt-virt:sajib080309.mp3|mp3}} * 07/30/09: Final discussions for hpca10 submission {{{{internal:projects:res-mgmt-virt:sajib073009.mp3|mp3}} * 07/27/09: Training models automatically and task assignment for paper {{{{internal:projects:res-mgmt-virt:sajib072709.mp3|mp3}} * 07/20/09: HPCA paper content outline {{{{internal:projects:res-mgmt-virt:sajib072009.mp3|mp3}} * 07/16/09: Discussion of middleware reviews and plans to address them {{{{internal:projects:res-mgmt-virt:sajib071609.mp3|mp3}} * 07/06/09: Debugging memory heatmap and structure of experiments section {{{{internal:projects:res-mgmt-virt:sajib070609.mp3|mp3}} * 07/02/09: Debugging benchmark behavior and error heatmaps {{{{internal:projects:res-mgmt-virt:sajib070209.mp3|mp3}} * 06/22/09: Results are acceptable. Next 3 steps discussion - insights on Elliot, other models, and new hybrid model? {{{{internal:projects:res-mgmt-virt:sajib062209.mp3|mp3}} * 06/18/09: Debugging ANN models and methodology {{{{internal:projects:res-mgmt-virt:sajib061809.mp3|mp3}} * 06/15/09: ANN advanced configuration and R models {{{{internal:projects:res-mgmt-virt:sajib061509.mp3|mp3}} * 06/09/09: Modeling options discussion {{{{internal:projects:res-mgmt-virt:sajib060909.mp3|mp3}} * 04/23/09: Paper tasks - experiments update, architecture figure, writing tasks {{{{internal:projects:res-mgmt-virt:sajib042309.mp3|mp3}} * 04/09/09: Designing graphs and Setting up Task-List related to Paper {{{{internal:projects:res-mgmt-virt:sajib040909.mp3|mp3}} * 04/02/09: Memory intensive benchmark requirements, graphs design, paper writing tasks {{{{internal:projects:res-mgmt-virt:sajib040209.mp3|mp3}} * 03/16/09: Sysbench output debugging. Next steps - explain sysbench output clearly, pilot other sysbench benchmarks, model oltp {{{{internal:projects:res-mgmt-virt:sajib031609.mp3|mp3}} * 03/11/09: Problems with Xen install, Benchmark identified (sysbench) {{{{internal:projects:res-mgmt-virt:sajib031109.mp3|mp3}} * 02/17/09: Middleware paper positioning and discussion of related work {{{{internal:projects:res-mgmt-virt:sajib021709.mp3|mp3}} * 02/10/09: Revisiting assumptions, dynamic model updates, enumeration of modeling techniques {{{{internal:projects:res-mgmt-virt:sajib021009.mp3|mp3}} * 02/03/09: Refining and broadening the paper direction {{{{internal:projects:res-mgmt-virt:sajib020309.mp3|mp3}} ===== Design ===== Sajib: Fill this in.