====== Direct KV ====== ===== Participants ===== * Leonardo Marmol * Swaminathan Sundararaman * Nisha Talagala * [[http://www.cs.fiu.edu/~raju/|Raju Rangaswami]] ===== Project Goals ===== Direct KV store ===== Notes on Performance ===== * Writes must have a latency of at least 20 ms * Reads must have a latency of at most 55 ms ===== Evaluation Results ===== Using NVMKV own benchmarking tool we obtained the following results. This tools generates a randomized workload in which every operation is performed on a different key. {{:internal:projects:directkv:cache_data_nolocks.png?1000|}} Raw data can be found here {{:internal:projects:directkv:cache_data_nolocks.pdf|}} === Write Amplification === * [[internal:projects:directkv:write_amp|Write Amplification]] === YCSB === * [[internal:projects:directkv:ycsb|YCSB Results]] Modification to the workloads * fieldcount = 1 (default 10) * fieldlength = 1024 (default 100) * recordcount = 1000000 (default 1000) * operationcount = 1000000 (default 1000) * Total Data Size = 976.56 MB Core Properties: https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties Workloads Descriptions: https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads ^ Workload ^ RunTime(ms) ^ Throughput(ops/sec) ^ AvgLat(us) ^ MinLat(us) ^ MaxLat(us) ^ Cache Size ^ Cache Stats (ios, hits, evic) ^ | a | 314016.0 | 3184.55 | 333.48 | 105 | 22067 | 0 | n/a | | b | 242333.0 | 4126.55 | 306.52 | 113 | 3220 | 0 | n/a | | c | 227708.0 | 4391.58 | 215.16 | 80 | 46375 | 0 | n/a | | d | 273213.0 | 3660.14 | 152.77 | 43 | 3190 | 0 | n/a | | e | 10148.0 | 98541.58 | 103.17 | 55 | 3014 | 0 | n/a | | f | 430920.0 | 2320.61 | 295.07 | 105 | 4478 | 0 | n/a | |-|||||||| | a | 227685.0 | 4392.03 | 372.39 | 109 | 17372 | 256 | 501047, 501047,236809 | | b | 64438.0 | 15518.79 | 403.26 | 131 | 3305 | 256 | 950246, 950245, 0 | | c | 14111.0 | 70866.69 | 11.51 | 10 | 29745 | 256 | 1000000, 999999, 0 | | d | 18858.0 | 91591.86 | 111.72 | 58 | 3103 | 256 | 950258, 950257, 0 | | e | 10918.0 | 53027.89 | 70.06 | 41 | 1358 | 256 | 0,0,0 | | f | 254522.0 | 3928.93 | 367.99 | 107 | 23322 | 256 | 1000000, 999999, 237210 | Raw data can be found here: http://gladiator.cs.fiu.edu/ycsb_workloads/ ===== LevelDB ===== LevelDB Options File: https://code.google.com/p/leveldb/source/browse/include/leveldb/options.h Official information about LevelDB http://leveldb.googlecode.com/svn/trunk/doc/index.html General information about LevelDB as well and tips and tricks about tuning http://docs.basho.com/riak/latest/ops/advanced/backends/leveldb/ YCSB/Mapkeeper: https://github.com/m1ch1/mapkeeper/tree/master/leveldb According to these benchmarks ran with Mapkeeper, WiredTiger outperforms LevelDB in every test. Details here https://github.com/wiredtiger/wiredtiger/wiki/YCSB-Mapkeeper-benchmark ===== Issues ===== * When running YCSB, for every workload I get the following error message. This was solved by applying a patch. Error, atomic write failed: 22 The output of ''dmesg'' shows the following ... [962679.776274] <6>fioinf Fusion-io 825GB ioScale2 0000:05:00.0: Attach succeeded. [962680.107683] <6>fioinf Fusion-io 825GB ioScale2 0000:05:00.0: atomic write too large, nr_pages = 4224 max 1088) * The cache is not printing stats at the end of a run. This was caused by the helper library not calling ''nvm_kv_close''. It was fixed with the following change diff --git a/helper/src/main/fio_kv_helper.cpp b/helper/src/main/fio_kv_helper.cpp index 7cb7a5d..56ca52f 100644 --- a/helper/src/main/fio_kv_helper.cpp +++ b/helper/src/main/fio_kv_helper.cpp @@ -127,6 +127,8 @@ void fio_kv_close(fio_kv_store_t *store) assert(store != NULL); assert(store->kv > 0); + nvm_kv_close(store->kv); + if (store->fd) { fsync(store->fd); close(store->fd); ===== Meetings ===== * 05/05/15: Final version comments discussion with Swami / Nisha {{{{internal:projects:directkv:directkv050515.mp3|mp3}} * 04/24/15: ATC Comments and plans for final version / shepherding {{{{internal:projects:directkv:directkv042415.mp3|mp3}} * 03/20/15: NVMKV bug & Future directions for NVM research {{{{internal:projects:directkv:directkv032015.mp3|mp3}} * 03/03/15: Parallelism bug with NVMKV {{{{internal:projects:directkv:directkv030315.mp3|mp3}} * 02/06/15: Parallelism bug with NVMKV {{{{internal:projects:directkv:directkv020615.mp3|mp3}} * 01/30/15: TBD {{{{internal:projects:directkv:directkv013015.mp3|mp3}} * 01/27/15: TBD {{{{internal:projects:directkv:directkv012715.mp3|mp3}} * 12/16/14: Next steps with ioDrive2 and paper writing {{{{internal:projects:directkv:directkv121614.mp3|mp3}} * 12/16/14: Next steps with ioDrive2 and paper writing {{{{internal:projects:directkv:directkv121614.mp3|mp3}} * 12/09/14: New machine data (needs more validation); plans for updating testbed with ioDrive2 {{{{internal:projects:directkv:directkv120914.mp3|mp3}} * 12/02/14: New machine data (doesn't look right) {{{{internal:projects:directkv:directkv120214.mp3|mp3}} * 10/30/14: Writing tasks {{{{internal:projects:directkv:directkv103014.mp3|mp3}} * 10/14/14: Eval/writing next steps {{{{internal:projects:directkv:directkv101414.mp3|mp3}} * 10/07/14: Plans moving forward without Eurosys submission {{{{internal:projects:directkv:directkv100714.mp3|mp3}} * 10/03/14: Paper submission plans and writing tasks for Eurosys'15 {{{{internal:projects:directkv:directkv100314.mp3|mp3}} * 09/17/14: Reviewing the current outline and paper writeup; next steps for writing {{{{internal:projects:directkv:directkv091714.mp3|mp3}} * 09/16/14: Outline/data discussion {{{{internal:projects:directkv:directkv091614.mp3|mp3}} * 09/10/14: Writing plans {{{{internal:projects:directkv:directkv091014.mp3|mp3}} * 09/05/14: Evaluation plans; writing plans {{{{internal:projects:directkv:directkv090514.mp3|mp3}} * 09/02/14: Plans for paper {{{{internal:projects:directkv:directkv090214.mp3|mp3}} * 08/28/14: Two papers instead of one? {{{{internal:projects:directkv:directkv082814.mp3|mp3}} * 08/19/14: Trade-offs detailed discussion and new plans for selling FAST paper {{{{internal:projects:directkv:directkv081914.mp3|mp3}} * 08/12/14: Selling FAST'15 paper as a trade-offs paper {{{{internal:projects:directkv:directkv081214.mp3|mp3}} * 08/05/14: FAST'15 paper writing: new direction to include memory/flash hybrid KV tradeoffs for cost/performance/energy {{{{internal:projects:directkv:directkv080514.mp3|mp3}} * 06/24/14: Feedback for FAST submission and plans {{{{internal:projects:directkv:directkv062414.mp3|mp3}} * 05/06/14: HotStorage'14 camera-ready final touches {{{{internal:projects:directkv:directkv050614.mp3|mp3}} * 04/29/14: HotStorage'14 camera-ready and FAST submission summer plan {{{{internal:projects:directkv:directkv042914.mp3|mp3}} * 04/15/14: HotStorage'14 reviewer feedback discussion {{{{internal:projects:directkv:directkv041514.mp3|mp3}} * 04/08/14: Initial multi-threading data for YCSB {{{{internal:projects:directkv:directkv040814.mp3|mp3}} * 04/01/14: FAST paper brainstorming {{{{internal:projects:directkv:directkv040114.mp3|mp3}} * 03/25/14: FAST paper -- initial thoughts {{{{internal:projects:directkv:directkv032514.mp3|mp3}} * 03/11/14: HotStorage evaluation data and Writing {{{{internal:projects:directkv:directkv031114.mp3|mp3}} * 03/04/14: Plans for HotStorage submission {{{{internal:projects:directkv:directkv030414.mp3|mp3}} * 02/25/14: ext2/ext4 do *worse* with LevelDB; HotStorage submission discussion {{{{internal:projects:directkv:directkv022514.mp3|mp3}} * 02/11/14: Updates on level-db write amplification experiments {{{{internal:projects:directkv:directkv021114.mp3|mp3}} * 01/14/14: Time-series plots {{{{internal:projects:directkv:directkv011414.mp3|mp3}} * 01/06/14: Next steps in data collection {{{{internal:projects:directkv:directkv010614.mp3|mp3}} * 12/23/13: Analyzing new LevelDB and DirectKV-with-cache data; and plans for submission {{{{internal:projects:directkv:directkv122313.mp3|mp3}} * 11/15/13: Directkv / driver issues; Leveldb cache design discussion {{{{internal:projects:directkv:directkv111513.mp3|mp3}} * 11/04/13: Initial cache numbers {{{{internal:projects:directkv:directkv110413.mp3|mp3}} * 10/28/13: Updates and next steps for cache implementation {{{{internal:projects:directkv:directkv102813.mp3|mp3}} * 10/21/13: DirectKV vs. LevelDB evaluation configurations, implementing write-through caching in DirectKV {{{{internal:projects:directkv:directkv102113.mp3|mp3}} * 10/14/13: DirectKV vs. LevelDB (initial data), Alternative comparison setups, Importance of Write-back caching, {{{{internal:projects:directkv:directkv101413.mp3|mp3}}