Performance Evaluation and Benchmarking for the Era of Artificial Intelligence

1. Front Matter

   Pages I-XIII

   PDF

2. Industry Panel on Defining Industry Standards for Benchmarking Artificial Intelligence

   • Raghunath Nambiar, Shahram Ghandeharizadeh, Gary Little, Christoph Boden, Ajay Dholakia

   Pages 1-6

3. UniBench: A Benchmark for Multi-model Database Management Systems

   • Chao Zhang, Jiaheng Lu, Pengfei Xu, Yuxing Chen

   Pages 7-23

4. PolyBench: The First Benchmark for Polystores

   • Jeyhun Karimov, Tilmann Rabl, Volker Markl

   Pages 24-41

5. Benchmarking Distributed Data Processing Systems for Machine Learning Workloads

   • Christoph Boden, Tilmann Rabl, Sebastian Schelter, Volker Markl

   Pages 42-57

6. Characterizing the Performance and Resilience of HCI Clusters with the TPCx-HCI Benchmark

   • H. Reza Taheri, Gary Little, Bhavik Desai, Andrew Bond, Doug Johnson, Greg Kopczynski

   Pages 58-70

7. Requirements for an Enterprise AI Benchmark

   • Cedric Bourrasset, France Boillod-Cerneux, Ludovic Sauge, Myrtille Deldossi, Francois Wellenreiter, Rajesh Bordawekar et al.

   Pages 71-81

8. Towards Evaluation of Tensorflow Performance in a Distributed Compute Environment

   • Miro Hodak, Ajay Dholakia

   Pages 82-93

9. A Comparison of Two Cache Augmented SQL Architectures

   • Shahram Ghandeharizadeh, Hieu Nguyen

   Pages 94-109

10. Benchmarking and Performance Analysis of Event Sequence Queries on Relational Database

    • Yuto Hayamizu, Ryoji Kawamichi, Kazuo Goda, Masaru Kitsuregawa

    Pages 110-125

11. Data Consistency Properties of Document Store as a Service (DSaaS): Using MongoDB Atlas as an Example

    • Chenhao Huang, Michael Cahill, Alan Fekete, Uwe Röhm

    Pages 126-139

12. Lessons Learned from the Industry’s First TPC Benchmark DS (TPC-DS)

    • Manan Trivedi, Zhenqiang Chen

    Pages 140-154

13. Back Matter

    Pages 155-155

    PDF

This book constitutes the thoroughly refereed post-conference proceedings of the 10th TPC Technology Conference on Performance Evaluation and Benchmarking, TPCTC 2018, held in conjunction with the 44th International Conference on Very Large Databases (VLDB 2018) in August 2018.

The 10 papers presented were carefully reviewed and selected from numerous submissions.

The TPC encourages researchers and industry experts to present and debate novel ideas and methodologies in performance evaluation, measurement, and characterization.

• artificial intelligence
• benchmarks
• computer architecture
• database management systems
• database performance
• databases
• decision support systems
• hardware
• industry standards
• information storage and retrieval
• internet
• optimization
• performance engineering
• performance evaluation
• query languages
• relational database
• software engineering
• World Wide Web

• Advanced Micro Systems, Inc., Santa Clara, USA

  Raghunath Nambiar

• Oracle Corporation, Redwood Shores, USA

  Meikel Poess

Policies and ethics