Clairvoyance: Resource Measurement and Prediction for Distributed Interactive Applications

The goal of this project is to develop techniques and technology to better measure and predict network and host behavior as it is visible to and affects adaptive applications, particularly distributed interactive applications. This project has large wings and ties to various projects under the Prescience Lab as well as to the Remos project at CMU.

People

  • Peter Dinda
  • Jason Skicewicz
  • Yi Qiao
  • Dong Lu
  • Ashish Gupta
  • Bin Lin
  • Sam Rossoff
  • Jay Bruins

    • Movies

  • RPS Demo (AVI, 98 MB)

    • Talks

    These are the most relevant talks since 1998. You may also find additional talks.
  • Time-sharing Parallel Applications With Performance Isolation And Control, (ICAC 2007), (pdf)
  • Power Reduction Through Measurement and Modeling of Users and CPUs: Summary, (SIGMETRICS 2007) (powerpoint poster)
  • The User In Experimental Computer Systems Research, (ExpCS 2007) (powerpoint)
  • Towards Scheduling Virtual Machines Based On Direct User Input, (VTDC 2006) (pdf slides)
  • Characterizing and Predicting TCP Throughput on the Wide Area Network (ICDCS 2005) (ppt)
  • Modeling and Taming Parallel TCP on the Wide Area Network (IPDPS 2005) (ppt)
  • Looking at the Server-side of P2P Systems (LCR 2004) (ppt)
  • Size-based Scheduling Policies with Inaccurate Scheduling Information (MASCOTS 2004) (ppt)
  • Applications of SRPT Scheduling with Inaccurate Information (MASCOTS 2004) (ppt poster)
  • An Empirical Study of the Multiscale Predictability of Network Traffic (HPDC 2004) (ppt)
  • A Prediction-based Real-time Scheduling Advisor (IPDPS 2002) (ppt)
  • The Measured Network Traffic of Compiler-parallelized Programs (ICPP 2001) (pdf)
  • Online Prediction of the Running Time of Tasks (HPDC 2001) (ppt)
  • Online Prediction of the Running Time of Tasks: Summary (SIGMETRICS 2001) (ppt)
  • Realistic CPU Workloads Through Host Load Trace Playback (LCR 2000) (ppt)
  • An Evaluation of Linear Models for Host Load Prediction (HPDC 99) (ppt)
  • The Case for Prediction-based Best-effort Real-time (WPDRTS 99) (ppt)
  • Understanding and Predicting Host Load (LCR98) (ppt)

    • Selected Papers

    These are the most relevant papers since 1999. You may also find additional papers.
  • B. Lin, Human-driven Optimization, Doctoral Dissertation, Technical Report NWU-EECS-07-04, Department of Electrical Engineering and Computer Science, Northwestern University, July, 2007. pdf

  • B. Lin, A. Sundararaj, P. Dinda, Time-sharing Parallel Applications With Performance Isolation And Control, Proceedings of the 4th IEEE International Conference on Autonomic Computing (ICAC 2007), June, 2007. pdf

  • P. Dinda, G. Memik, R. Dick, B. Lin, A. Mallik, A. Gupta, S. Rossoff, The User In Experimental Computer Systems Research, Proceedings of the Workshop on Experimental Computer Science (ExpCS 2007), June, 2007. pdf

  • B. Lin, A. Mallik, P. Dinda, G. Memik, R. Dick, Power Reduction Through Measurement and Modeling of Users and CPUs: Summary, Proceedings of ACM SIGMETRICS 2007, June, 2007. pdf

  • A. Mallik, B. Lin, G. Memik, P. Dinda, and R. Dick, User-Driven Frequency Scaling, Computer Architecture Letters, Volume 5, Number 2, July-December, 2006. pdf.

  • B. Lin, P. Dinda,Towards Scheduling Virtual Machines Based On Direct User Input, Proceedings of the 1st International Workshop on Virtualization Technology in Distributed Computing (VTDC 2006), November, 2006, pdf

  • A. Mallik, B. Lin, P. Dinda, G. Memik, and R. Dick, Process and User Driven Dynamic Voltage and Frequency Scaling, Technical Report NWU-EECS-06-11, Department of Electrical Engineering and Computer Science, Northwestern University, August, 2006, pdf.

  • S. Rossoff, and P. Dinda, Prospects for Speculative Remote Display, Technical Report NWU-EECS-06-08, Department of Electrical Engineering and Computer Science, Northwestern University, August, 2006, pdf.

  • B. Lin, and P. Dinda, Putting the User in Direct Control of CPU Scheduling, Technical Report NWU-EECS-06-07, Department of Electrical Engineering and Computer Science, Northwestern University, July, 2006. pdf.

  • P. Dinda, Design, Implementation, and Performance of an Extensible Toolkit for Resource Prediction In Distributed Systems, IEEE Transactions on Parallel and Distributed Systems, Volume 17, Number 2, February, 2006.

  • D. Lu, P. Dinda, Y. Qiao, H. Sheng, Effects and Implications of File Size/Service Time Correlation on Web Server Scheduling Policies, Proceedings of the 13th IEEE/ACM International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2005), September, 2005. pdf

  • D. Lu, Y. Qiao, P. Dinda, and F. Bustamante, Characterizing and Predicting TCP Throughput on the Wide Area Network, Proceedings of the 25th International Conference on Distributed Computing (ICDCS 2005). pdf

  • D. Lu, Y. Qiao, P. Dinda, and F. Bustamante, Modeling and Taming Parallel TCP on the Wide Area Network, Proceedings of the 19th International Parallel and Distributed Processing Symposium (IPDPS 2005). pdf

  • Y. Qiao, D. Lu, F. Bustamante, P. Dinda, Looking at the Server Side of Peer-to-Peer Systems, Proceedings of the Seventh Workshop on Langauges, Compilers and Run-time Support for Scalable Systems (LCR 2004). pdf

  • D. Lu, H. Sheng, P. Dinda, Size-based Scheduling Policies With Inaccurate Scheduling Information, Proceedings of the 12th IEEE/ACM International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2004), pdf

  • D. Lu, Y. Qiao, P. Dinda, and F. Bustamante, Modeling and Taming Parallel TCP on the Wide Area Network, Technical Report NWU-CS-04-35, May, 2004, pdf.

  • A. Gupta, B. Lin, P. Dinda, Measuring And Understanding User Comfort With Resource Borrowing, Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing (HPDC 2004), pdf

  • Y. Qiao, J. Skicewicz, P. Dinda, An Empirical Study of the Multiscale Predictability of Network Traffic, Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing (HPDC 2004), pdf

  • D. Lu., H. Sheng, and P. Dinda, Effects and Implications of File Size/Service Time Correlation on Web Server Scheduling Policies, Technical Report NWU-CS-04-33, Department of Computer Science, Northwestern University, April, 2004. pdf.

  • D. Lu, Y. Qiao, P. Dinda, and F. Bustamante, Characterizing and Predicting TCP Throughput on the Wide Area Network, Technical Report NWU-CS-04-34, Department of Computer Science, Northwestern University, April, 2004. pdf.

  • A. Gupta, B. Lin, and P. Dinda, A Framework and Toolkit for Understanding User Comfort with Resource Borrowing, Technical Report NWU-CS-04-28, Department of Computer Science, Northwestern University, February, 2004. pdf.

  • Y. Qiao, J. Skicewicz, P. Dinda, Multiscale Predictability of Network Traffic, Technical Report NWU-CS-02-13, Department of Computer Science, Northwestern University, October, 2002. (pdf). Revised, March 2004 (pdf)

  • Y. Qiao, and P. Dinda, Network Traffic Analysis, Classification, and Prediction, Technical Report NWU-CS-02-11, Department of Computer Science, Northwestern University, January, 2003. pdf.

  • P. Dinda, Online Prediction of the Running Time of Tasks, Cluster Computing, To appear in 2002. (This is the journal version of the SIGMETRICS and HPDC papers below)

  • P. Dinda, A Prediction-based Real-time Scheduling Advisor, Proceedings of the 16th International Parallel and Distributed Processing Symposium (IPDPS 2002). (pdf)

  • P. Dinda, Online Prediction of the Running Time of Tasks, Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing (HPDC 2001). (pdf)

  • P. Dinda, B. Garcia, K. Leung, The Measured Network Traffic of Compiler-Parallelized Programs, Proceedings of the 30th International Conference on Parallel Processing (ICPP 2001). (pdf)

  • P. Dinda, Online Prediction of the Running Time of Tasks: Summary, Proceedings of the ACM SIGMETRICS 2001 Conference on Measurement and Modeling of Computer Systems (SIGMETRICS 2001). (pdf)

  • P. Dinda, D. O'Hallaron, Host Load Prediction Using Linear Models , Cluster Computing, Volume 3, Number 4, 2000. (This is the journal version of the HPDC '99 paper. It describ es the study in more detail and gives performace data for the on-line host load prediction system.) (pdf)

  • P. Dinda, D. O'Hallaron, Realistic CPU Workloads Through Host Load Trace Playback, Proceedings of the 5th Workshop on Languages, Compilers, and Run-time Systems for Scalable Computers, Rochester, NY, May, 2000, Springer LNCS 1915, pp. 265-280. (postscript)

  • P. Dinda, The Statistical Properties of Host Load , Scientific Programming, 7:3-4, Fall, 1999. (This is a vastly extended version of the study published in LCR 98 (below) and presents results for a second set of machines. This is also available below as CMU-CS-98-175)

  • P. Dinda, D. O'Hallaron, An Evaluation of Linear Models for Host Load Prediction, Proceedings of the 8th IEEE International Symposium on High-Performance Distributed Computing (HPDC-8) Aug 1999, Redondo Beach, CA. (pdf).

  • P. Dinda, B. Lowekamp, L. Kallivokas, D. O'Hallaron, The Case for Prediction-based Best-effort Real-time Systems, Proceedings of the 7th International Workshop on Parallel and Distributed Real-Time Systems (WPDRTS 1999), San Juan, April, 1999, pp. 309--318 (postscript)

    • Codes

  • The Prescience Lab, and, particularly, RPS, include the tools developed by this project.
  • TameParallelTCP: Estimate throughput and impact of parallel TCP flows

    • Acknowledgement

    This material is based upon work supported by the National Science Foundation under Grant No. ANI-0093221. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).
    Peter Dinda
    Last modified: Thu Jul 26 17:08:56 CDT 2007