Bursting HPC to AWS Spot instances

Bursting HPC to AWS Spot instances

AWS Spot instances is one the most advanced features in cloud computing and very difficult to take advantage. The idea is to bid for unused CPU on AWS at a lower rate than standard and use them when required. I am just exploring how to setup a HPC infrastructure that can burst into AWS and AWS Spot instances.

Cluster Management Software

Research

  • http://aws.amazon.com/hpc/getting-started/
  • https://gigaom.com/2013/10/08/bidding-strategies-arbitrage-aws-spot-market-is-where-computing-and-finance-meet/
  • http://blog.diffbot.com/setting-up-a-machine-learning-farm-in-the-cloud-with-spot-instances-auto-scaling/
  • http://www.appneta.com/blog/aws-spot-instances/
  • http://scn.sap.com/community/developer-center/hana/blog/2013/11/11/using-aws-spot-instances-with-sap-cal
  • http://sourceforge.net/p/xcat/wiki/Main_Page/
  • http://www-03.ibm.com/systems/power/hardware/benchmarks/hpc.html
  • http://star.mit.edu/cluster/
  • https://vimeo.com/27993626
  • http://area.autodesk.com/blogs/cory/setting-up-an-amazon-ec2-render-farm-with-backburner
  • http://www.thinkboxsoftware.com/deadline/
  • https://www.youtube.com/watch?v=pmM_8hsJt0E
  • https://www.youtube.com/watch?v=Q1SKzK3E_jE
  • https://www.youtube.com/watch?v=HwmXR-u3zXY
  • http://www.luxrender.net/forum/viewtopic.php?f=13&t=8584
  • https://www.youtube.com/watch?v=pmM_8hsJt0E
  • https://render.st/features/
  • http://www.thinkboxsoftware.com/deadline/

Leave a comment