Secrets of YARN application development

Scale
05/26/2014 - 15:30 to 16:10
Frannz Club
long talk (40 min)

Session abstract: 

 

Learning from the mistakes of others is better than learning from your own -this talk lets anyone writing a Hadoop YARN application learn from mine. Client, Application Master, and worker design, RPC service interfaces, secure operation, failure-handling and test strategies, are all key issues you need to get right -the key points being "Model-View-Controller" is still a good architecture, while mock tests are the secret to testing that model.

 

This talk tells people writing YARN applications what they need to know -to help the build YARN applications that can work with all the data waiting for them in Hadoop clusters

Core Concepts

  • Why write a YARN application?

  • Service architecture: your app as an aggregation of services

  • Algorithms: scalability and resilience

  • Separating Model from service

  • Container launch contexts: configuring, debugging

  • Preflight checks -what & why

  • Testability through RPC

  • Testing: Mock, Mini vs real YARN clusters

Video: 

Slide: