Clever Geek Handbook

Integration of Kubernetes with yarn

Do you know how to integrate Kubernetes with an existing yarn cluster?

Below is information about the features that Kubernetes yarn integration offers.

https://hortonworks.com/blog/docker-kubernetes-apache-hadoop-yarn/

However, the tutorial below provides instructions for setting up a local cluster.

Alternative or second question:

Is it advisable / rational to run a Kubernetes cluster on the same physical cluster as Hadoop?

Thanks in advance...

+3


source to share


1 answer


Hortonworks' solution seems strange to me. Kubernetes manages docker containers on physical machines, cluster scheduling schedule (built by a set of Linux boxes). The natural way to integrate is to allow the yarn to run on docker containers managed by Kubernet. Why vice versa?



IMO the correct solution is not integrating kubernetes into your existing yarn cluster. Instead, you have to rebuild your cluster with a microservice centric architecture, which means that each service (yarn is one of them) runs in docker containers managed by Kubernetes. In the long run, this will be a more manageable, maintenance-friendly approach for a large data cluster.

0


source






More articles:


All Articles