Why can't more than 32 cores be requested from YARN to complete a job?
Setting:
- Not. nodes: 3
- Not. cores: 32 cores per machine.
- RAM: 410 GB per machine
- Spark Version: 1.2.0
- Hadoop version: 2.4.0 (Hortonworks)
Purpose:
- I want to run a Spark job with more than 32 executor cores.
Problem:
When I request more than 32 executor cores for a Spark job, I get the following error:
Uncaught exception: Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured, requestedVirtualCores=150, maxVirtualCores=32
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:212)
at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:96)
at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:501)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
Here are some properties from my yarn-site.xml file:
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>400000</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>3072</value>
</property>
From the top to properties, I thought I could request 400k / 3072 = ~ 130 cores. But that still limits me to 32. How can I assign more than 32 executor cores to Spark?
Please let me know if more information is needed and I will update the question.
EDIT 1:
Vcore settings from yarn-site.xml file
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
+3
source to share