Hadoop 2.5.2 mvn build fail Windows 7 x64 - compile-ms-native-dll on hasoop-common?

Question

Hadoop 2.5.2 mvn build fail Windows 7 x64 - compile-ms-native-dll on hasoop-common?

I am following the Hadoop2OnWindows Wiki here and I have read BUILDING.txt

in the source tree many times. I think I have all the dependencies figured out, but still I get a strange error Failed to execute goal (compile-ms-native-dll)

(full log below).

I have installed Windows SDK 7.1

I have the following things in my PATH:

Cygwin/bin
MSBuild ( C:\Windows\Microsoft.NET\Framework64\v4.0.30319

)
Buffer Protocol 2.5.0
Zlib and Zlib / include (so it can access zlib.h)
Findbugs
Maven

I also installed Platform=x64

, and of course mine JAVA_HOME

and M2_HOME

also installed.

I am running mvn package -Pdist,native-win,docs -DskipTests -Dtar

on Windows SDK 7.1 command line.

I used to have Gnuwin32

one that was giving me the error below but moved to Cygwin/bin

, suspecting something might be inappropriate with Gnuwin32

, but I still get the following error. What should I do?

Here's the error log

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (com
pile-ms-native-dll) on project hadoop-common: Command execution failed. Process
exited with an error: 1(Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal o
rg.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-native-dll) on project h
adoop-common: Command execution failed.
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:216)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
ct(LifecycleModuleBuilder.java:116)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
ct(LifecycleModuleBuilder.java:80)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThre
adedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(Lifecycl
eStarter.java:120)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:347)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:154)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:582)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun
cher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
a:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
uncher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution fai
led.
        at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:345)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(Default
BuildPluginManager.java:132)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:208)
        ... 19 more

Only when hadoop-common

it doesn't work. Here are some more logs that might be helpful, I just can't figure out what it is, but

[INFO]
[INFO] --- exec-maven-plugin:1.2:exec (compile-ms-native-dll) @ hadoop-common --
-
[DEBUG] Configuring mojo org.codehaus.mojo:exec-maven-plugin:1.2:exec from plugi
n realm ClassRealm[plugin>org.codehaus.mojo:exec-maven-plugin:1.2, parent: sun.m
isc.Launcher$AppClassLoader@1e4ad14c]
[DEBUG] Configuring mojo 'org.codehaus.mojo:exec-maven-plugin:1.2:exec' with bas
ic configurator -->
[DEBUG]   (f) arguments = [C:\hdc\hadoop-common-project\hadoop-common/src/main/n
ative/native.sln, /nologo, /p:Configuration=Release, /p:OutDir=C:\hdc\hadoop-com
mon-project\hadoop-common\target/bin/, /p:CustomSnappyPrefix=, /p:CustomSnappyLi
b=, /p:CustomSnappyInclude=, /p:RequireSnappy=false]
[DEBUG]   (f) basedir = C:\hdc\hadoop-common-project\hadoop-common
[DEBUG]   (f) classpathScope = runtime
[DEBUG]   (f) executable = msbuild
[DEBUG]   (f) longClasspath = false
[DEBUG]   (f) project = MavenProject: org.apache.hadoop:hadoop-common:2.5.2 @ C:
\hdc\hadoop-common-project\hadoop-common\pom.xml
[DEBUG]   (f) session = org.apache.maven.execution.MavenSession@1e54fd3d
[DEBUG]   (f) skip = false
[DEBUG] -- end configuration --
[DEBUG] Executing command line: msbuild C:\hdc\hadoop-common-project\hadoop-comm
on/src/main/native/native.sln /nologo /p:Configuration=Release /p:OutDir=C:\hdc\
hadoop-common-project\hadoop-common\target/bin/ /p:CustomSnappyPrefix= /p:Custom
SnappyLib= /p:CustomSnappyInclude= /p:RequireSnappy=false
Building the projects in this solution one at a time. To enable parallel build,
please add the "/m" switch.
Build started 23-11-2014 00:07:43.
Project "C:\hdc\hadoop-common-project\hadoop-common\src\main\native\native.sln"
on node 1 (default targets).
ValidateSolutionConfiguration:
  Building solution configuration "Release|x64".
Project "C:\hdc\hadoop-common-project\hadoop-common\src\main\native\native.sln"
(1) is building "C:\hdc\hadoop-common-project\hadoop-common\src\main\native\nati
ve.vcxproj" (2) on node 1 (default targets).
InitializeBuildStatus:
  Touching "..\..\..\target\native\Release\native.unsuccessfulbuild".
ClCompile:
  C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\CL.exe /c /I.
.\winutils\include /I..\..\..\target\native\javah /IC:\Java\jdk1.7.0_71\include
/IC:\Java\jdk1.7.0_71\include\win32 /I.\src /Zi /nologo /W3 /WX- /O2 /Oi /GL /D
WIN32 /D NDEBUG /D _WINDOWS /D _USRDLL /D NATIVE_EXPORTS /D _WINDLL /D _UNICODE
/D UNICODE /Gm- /EHsc /MD /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Fo"..\..
\..\target\native\Release\\" /Fd"..\..\..\target\native\Release\vcWindows7.1SDK.
pdb" /Gd /TC /wd4244 /errorReport:queue src\org\apache\hadoop\io\compress\zlib\Z
libCompressor.c src\org\apache\hadoop\io\compress\zlib\ZlibDecompressor.c
  ZlibCompressor.c
c:\hdc\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop
\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083: C
annot open include file: 'zlib.h': No such file or directory [C:\hdc\hadoop-comm
on-project\hadoop-common\src\main\native\native.vcxproj]
  ZlibDecompressor.c
c:\hdc\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop
\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083: C
annot open include file: 'zlib.h': No such file or directory [C:\hdc\hadoop-comm
on-project\hadoop-common\src\main\native\native.vcxproj]
Done Building Project "C:\hdc\hadoop-common-project\hadoop-common\src\main\nativ
e\native.vcxproj" (default targets) -- FAILED.
Done Building Project "C:\hdc\hadoop-common-project\hadoop-common\src\main\nativ
e\native.sln" (default targets) -- FAILED.

Build FAILED.

"C:\hdc\hadoop-common-project\hadoop-common\src\main\native\native.sln" (default
 target) (1) ->
"C:\hdc\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj" (def
ault target) (2) ->
(ClCompile target) ->
  c:\hdc\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hado
op\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083:
 Cannot open include file: 'zlib.h': No such file or directory [C:\hdc\hadoop-co
mmon-project\hadoop-common\src\main\native\native.vcxproj]
  c:\hdc\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hado
op\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083:
 Cannot open include file: 'zlib.h': No such file or directory [C:\hdc\hadoop-co
mmon-project\hadoop-common\src\main\native\native.vcxproj]

    0 Warning(s)
    2 Error(s)

+3

java windows maven hadoop msbuild

ComputerFellow 22 nov. 14 at 18:43

source to share

3 answers

I personally have used the command mvn clean package -Pdist,native-win -DskipTests -Dtar -Dmaven.javadoc.skip=true

successfully to build version 2.4.1 on windows 8. I had many different problems but don't remember this being one of them. But it seems to be complaining about zlib.

0

Lauri Peltonen 24 nov. 14 at 12:26

source to share

Download protobuf and add it to your PATH

https://code.google.com/p/protobuf/downloads/detail?name=protoc-2.5.0-win32.zip&can=2&q=

0

Surendran Duraisamy 06 jan. 15 at 5:22 am

source to share

THN · Accepted Answer · 2015-01-17T19:33:26+0000

From the log, it is obvious that you are missing zlib.h Have you set the env var ZLIB_HOME to the directory containing zlib.h? Please note that the zlib.h directory in PATH is incorrect and unnecessary, only the zlib bin directory is needed as stated in the build guide https://svn.apache.org/viewvc/hadoop/common/branches/branch-2/BUILDING .txt? view = markup

Also, zlib.h requires some header not to be present on Windows, so you will need to download those headers and place in the same folder as zlib.h. For more information, see Is there a replacement for unistd.h for Windows (Visual C)? to get unistd.h, and https://gist.github.com/ashelly/7776712 to get getopt.h

For those on Win32 (like me), you may have faced a lot more problems. The key decision should be to edit the .sln and .vcxprj files from winutils and its own package to be compatible with the Win32 platform.

Hadoop 2.5.2 mvn build fail Windows 7 x64 - compile-ms-native-dll on hasoop-common?

More articles: