How to view complete dependency tree for nested Go dependencyd
I am trying to debug the following build error in our CI where "A depends on B, which cannot be built because it depends on C." I am building my data service that does not directly depend on kafkaAvailMonitor.go making it difficult to track down this error. In other words:
data (what I'm building) depends on (?) which depends on kafkaAvailMonitor.go
It might seem like a trivial fix for a developer to just "go", but I can't do that as part of the release process - I need to find the person who added the dependency and ask them to fix it.
I know there are tools out there to visualize the dependency tree and other more complex build systems, but this seems like a pretty straightforward problem: is there a way to go through the entire trail to see what is causing the build issue? In response to one of the comments - all the programming languages I have used to date, for example. Java / Python provide this information out of the box, and it's very useful for debugging code that I didn't write specifically because I can't just look in one place like the POM or requirements.txt to see who added the dependency.
Also in response to a comment - if the following is not a stack trace, what is it? List of errors written to stdout by binary? Aren't these errors pushed onto the stack for the first time by a go binary that gets executed? It seems like a silly distinction to say "this is not a stack trace", especially when it has nothing to do with my question, but I removed the word "stack" from the question if it makes people happy.
go build -a -v
../../../msgq/kafkaAvailMonitor.go:8:2: cannot find package
"github.com/Shopify/sarama/tz/breaker" in any of:
/usr/lib/go-1.6/src/github.com/Shopify/sarama/tz/breaker (from $GOROOT)
/home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker (from $GOPATH)
/home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
/home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker
/home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
source to share
if the following is not a stack trace, what is it?
This is a list of paths that Go looks for your missing package.
I don't know who is importing kafkaAvailMonitor.go
It is not "imported", just a portion of your sources and compiled.
Except that it cannot compile because it needs github.com/Shopify/sarama/tz/breaker
one that is not in GOROOT
or GOPATH
.
However, check what go list
will return in your direct package to see if it is mentioned kafkaAvailMonitor
.
go list
can display both packages that directly depend on your package and a complete set of transitive dependencies.
% go list -f '{{ .Imports }}' github.com/davecheney/profile
[io/ioutil log os os/signal path/filepath runtime runtime/pprof]
% go list -f '{{ .Deps }}' github.com/davecheney/profile
[bufio bytes errors fmt io io/ioutil log math os os/signal path/filepath reflect run
Then you can script go to view the list of all dependencies.
See. This bash script for example , by Noel Cower ( nilium
)
#!/usr/bin/env bash
# Usage: lsdep [PACKAGE...]
#
# Example (list github.com/foo/bar and package dir deps [the . argument])
# $ lsdep github.com/foo/bar .
#
# By default, this will list dependencies (imports), test imports, and test
# dependencies (imports made by test imports). You can recurse further by
# setting TESTIMPORTS to an integer greater than one, or to skip test
# dependencies, set TESTIMPORTS to 0 or a negative integer.
: "${TESTIMPORTS:=1}"
lsdep_impl__ () {
local txtestimps='{{range $v := .TestImports}}{{print . "\n"}}{{end}}'
local txdeps='{{range $v := .Deps}}{{print . "\n"}}{{end}}'
{
go list -f "${txtestimps}${txdeps}" "$@"
if [[ -n "${TESTIMPORTS}" ]] && [[ "${TESTIMPORTS:-1}" -gt 0 ]]
then
go list -f "${txtestimps}" "$@" |
sort | uniq |
comm -23 - <(go list std | sort) |
TESTIMPORTS=$((TESTIMPORTS - 1)) xargs bash -c 'lsdep_impl__ "$@"' "$0"
fi
} |
sort | uniq |
comm -23 - <(go list std | sort)
}
export -f lsdep_impl__
lsdep_impl__ "$@"
source to share
The above answer still doesn't show me the dependency tree, so I took the time to write a script to do what I needed - hope this helps other people.
The problem with the above solution (others suggested as go) is that it only tells me the top level. They don't "cross the tree." This is the result I am getting - it doesn't help in my opinion.
.../npd/auth/
.../mon/mlog
.../auth/service
This is what I am trying to get - I know auth is broken (top) and this breaker is broken (bottom) from go build, but I have no idea what's in between - my script below gives me this output.
.../npd/auth/
.../npd/auth/service
.../npd/auth/resource
.../npd/auth/storage
.../npd/middleware
.../npd/metrics/persist
.../npd/kafka
.../vendor-library/src/github.com/Shopify/sarama
.../vendor-library/src/github.com/Shopify/sarama/vz/breaker
My script
import subprocess
import os
folder_locations=['.../go/src','.../vendor-library/src']
def getImports(_cwd):
#When the commands were combined they overflowed the bugger and I couldn't find a workaround
cmd1 = ["go", "list", "-f", " {{.ImportPath}}","./..."]
cmd2 = ["go", "list", "-f", " {{.Imports}}","./..."]
process = subprocess.Popen(' '.join(cmd1), cwd=_cwd,shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out1, err = process.communicate()
process = subprocess.Popen(' '.join(cmd2), cwd=_cwd,shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out2, err = process.communicate()
out2clean=str(out2).replace("b'",'').replace('[','').replace(']','').replace("'",'')
return str(out1).split('\\n'),out2clean.split('\\n')
def getFullPath(rel_path):
for i in folder_locations:
if os.path.exists(i+'/'+rel_path):
return i+'/'+rel_path
return None
def getNextImports(start,depth):
depth=depth+1
indent = '\t'*(depth+1)
for i,val in enumerate(start.keys()):
if depth==1:
print (val)
out1,out2=getImports(val)
noDeps=True
for j in out2[i].split(' '):
noDeps=False
_cwd2=getFullPath(j)
new_tree = {_cwd2:[]}
not_exists = (not _cwd2 in alltmp)
if not_exists:
print(indent+_cwd2)
start[val].append(new_tree)
getNextImports(new_tree,depth)
alltmp.append(_cwd2)
if noDeps:
print(indent+'No deps')
_cwd = '/Users/.../npd/auth'
alltmp=[]
start_root={_cwd:[]}
getNextImports(start_root,0)
source to share