Go: Effects of empty curly braces on memory allocation of array initialization
I have played around in different ways to initialize / declare arrays in golang. I have different behaviors / results.
go version go1.3 darwin / amd64
version 1:
func main() {
a := [100000000]int64{}
var i int64
for i = 0; i < 100000000; i++ {
a[i] = i
}
}
Produces a 763 MB binary. It crashes in a few seconds when I run it with this message.
runtime: goroutine stack exceeds 1,000,000,000-byte limit
fatal error: stack overflow
version 2:
func main() {
var a [100000000]int64
var i int64
for i = 0; i < 100000000; i++ {
a[i] = i
}
}
Produces 456KB binary. It works in less than one second.
Question:
Can someone help me understand why these differences (and others I may have missed)? Thank!
change
Working hours:
I built two different snippets and ran the compiled versions, so no compile time was added. The first time I run version1 it is very slow. Here is the result.
go build version1.go
go build version2.go
These are the execution outputs
version 1
first start
time ./version1
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow
runtime stack:
runtime.throw(0x2fb42a8e)
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/panic.c:520 +0x69
runtime.newstack()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/stack.c:770 +0x486
runtime.morestack()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/asm_amd64.s:228 +0x61
goroutine 16 [stack growth]:
main.main()
/Users/ec/repo/offers/lol/version1.go:3 fp=0x2b7b85f50 sp=0x2b7b85f48
runtime.main()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/proc.c:247 +0x11a fp=0x2b7b85fa8 sp=0x2b7b85f50
runtime.goexit()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/proc.c:1445 fp=0x2b7b85fb0 sp=0x2b7b85fa8
created by _rt0_go
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/asm_amd64.s:97 +0x120
goroutine 17 [runnable]:
runtime.MHeap_Scavenger()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/mheap.c:507
runtime.goexit()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/proc.c:1445
./version1 0.00s user 0.10s system 1% cpu 7.799 total
second run
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow
runtime stack:
runtime.throw(0x2fb42a8e)
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/panic.c:520 +0x69
runtime.newstack()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/stack.c:770 +0x486
runtime.morestack()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/asm_amd64.s:228 +0x61
goroutine 16 [stack growth]:
main.main()
/Users/ec/repo/offers/lol/version1.go:3 fp=0x2b7b85f50 sp=0x2b7b85f48
runtime.main()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/proc.c:247 +0x11a fp=0x2b7b85fa8 sp=0x2b7b85f50
runtime.goexit()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/proc.c:1445 fp=0x2b7b85fb0 sp=0x2b7b85fa8
created by _rt0_go
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/asm_amd64.s:97 +0x120
goroutine 17 [runnable]:
runtime.MHeap_Scavenger()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/mheap.c:507
runtime.goexit()
/usr/local/Cellar/go/1.3/libexec/src/pkg/runtime/proc.c:1445
./version1 0.00s user 0.10s system 98% cpu 0.102 total
version 2
first start
time ./version2
./version2 0.16s user 0.26s system 99% cpu 0.429 total
second run
time ./version2
./version2 0.17s user 0.25s system 97% cpu 0.421 total
source to share
In version 1, you declare a literal array [100000000]int64{}
, which is immediately allocated by the compiler.
Version 2, you declare the type a
as [100000000]int64
.
If you only have a variable declaration, the content is not known at compile time. In version 2, the compiler knows what a
a type is [100000000]int64
, but no memory is allocated until runtime.
When you use a literal, this exact memory representation is written to a binary file. It works the same as if you were declaring a literal string
against a type variable string
; the string literal will be written in place, while the variable declaration is just a placeholder.
Although the current compiler (jump 1.3) allows a
for heap escape, literal data is expected to be on the stack frame. You can see this in the build output (frame size is 800000016):
TEXT "".func1+0(SB),$800000016-0
If you really need a literal larger than the one that can fit on the stack, you can put it in a global variable. The following works just fine:
var a = [100000000]int64{1}
func func1() {
var i int64
for i = 0; i < 100000000; i++ {
a[i] = i
}
}
I needed to initialize at least one value in a
here because it seems like the compiler might exclude this literal if it is null.
source to share
This is not really an answer, but maybe someone might find this helpful.
In my case it happens when I try to json.Marshal with the following structure:
type Element struct {
Parent *Element
Child []*Element
}
Where parent points to the element that has the current element in Child.
So I just mark "Parent" as json:"-"
which should be ignored when sorting.
source to share