Embedding binary data in a script efficiently
I've seen some installation files (like the huge ones, install.sh for Matlab or Mathematica) for Unix-like systems, they had to embed quite a lot of binary data like icons, sound, graphics, etc. into the script. I am wondering how this can be done, as it might be useful for simplifying the file structure.
I'm especially interested in doing this with Python and / or Bash.
The existing methods that I know in Python:
- Just use byte string:,
x = b'\x23\xa3\xef' ...
horribly inefficient, takes up half MB for a 100KB WAV file. - base64, better than option 1, increase the size 4/3 times.
I'm wondering if there are other (better) ways to do this?
source to share
You can use base64 + compression (like bz2 ) if it fits your data (like if you don't embed the already compressed data).
For example, to create your data (for example, your data consists of 100 null bytes followed by 200 bytes with the value 0x01):
>>> import bz2
>>> bz2.compress(b'\x00' * 100 + b'\x01' * 200).encode('base64').replace('\n', '')
'QlpoOTFBWSZTWcl9Q1UAAABBBGAAQAAEACAAIZpoM00SrccXckU4UJDJfUNV'
And use it (in your script) to write data to a file:
import bz2
data = 'QlpoOTFBWSZTWcl9Q1UAAABBBGAAQAAEACAAIZpoM00SrccXckU4UJDJfUNV'
with open('/tmp/testfile', 'w') as fdesc:
fdesc.write(bz2.decompress(data.decode('base64')))
source to share
Here's a quick and dirty way. Create the following script called MyInstaller
:
#!/bin/bash
dd if="$0" of=payload bs=1 skip=54
exit
Then add your binary to your script and make it executable:
cat myBinary >> myInstaller
chmod +x myInstaller
When you run the script, it will copy the binary to a new file specified in the path of=
. It can be a tar file or something else, so after the dd command you can do some additional processing (unarchiving, setting execute permissions, etc.). Just adjust the number in "skip" to reflect the total length of the script before the start of the binary data.
source to share