Numpy parse int into bit groupings
I have np.array
ofnp.uint8
a = np.array([randint(1,255) for _ in range(100)],dtype=np.uint8)
and I want to break it down into low and high nibbles
I could get low nibble
low = np.bitwise_and(a,0xF)
and I could get high lumps with
high = np.bitwise_and(np.right_shift(a,4),0xF)
is there a way to do something like
>>> numpy.keep_bits(a,[(0,3),(4,7)])
numpy.array([
[low1,high1],
[low2,high2],
...
[lowN,highN]
])
I don’t even know what it’s going to be called ... but I thought maybe some zero guru would know a cool way to do this (in fact I’m looking to do this with uint32 and much more varied nibbles
basically something like struct.unpack
, but for vectorized numpy operations
EDIT : I went with a modified version of the accepted answer below
here is my final code for anyone interested
def bitmask(start,end):
"""
>>> bitmask(0,2) == 0b111
>>> bitmask(3,5) == 0b111000
:param start: start bit
:param end: end bit (unlike range, end bit is inclusive)
:return: integer bitmask for the specified bit pattern
"""
return (2**(end+1-start)-1)<<start
def mask_and_shift(a,mask_a,shift_a):
"""
:param a: np.array
:param mask_a: array of masks to apply (must be same size as shift_a)
:param shift_a: array of shifts to apply (must be same size as mask_a)
:return: reshaped a, that has masks and shifts applied
"""
masked_a = numpy.bitwise_and(a.reshape(-1,1), mask_a)
return numpy.right_shift(masked_a,shift_a)
def bit_partition(rawValues,bit_groups):
"""
>>> a = numpy.array([1,15,16,17,125,126,127,128,129,254,255])
>>> bit_partition(a,[(0,2),(3,7)])
>>> bit_partition(a,[(0,2),(3,5),(6,7)])
:param rawValues: np.array of raw values
:param bit_groups: list of start_bit,end_bit values for where to bit twiddle
:return: np.array len(rawValues)xlen(bit_groups)
"""
masks,shifts = zip(*[(bitmask(s,e),s) for s,e in bit_groups])
return mask_and_shift(rawValues,masks,shifts)
source to share
One-line, using broadcast, for the four bits of the lower and upper nibbles:
In [38]: a
Out[38]: array([ 1, 15, 16, 17, 127, 128, 255], dtype=uint8)
In [39]: (a.reshape(-1,1) & np.array([0xF, 0xF0], dtype=np.uint8)) >> np.array([0, 4], dtype=np.uint8)
Out[39]:
array([[ 1, 0],
[15, 0],
[ 0, 1],
[ 1, 1],
[15, 7],
[ 0, 8],
[15, 15]], dtype=uint8)
To summarize this, replace the hard-coded values [0xF, 0xF0]
and [0, 4]
with the corresponding bit masks and offsets. For example, to split values into three groups containing the highest two bits followed by the other two groups of three bits, you can do this:
In [41]: masks = np.array([0b11000000, 0b00111000, 0b00000111], dtype=np.uint8)
In [42]: shifts = np.array([6, 3, 0], dtype=np.uint8)
In [43]: a
Out[43]: array([ 1, 15, 16, 17, 127, 128, 255], dtype=uint8)
In [44]: (a.reshape(-1,1) & np.array(masks, dtype=np.uint8)) >> np.array(shifts, dtype=np.uint8)
Out[44]:
array([[0, 0, 1],
[0, 1, 7],
[0, 2, 0],
[0, 2, 1],
[1, 7, 7],
[2, 0, 0],
[3, 7, 7]], dtype=uint8)
source to share
So, I won't comment on the specific boolean operators you want to implement as bit hacking is not really my specialty, but I can tell you where you should look in numpy
to implement this kind of custom operator.
If you look at the source numpy
, you will notice that almost all of the bit-maniuplation methods in numpy
are just instances _MaskedBinaryOperation
, for example, the definition is bitwise_and
simple:
bitwise_and = _MaskedBinaryOperation(umath.bitwise_and)
The magic here comes in the form of a module umath
that calls down, usually for the low-level libraries it is built on numpy
. If you really want to, you can add your operator there, but I don't think it's worth cheating at this level.
However, this is not the only way to enable these features in numpy
. In fact, the module umath
has a really handy function called frompyfunc
that will allow you to turn an arbitrary python function into one of these handy operators umath
. The documentation can be found here . An example of creating such a function is shown below:
>>> oct_array = np.frompyfunc(oct, 1, 1)
>>> oct_array(np.array((10, 30, 100)))
array([012, 036, 0144], dtype=object)
>>> np.array((oct(10), oct(30), oct(100))) # for comparison
array(['012', '036', '0144'],
dtype='|S4')
If you are defining the specifics of the bitwise operator that you want to implement, using that interface would be the best way to implement it.
This doesn't answer 100% of your question, but I assumed your question was much more about using some custom bitwise operator in the correct form numpy
, rather than digging into the bitwise operator itself. Let me know if this is inaccurate and I can put together an example using the bitwise operator you mentioned above.
source to share