Moving from RGB to scalars in matplotlib colormap

I am using matplotlib to convert arrays of 400 floats (read from a text file) to 100x100 pixel images using a predefined color scheme:

vec = [ float(i) for i in l.split()[1:] ]
v = normalise(np.array(vec).reshape((20,20)))
im = Image.fromarray(np.uint8(cm.coolwarm(v)*255))
im = im.resize((100, 100))

      

        
        
        
      

    

This works great. But now I would like to reverse this process. That is, given the image, I would like to go from the RGB values ​​of each pixel back to scalars (to get the original array). How should I do it?

NB: my question is related to this , except that I don't want to take any image and convert it back to an array - only the ones that were generated by my color palette in the first place.