Using second impulse as part of a new cost function? (Tensor flow and / or kera)
I am currently trying to take Adam's second-order time, v_t, and use that as an optional term in my cost function. How can I implement something like this:
Cost = Cross Entropy + v_t*some_function(weights)
Can this be done in python? Or do I need to write my own C ++ code for this? Is it also easy to do this within Keras? Here's the code for the cost function I'm trying to add to keras:
def my_loss(y_pred, y_true, current_weights, v_t):
normal_loss=K.categorial_cross_entropy(y_pred,y_true)
additional_term=K.dot(K.square(current_weights - K.some_function(current_weights)), v_t)
return normal_loss + additional_term
source to share
This will be problematic no matter what structure you do it with. In particular, from the ADAM paper, the relevant lines are:
g_t = d Cost / d weights
v_t = beta2 * v_{t-1} + (1 - beta2) g_t^2
Now, if you have to include v_t in the cost, it will be an implicit equation:
g_t = d Cross Entropy / d weights + d (v_t*some_function) / d weights
v_t = beta2 * v_{t-1} + (1 - beta2) g_t^2
Notice how v_t appears in both equations. We can expand it as such for clarity.
v_t = beta2 * v_{t-1} + (1 - beta2) [d Cross Entropy / d weights + d (v_t*some_function) / d weights]^2
You can try to solve this exactly, but in doing so you have to use some form of implicit solver, which will be very computationally expensive. One way: fixed point iteration .
source to share