Deep Learning Demystified

Demystified: Categorical Cross-Entropy

A quick primer on cross entropy as a loss function for deep learning models

Classification using a Feedforward Network

Image for post
Image for post
from keract import get_activations, display_activationsthis_x  = np.array([random_choice_classification])activations = get_activations(model,this_x, auto_compile=True)
[9.595770358861166e-13,
6.149275577627122e-07,
0.9999992847442627, # we see our model correctly predicts the class
1.340575437325242e-07,
3.4247338298598606e-10,
1.325630837739622e-11,
1.2711021471489836e-12,
1.4796790459570275e-09,
9.361995623180519e-09,
2.2075417882572168e-10]

Categorical Cross Entropy

Image for post
Image for post
Cross entropy
def rel_entropy(p,q):
assert(len(p) == len(q))
_sum = 0
for i, _p in enumerate(p):
# we add a small constant to avoid taking the log of zero
_sum += _p * math.log2(q[i]+1e-20)
return -_sum
Image for post
Image for post

Written by

Data Scientist

Get the Medium app