To understand GAN we just need to get two  parts:

Generator and Discreminator.

For Generator,

 we append a Gauss distribution

we append a Relu

WE append a dense layer

we BN  we relu we dense repeat

we dense finally 

For discriminator,

we append linear layer

 we relu

we linear 

we relu

we seperate two kinds

Important is the process to forward the generator and discriminator

save the image 

def generator_forward(test_noises):
    fake_data = net_g(test_noises)
    fake_out = net_d(fake_data)
    loss_g = adversarial_loss(fake_out, ops.ones_like(fake_out))
    return loss_g
def discriminator_forward(real_data, test_noises):
    fake_data = net_g(test_noises)
    fake_out = net_d(fake_data)
    real_out = net_d(real_data)
    real_loss = adversial_loss(real_out, ops.ones_like(real_out))
    fake_loss = adversial_loss(fake_out, ops.zeros_like(fake_out))
    return loss_d

just train them.

where :

adversial_loss=nn.BCELoss(reduction='mean')

optimizer_d = nn.Adam(net_d.trainable_params(),learning_rate = lr, beta1 = 0.5, bet2 = 0.999)

Rem:Adam is a kind of optimizer 

Using EMA

def init_adam_states(feature_dim):
    v_w, v_b = torch.zeros((feature_dim,1)),torch.zeros(1)
    s_w, s_b = torch.zeros((feature_dim,1)), torch.zeros(1)
    return ((v_w, s_w),(v_b,s_b))
def adam(params, states, hyperparams):
    beta1, beta2, eps = 0.9,0.999, 1e-6
    for p, (v, s) in zip(params, states):
        with torch.no_grad():
            v[:] = beta1 * v + (1 - beta1) * p.grad
            s[:] = beta2 * v + (1 - beta2) * torch.square(p.grad)# it would be better if we use s[:] = s + (1- beta2) * torch.sign(torch.square(p.grad) - s)* torch.square(p.grad) 
            v_bias_corr = v / (1 - beta1 ** hyperparams['t'])
            s_bias_corr = s / (1- beta2 ** hyperparams['t'])
            p[:]  -= hyperparams['lr'] * v_bias_corr / (torch.sqrt(s_bias_corr) +eps)
        p.grad.data.zero_()
    hyperparams['t'] += 1
 
    

RMS PROP is some kind the same.

def rmsprop_2d(x1, x2, s1, s2):

        g1,g2, eps = 0.2 * x1, 4* x2, 1e-6

        s1 = gamma *s1 + (1- gamma) * g1 **2

        s2 = gamma * s2 + (1- gamma) *g2 ** 2

        x1 -= eta /math.sqrt(s1 + eps) * g1

        x2 -= eta / math.sqrt(s2 + eps) *g2

        return x1, x2, s1, s2

def f_2d(x1, x2):

        return 0.1 *x1 **2 + 2*x2 **2

eta,gamma =0.4, 0.9 

def init_rmsprop_states(feature_dim):

        s_w = torch.zeros((feature_dim, 1))

        s_b = torch.zeros(1)

        return (s_w, s_b)

def rmsprop(params, states, hyperparams):

        gamma, eps = hyperparams['gamma'], 1e-6

        for p, s in zip(params, states):

                with torch.no_grad():

                        s[:] = gamma *s +(1- gamma) * torch.square*(p.grad)

                        p[:] -= hyperparams['lr'] * p.grad / torch.sqrt(s + eps)

                p.grad.data.zero_()