bug in backward() or help needed #2077
Comments
|
Btw you can still write on discord, just embed the logs as a file. Would love to help, but this will be difficult to reproduce without a minimal example. |
This week I'll prepare some example an Upload code with more details, but based on working with pyTorch I thing it is not a bug, but burn not support this feature yet, I thing in backwards() are missing support for retain_graph=True and this is cause. |
|
Ahhh ok that might be the case. We have an open issue for that (#1802). |
|
@laggui for epoch in 1..config_generator.num_epochs + 1 {
let mut total_training_luma_ssim = 0.0;
let mut iter_training = 0;
for (iteration, batch) in dataloader_train.iter().enumerate() {
let image_lr = batch.clone().input_luma;
let image_hr = batch.clone().output_luma;
// ------ First Step -----
let degenerate_lr = model_degenerator.forward(image_hr.clone());
let generate_hr = model_generator.forward(degenerate_lr.clone());
let loss_charbonier_degenerate =
charbonier_loss(degenerate_lr.clone(), image_lr.clone(), 1e-3);
let loss_ssim_degenerate =
ssim_simple(degenerate_lr.clone(), image_lr.clone(), true, 1.0);
let loss_degenerate_recon = loss_charbonier_degenerate.mul_scalar(0.1)
+ loss_ssim_degenerate.clone().mul_scalar(0.9);
let cycle_hr = cycle_loss(image_hr.clone(), generate_hr, lambda);
let identify_lr = identify_loss(image_lr.clone(), degenerate_lr.clone(), lambda);
let total_loss_deg = loss_degenerate_recon.clone() + cycle_hr + identify_lr;
let grads = total_loss_deg.backward();
let grads_degenerator = GradientsParams::from_grads(grads, &model_degenerator);
// ------ Second Step -----
let generate_hr = model_generator.forward(degenerate_lr);
let degenerate_lr = model_degenerator.forward(generate_hr.clone());
let cycle_lr: Tensor<B, 1> =
cycle_loss(image_lr.clone(), degenerate_lr.clone(), lambda);
let identify_hr = identify_loss(image_hr.clone(), generate_hr.clone(), lambda);
let loss_charbonier_generate =
charbonier_loss(generate_hr.clone(), image_hr.clone(), 1e-3);
let loss_ssim_generate = ssim_simple(generate_hr, image_hr, true, 1.0);
let loss_generator = loss_charbonier_generate.mul_scalar(0.1)
+ loss_ssim_generate.clone().mul_scalar(0.9);
let total_loss_gen = loss_generator.clone() + cycle_lr + identify_hr;
let grads = total_loss_gen.backward();
let grads_generator = GradientsParams::from_grads(grads, &model_generator);
model_degenerator = optim_degenerator.step(
config_degenerator.l_rate,
model_degenerator,
grads_degenerator,
);
model_generator =
optim_generator.step(config_generator.l_rate, model_generator, grads_generator);
// Stats
total_training_luma_ssim += loss_ssim_generate.clone().into_scalar().elem::<f32>();
iter_training += 1;
println!(
"[Train - Epoch {} Iteration {}] Loss D_Gen {:.4} Loss Gen: {:.4} SSIM D_Gen: {:.4} SSIM Gen: {:.4}",
epoch,
iteration,
total_loss_deg.into_scalar(),
total_loss_gen.into_scalar(),
loss_ssim_degenerate.into_scalar(),
loss_ssim_generate.into_scalar(),
);
}
|
|
While the lack of an equivalent to Without I'll keep this open. Hopefully we can reproduce this somehow with a small example. |
I have a strange issue with backward() I have two generators, gen1 and gen2, I calculate loss on three ways, loss_1, loss_2, loss_3
All compute for gen1 are ok
Part 1.
let out = gen1.forward(input);
let out2 = gen2.forward(out.clone())
... calculate loss
then total_loss = loss_1 + loss_2 + loss_3
and:
let grads1 = total_loss.backward();
let grads_gen1 = GradientsParams::from_grads(grads, &gen1);
This works fine
And then:
Part 2.
let out3 = gen2.forward(out)
let out = gen1.forward(out3)
... calculate loss
then in similar way but for others arguments total_loss = loss_1 + loss_2 + loss_3
and:
let grads2 = total_loss.backward();
let grads_gen2= GradientsParams::from_grads(grads, &gen2);
and update models
.. .step()...
all compute are done in one loop. And here for grads2 I have a issue, when I remove a loss_3 from total_loss everything works fine but with loss_3 I have this error msg:
where 1280 it's output size for tensor for width and height. All padding etc. should be ok for gen1 and gen2 because in otherways Part 1 wouldn't work. Do you have any idea or some sugestion what could be cause? I wanted write on discord on the channel #help, but with logs msg exceeded 2000 chars
The text was updated successfully, but these errors were encountered: