why lucid can run with varying image size by transform_f

[totti0223]

I want to know why lucid can iterate when feeding image that its size dynamically changes by transform_f per every iteration. For example, if I run the below code,

import lucid.optvis.transform as lucid_t
from lucid.optvis. render import make_transform_f
import numpy as np

#from lucid.optvis transform.py
standard_transforms = [
    lucid_t.pad(12, mode='constant', constant_value=.5),
    lucid_t.jitter(8),
    lucid_t.random_scale([1 + (i-5)/50. for i in range(11)]),
    lucid_t.random_rotate(list(range(-10, 11)) + 5*[0]),
    lucid_t.jitter(4),
  ]

prior = np.random.normal(size=(1,224,224,3))

input_image = K.variable(prior)
transform_f = make_transform_f(standard_transforms)
transformed = transform_f(input_image)

post = K.eval(transformed)

print(prior.shape)
print(post.shape)

I get

(1, 224, 224, 3)
(1, 221, 221, 3)

*I know values changes everytime because of the random scale function.

feature maximization is a gradient ascent so in iteration t, with above transformation

image(t+1) = transform_f(image(t)) + gradient

where their tensor xy size is
image(t) (224,224)
image(t+1) (221,221)

however the final output of lucid which generates after iterations is a image of size (224,224) no matter what if I start with (224,224),

so is lucid either doing the following or any other?

1. after transform_f, resizing the size back to (224,224) and calculating the gradients by that size in every step.
   or
2. calculate the gradients in size (221,221), update the image by adding gradients, resizing back to (224,224) before the next calculation step.

would be helpful if you can tell me where the codes that lucid is doing inside.

thanks

[colah]

Hi @totti0223! I can see how this might be a bit confusing...

The update equation is*

image(t+1) = image(t) + gradient

where gradient is the gradient on the original, untransformed image. While the gradient with respect the transformed image might have a shape of (221, 221), the gradient with respect to the original image always has the shape (224, 224).

In order to get this, TensorFlow not only backprops through the neural network, but it then also backprops through the transformation to get to the original image!

*This isn't quite true, since we typically optimize with ADAM rather than normal gradient descent.

[colah]

The relevant code is lucid.optvis.render:182, when we instantiate an optimizer to minimize loss.

vis_op = optimizer.minimize(-loss, global_step=global_step)

But that's kind of opaque. The optimizer calls tf.gradients which automatically differentiates everything, including through the transformation.

[hegman12]

Also, image shape wont change every iteration. rather it changes every time you call optiviz.render.render_viz (or every time you call make_t_image(param_f)) if you are using lower level API). That means, gradients shape wont change for every iteration.
So, if you specify image size as (244,244,3), and after applying transformation, lucid determines the shape of image as (221,221,3), then after backprop, your gradients will have shape (221,221,3). In other words, even though you specify 244 as shape, lucid actually runs with image shape of 221.

[totti0223]

Hi @totti0223! I can see how this might be a bit confusing...

The update equation is*

image(t+1) = image(t) + gradient

where gradient is the gradient on the original, untransformed image. While the gradient with respect the transformed image might have a shape of (221, 221), the gradient with respect to the original image always has the shape (224, 224).

In order to get this, TensorFlow not only backprops through the neural network, but it then also backprops through the transformation to get to the original image!

*This isn't quite true, since we typically optimize with ADAM rather than normal gradient descent.

Thx. finally got the point! i was looking through other implementations of feature visualizations and noticed that others tend to stop the backpropagation post transformation, so that transformation had to be the same size pre transoformation, or adding a crop or resize to the final transformation. i am currently using part of the lucid code to migrate to my environment to run with my built model and that single point made it more lucid like images.

[totti0223]

Also, image shape wont change every iteration. rather it changes every time you call optiviz.render.render_viz (or every time you call make_t_image(param_f)) if you are using lower level API). That means, gradients shape wont change for every iteration.
So, if you specify image size as (244,244,3), and after applying transformation, lucid determines the shape of image as (221,221,3), then after backprop, your gradients will have shape (221,221,3). In other words, even though you specify 244 as shape, lucid actually runs with image shape of 221.

Thanks for your comment. after reading the comment of @colah, seems confusing again. i first thought that gradients are calculated by the transformed shape, but if so, doing numbers of iterations make the image & gradients shape really tiny (224 to 221, 221 to 218..... 1?) or explode in case of enlarging resize. that is why I posted a question but as you are saying

Also, image shape wont change every iteration.

maybe i am still missing something.

[colah]

Also, image shape wont change every iteration.

If you use transformations like random_scale, the transformed image has a random shape, which may change between iterations.

The actual optimized image does not change shape.

lucid determines the shape of image as (221,221,3), then after backprop, your gradients will have shape (221,221,3). In other words, even though you specify 244 as shape, lucid actually runs with image shape of 221.

When you backprop to the transformed image, the gradient will have a shape of (221,221,3). TensorFlow will continue backproping, back through the transformation, resulting in a gradient of (224,224,3) on the optimized image.

[totti0223]

Also, image shape wont change every iteration.

If you use transformations like random_scale, the transformed image has a random shape, which may change between iterations.

The actual optimized image does not change shape.

lucid determines the shape of image as (221,221,3), then after backprop, your gradients will have shape (221,221,3). In other words, even though you specify 244 as shape, lucid actually runs with image shape of 221.

When you backprop to the transformed image, the gradient will have a shape of (221,221,3). TensorFlow will continue backproping, back through the transformation, resulting in a gradient of (224,224,3) on the optimized image.

got your point. was just a matter of understanding expressions 👍

[hegman12]

TensorFlow will continue backproping, back through the transformation, resulting in a gradient of (224,224,3) on the optimized image.

Got it, so final computation graph = transformations computation graph + model computation graph. I think i was still in the hangover of issue #101