Apply transformation for Tensor without including it in Backward

Let's say I have n layered neural network. After running l layers, I want to apply some transformation to the l^th layer output, without including that transformation in backpropagation.

For e.g. :

output_layer_n = self.LinearLayer(output_layer_prev)

#apply some transformation to output_layer_n, but don't want to take autograd w.r.t. this transformation, basically this transformation function doesn't have any parameter 

output_layer_n.data = TransformationFunction(output_layer_n.data)

So how should I go about implementing it? What I want is not to take gradient accounted for TransformationFunction() in my code.

edited Dec 14 '18 at 18:57

Maxim

31.9k2177128

asked Nov 22 '18 at 4:48

random_28

733516

add a comment |

Let's say I have n layered neural network. After running l layers, I want to apply some transformation to the l^th layer output, without including that transformation in backpropagation.

For e.g. :

output_layer_n = self.LinearLayer(output_layer_prev)

#apply some transformation to output_layer_n, but don't want to take autograd w.r.t. this transformation, basically this transformation function doesn't have any parameter 

output_layer_n.data = TransformationFunction(output_layer_n.data)

So how should I go about implementing it? What I want is not to take gradient accounted for TransformationFunction() in my code.

edited Dec 14 '18 at 18:57

Maxim

31.9k2177128

asked Nov 22 '18 at 4:48

random_28

733516

add a comment |

Let's say I have n layered neural network. After running l layers, I want to apply some transformation to the l^th layer output, without including that transformation in backpropagation.

For e.g. :

output_layer_n = self.LinearLayer(output_layer_prev)

#apply some transformation to output_layer_n, but don't want to take autograd w.r.t. this transformation, basically this transformation function doesn't have any parameter 

output_layer_n.data = TransformationFunction(output_layer_n.data)

So how should I go about implementing it? What I want is not to take gradient accounted for TransformationFunction() in my code.

edited Dec 14 '18 at 18:57

Maxim

31.9k2177128

asked Nov 22 '18 at 4:48

random_28

733516

Let's say I have n layered neural network. After running l layers, I want to apply some transformation to the l^th layer output, without including that transformation in backpropagation.

For e.g. :

output_layer_n = self.LinearLayer(output_layer_prev)

#apply some transformation to output_layer_n, but don't want to take autograd w.r.t. this transformation, basically this transformation function doesn't have any parameter 

output_layer_n.data = TransformationFunction(output_layer_n.data)

So how should I go about implementing it? What I want is not to take gradient accounted for TransformationFunction() in my code.

python pytorch backpropagation tensor

edited Dec 14 '18 at 18:57

Maxim

31.9k2177128

asked Nov 22 '18 at 4:48

random_28

733516

edited Dec 14 '18 at 18:57

Maxim

31.9k2177128

asked Nov 22 '18 at 4:48

random_28

733516

edited Dec 14 '18 at 18:57

Maxim

31.9k2177128

edited Dec 14 '18 at 18:57

Maxim

31.9k2177128

edited Dec 14 '18 at 18:57

Maxim

31.9k2177128

asked Nov 22 '18 at 4:48

random_28

733516

asked Nov 22 '18 at 4:48

random_28

733516

asked Nov 22 '18 at 4:48

random_28

733516

add a comment |

1 Answer
1

active

oldest

votes

If you just don't want to compute gradients for your TransformationFunction, it is easiest to turn off gradient computation for all parameters involved in this computation by setting the requires_grad flag to False.

Excluding subgraphs from backward:

If there’s a single input to an operation that requires gradient, its
output will also require gradient. Conversely, only if all inputs
don’t require gradient, the output also won’t require it. Backward
computation is never performed in the subgraphs, where all Tensors
didn’t require gradients.

This is especially useful when you want to freeze part of your model,
or you know in advance that you’re not going to use gradients w.r.t.
some parameters. For example if you want to finetune a pretrained CNN,
it’s enough to switch the requires_grad flags in the frozen base, and
no intermediate buffers will be saved, until the computation gets to
the last layer, where the affine transform will use weights that
require gradient, and the output of the network will also require
them.

Here is a small example which would do so:

import torch

import torch.nn as nn



# define layers

normal_layer = nn.Linear(5, 5)

TransformationFunction = nn.Linear(5, 5)

# disable gradient computation for parameters of TransformationFunction

# here weight and bias

TransformationFunction.weight.requires_grad = False

TransformationFunction.bias.requires_grad   = False



# input 

inp = torch.rand(1, 5)



# do computation

out = normal_layer(inp)

out = TransformationFunction(out)



# loss

loss = torch.sum(out)

# backward

loss.backward()



# gradient for l1

print('Gradients for "normal_layer"', normal_layer.weight.grad, normal_layer.bias.grad)

# gradient for l2

print('Gradients for "TransformationFunction"', TransformationFunction.weight.grad, TransformationFunction.bias.grad)

Output:

Gradients for "normal_layer" tensor([[0.1607, 0.0215, 0.0192, 0.2595, 0.0811],

        [0.0788, 0.0105, 0.0094, 0.1272, 0.0398],

        [0.1552, 0.0207, 0.0186, 0.2507, 0.0784],

        [0.1541, 0.0206, 0.0184, 0.2489, 0.0778],

        [0.2945, 0.0393, 0.0352, 0.4756, 0.1486]]) tensor([0.2975, 0.1458, 0.2874, 0.2853, 0.5452])

Gradients for "TransformationFunction" None None

I hope this is what you were looking for, if not please edit your question with more detail!

answered Nov 22 '18 at 9:27

blue-phoenox

4,251101745

Although the parameters of TransformationFucttion are non-trainable, the transformation done by TransformationFunction will have its effect on the gradient computation for previous layers. Right ? I actually want to avoid this.

– random_28
Nov 23 '18 at 22:13

@random_28 Thanks for your reply. You are right, and I'm afraid there is no way out of the box to avoid this, at least for the following layers you will change the inputs and thus the gradients. There might be workarounds, but since I'm not exactly sure what you want to do it is difficult to find something suitable. Why do you do you want to make computations during training time which you don't want to include in the graph. Could explain what exactly you wan't to do? It would be great/helpful if you could supplement your question with a short working code example and your desired output.

– blue-phoenox
Nov 24 '18 at 12:17

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53424056%2fapply-transformation-for-tensor-without-including-it-in-backward%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

Excluding subgraphs from backward:

If there’s a single input to an operation that requires gradient, its
output will also require gradient. Conversely, only if all inputs
don’t require gradient, the output also won’t require it. Backward
computation is never performed in the subgraphs, where all Tensors
didn’t require gradients.

This is especially useful when you want to freeze part of your model,
or you know in advance that you’re not going to use gradients w.r.t.
some parameters. For example if you want to finetune a pretrained CNN,
it’s enough to switch the requires_grad flags in the frozen base, and
no intermediate buffers will be saved, until the computation gets to
the last layer, where the affine transform will use weights that
require gradient, and the output of the network will also require
them.

Here is a small example which would do so:

import torch

import torch.nn as nn



# define layers

normal_layer = nn.Linear(5, 5)

TransformationFunction = nn.Linear(5, 5)

# disable gradient computation for parameters of TransformationFunction

# here weight and bias

TransformationFunction.weight.requires_grad = False

TransformationFunction.bias.requires_grad   = False



# input 

inp = torch.rand(1, 5)



# do computation

out = normal_layer(inp)

out = TransformationFunction(out)



# loss

loss = torch.sum(out)

# backward

loss.backward()



# gradient for l1

print('Gradients for "normal_layer"', normal_layer.weight.grad, normal_layer.bias.grad)

# gradient for l2

print('Gradients for "TransformationFunction"', TransformationFunction.weight.grad, TransformationFunction.bias.grad)

Output:

Gradients for "normal_layer" tensor([[0.1607, 0.0215, 0.0192, 0.2595, 0.0811],

        [0.0788, 0.0105, 0.0094, 0.1272, 0.0398],

        [0.1552, 0.0207, 0.0186, 0.2507, 0.0784],

        [0.1541, 0.0206, 0.0184, 0.2489, 0.0778],

        [0.2945, 0.0393, 0.0352, 0.4756, 0.1486]]) tensor([0.2975, 0.1458, 0.2874, 0.2853, 0.5452])

Gradients for "TransformationFunction" None None

I hope this is what you were looking for, if not please edit your question with more detail!

answered Nov 22 '18 at 9:27

blue-phoenox

4,251101745

Although the parameters of TransformationFucttion are non-trainable, the transformation done by TransformationFunction will have its effect on the gradient computation for previous layers. Right ? I actually want to avoid this.

– random_28
Nov 23 '18 at 22:13

@random_28 Thanks for your reply. You are right, and I'm afraid there is no way out of the box to avoid this, at least for the following layers you will change the inputs and thus the gradients. There might be workarounds, but since I'm not exactly sure what you want to do it is difficult to find something suitable. Why do you do you want to make computations during training time which you don't want to include in the graph. Could explain what exactly you wan't to do? It would be great/helpful if you could supplement your question with a short working code example and your desired output.

– blue-phoenox
Nov 24 '18 at 12:17

add a comment |

Excluding subgraphs from backward:

If there’s a single input to an operation that requires gradient, its
output will also require gradient. Conversely, only if all inputs
don’t require gradient, the output also won’t require it. Backward
computation is never performed in the subgraphs, where all Tensors
didn’t require gradients.

This is especially useful when you want to freeze part of your model,
or you know in advance that you’re not going to use gradients w.r.t.
some parameters. For example if you want to finetune a pretrained CNN,
it’s enough to switch the requires_grad flags in the frozen base, and
no intermediate buffers will be saved, until the computation gets to
the last layer, where the affine transform will use weights that
require gradient, and the output of the network will also require
them.

Here is a small example which would do so:

import torch

import torch.nn as nn



# define layers

normal_layer = nn.Linear(5, 5)

TransformationFunction = nn.Linear(5, 5)

# disable gradient computation for parameters of TransformationFunction

# here weight and bias

TransformationFunction.weight.requires_grad = False

TransformationFunction.bias.requires_grad   = False



# input 

inp = torch.rand(1, 5)



# do computation

out = normal_layer(inp)

out = TransformationFunction(out)



# loss

loss = torch.sum(out)

# backward

loss.backward()



# gradient for l1

print('Gradients for "normal_layer"', normal_layer.weight.grad, normal_layer.bias.grad)

# gradient for l2

print('Gradients for "TransformationFunction"', TransformationFunction.weight.grad, TransformationFunction.bias.grad)

Output:

Gradients for "normal_layer" tensor([[0.1607, 0.0215, 0.0192, 0.2595, 0.0811],

        [0.0788, 0.0105, 0.0094, 0.1272, 0.0398],

        [0.1552, 0.0207, 0.0186, 0.2507, 0.0784],

        [0.1541, 0.0206, 0.0184, 0.2489, 0.0778],

        [0.2945, 0.0393, 0.0352, 0.4756, 0.1486]]) tensor([0.2975, 0.1458, 0.2874, 0.2853, 0.5452])

Gradients for "TransformationFunction" None None

I hope this is what you were looking for, if not please edit your question with more detail!

answered Nov 22 '18 at 9:27

blue-phoenox

4,251101745

Although the parameters of TransformationFucttion are non-trainable, the transformation done by TransformationFunction will have its effect on the gradient computation for previous layers. Right ? I actually want to avoid this.

– random_28
Nov 23 '18 at 22:13

@random_28 Thanks for your reply. You are right, and I'm afraid there is no way out of the box to avoid this, at least for the following layers you will change the inputs and thus the gradients. There might be workarounds, but since I'm not exactly sure what you want to do it is difficult to find something suitable. Why do you do you want to make computations during training time which you don't want to include in the graph. Could explain what exactly you wan't to do? It would be great/helpful if you could supplement your question with a short working code example and your desired output.

– blue-phoenox
Nov 24 '18 at 12:17

add a comment |

Excluding subgraphs from backward:

If there’s a single input to an operation that requires gradient, its
output will also require gradient. Conversely, only if all inputs
don’t require gradient, the output also won’t require it. Backward
computation is never performed in the subgraphs, where all Tensors
didn’t require gradients.

This is especially useful when you want to freeze part of your model,
or you know in advance that you’re not going to use gradients w.r.t.
some parameters. For example if you want to finetune a pretrained CNN,
it’s enough to switch the requires_grad flags in the frozen base, and
no intermediate buffers will be saved, until the computation gets to
the last layer, where the affine transform will use weights that
require gradient, and the output of the network will also require
them.

Here is a small example which would do so:

import torch

import torch.nn as nn



# define layers

normal_layer = nn.Linear(5, 5)

TransformationFunction = nn.Linear(5, 5)

# disable gradient computation for parameters of TransformationFunction

# here weight and bias

TransformationFunction.weight.requires_grad = False

TransformationFunction.bias.requires_grad   = False



# input 

inp = torch.rand(1, 5)



# do computation

out = normal_layer(inp)

out = TransformationFunction(out)



# loss

loss = torch.sum(out)

# backward

loss.backward()



# gradient for l1

print('Gradients for "normal_layer"', normal_layer.weight.grad, normal_layer.bias.grad)

# gradient for l2

print('Gradients for "TransformationFunction"', TransformationFunction.weight.grad, TransformationFunction.bias.grad)

Output:

Gradients for "normal_layer" tensor([[0.1607, 0.0215, 0.0192, 0.2595, 0.0811],

        [0.0788, 0.0105, 0.0094, 0.1272, 0.0398],

        [0.1552, 0.0207, 0.0186, 0.2507, 0.0784],

        [0.1541, 0.0206, 0.0184, 0.2489, 0.0778],

        [0.2945, 0.0393, 0.0352, 0.4756, 0.1486]]) tensor([0.2975, 0.1458, 0.2874, 0.2853, 0.5452])

Gradients for "TransformationFunction" None None

I hope this is what you were looking for, if not please edit your question with more detail!

answered Nov 22 '18 at 9:27

blue-phoenox

4,251101745

Excluding subgraphs from backward:

If there’s a single input to an operation that requires gradient, its
output will also require gradient. Conversely, only if all inputs
don’t require gradient, the output also won’t require it. Backward
computation is never performed in the subgraphs, where all Tensors
didn’t require gradients.

This is especially useful when you want to freeze part of your model,
or you know in advance that you’re not going to use gradients w.r.t.
some parameters. For example if you want to finetune a pretrained CNN,
it’s enough to switch the requires_grad flags in the frozen base, and
no intermediate buffers will be saved, until the computation gets to
the last layer, where the affine transform will use weights that
require gradient, and the output of the network will also require
them.

Here is a small example which would do so:

import torch

import torch.nn as nn



# define layers

normal_layer = nn.Linear(5, 5)

TransformationFunction = nn.Linear(5, 5)

# disable gradient computation for parameters of TransformationFunction

# here weight and bias

TransformationFunction.weight.requires_grad = False

TransformationFunction.bias.requires_grad   = False



# input 

inp = torch.rand(1, 5)



# do computation

out = normal_layer(inp)

out = TransformationFunction(out)



# loss

loss = torch.sum(out)

# backward

loss.backward()



# gradient for l1

print('Gradients for "normal_layer"', normal_layer.weight.grad, normal_layer.bias.grad)

# gradient for l2

print('Gradients for "TransformationFunction"', TransformationFunction.weight.grad, TransformationFunction.bias.grad)

Output:

Gradients for "normal_layer" tensor([[0.1607, 0.0215, 0.0192, 0.2595, 0.0811],

        [0.0788, 0.0105, 0.0094, 0.1272, 0.0398],

        [0.1552, 0.0207, 0.0186, 0.2507, 0.0784],

        [0.1541, 0.0206, 0.0184, 0.2489, 0.0778],

        [0.2945, 0.0393, 0.0352, 0.4756, 0.1486]]) tensor([0.2975, 0.1458, 0.2874, 0.2853, 0.5452])

Gradients for "TransformationFunction" None None

I hope this is what you were looking for, if not please edit your question with more detail!

answered Nov 22 '18 at 9:27

blue-phoenox

4,251101745

answered Nov 22 '18 at 9:27

blue-phoenox

4,251101745

answered Nov 22 '18 at 9:27

blue-phoenox

4,251101745

answered Nov 22 '18 at 9:27

blue-phoenox

4,251101745

Although the parameters of TransformationFucttion are non-trainable, the transformation done by TransformationFunction will have its effect on the gradient computation for previous layers. Right ? I actually want to avoid this.

– random_28
Nov 23 '18 at 22:13

@random_28 Thanks for your reply. You are right, and I'm afraid there is no way out of the box to avoid this, at least for the following layers you will change the inputs and thus the gradients. There might be workarounds, but since I'm not exactly sure what you want to do it is difficult to find something suitable. Why do you do you want to make computations during training time which you don't want to include in the graph. Could explain what exactly you wan't to do? It would be great/helpful if you could supplement your question with a short working code example and your desired output.

– blue-phoenox
Nov 24 '18 at 12:17

add a comment |

Although the parameters of TransformationFucttion are non-trainable, the transformation done by TransformationFunction will have its effect on the gradient computation for previous layers. Right ? I actually want to avoid this.

– random_28
Nov 23 '18 at 22:13

@random_28 Thanks for your reply. You are right, and I'm afraid there is no way out of the box to avoid this, at least for the following layers you will change the inputs and thus the gradients. There might be workarounds, but since I'm not exactly sure what you want to do it is difficult to find something suitable. Why do you do you want to make computations during training time which you don't want to include in the graph. Could explain what exactly you wan't to do? It would be great/helpful if you could supplement your question with a short working code example and your desired output.

– blue-phoenox
Nov 24 '18 at 12:17

Although the parameters of TransformationFucttion are non-trainable, the transformation done by TransformationFunction will have its effect on the gradient computation for previous layers. Right ? I actually want to avoid this.

– random_28
Nov 23 '18 at 22:13

@random_28 Thanks for your reply. You are right, and I'm afraid there is no way out of the box to avoid this, at least for the following layers you will change the inputs and thus the gradients. There might be workarounds, but since I'm not exactly sure what you want to do it is difficult to find something suitable. Why do you do you want to make computations during training time which you don't want to include in the graph. Could explain what exactly you wan't to do? It would be great/helpful if you could supplement your question with a short working code example and your desired output.

– blue-phoenox
Nov 24 '18 at 12:17

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu