Fast neural style transfer II.

Fast neural style transfer II.

- 2 mins

Updates on previous posts

I wrote about neural style transfer previously in this and this post and I finally corrected my code for the latter to work properly. :)

Fast neural style transfer

Since I already wrote a blog post about fast neural style transfer I’ll just sum up the main idea here quickly. Taking a transform network and a loss network jointly where only the weights of the transform network is updated. After convergence the transfrom model can be used in a feed-forward manner to generate stilizied images.

Feature loss


\[L_{content}^{\phi, j} (\hat{y}, y) = \frac{1}{C_j H_j W_j} || \phi(\hat{y}) - \phi{y} || ^{2}_{2}\]

To make the content resamble the output image we simple take the Euclidian squared distance between the extracted feautres from the original image \(y\) and the transformed image \(\hat{y}\).

The style loss is a bit trickier. Since we want to keep the style of the style image and not its spatial features we drop spatial structure in a way that was introduced in the original neural style transfer paper the Gram matrix:

gram matrix

Where \(phi_j\) is a feature map extracted at some intermediate layer from the loss network. Given these Gram-metrices from several intermediate layers we can take the Forbenius-norm of the difference between the Gram-matrix of the style image through the loss net and the transformed image thourgh the loss net.

style loss


Van Gogh Picasso

To train the network with the imagenette dataset:

pipenv shell
pipenv install
pipenv run python --starry_night --width=<width_int> --height=<height-int>


Original Van Gogh Picasso


Van Gogh Picasso
mount-vesuvio mount-vesuvio
Van Gogh Picasso
outside-scene-with-friends outside-scene-with-friends

There are some burn-out probably due to the original images I’ve taken because the transfer model cannot handle the extreme pixel values. In the Van Gogh stilizied images the whites are overexposed while in the Picasso stilizied images the sky seems to mess with algorithm.

My implementation is available here

@Regards, Alex


[1] Original implementation by J. C. Johnson.

[2] A residual PyTorch implementation by E. Linder-Norén

[3] The research paper that introduced fast neural style transfer by J. C. Johnson

Alex Olar

Alex Olar

Christian, foodie, physicist, tech enthusiast

comments powered by Disqus
rss facebook twitter github gitlab youtube mail spotify lastfm instagram linkedin google google-plus pinterest medium vimeo stackoverflow reddit quora quora