Fast neural style transfer II.

- 2 mins

I wrote about neural style transfer previously in this and this post and I finally corrected my code for the latter to work properly. :)

Fast neural style transfer

Since I already wrote a blog post about fast neural style transfer I’ll just sum up the main idea here quickly. Taking a transform network and a loss network jointly where only the weights of the transform network is updated. After convergence the transfrom model can be used in a feed-forward manner to generate stilizied images.

Feature loss

$L_{content}^{\phi, j} (\hat{y}, y) = \frac{1}{C_j H_j W_j} || \phi(\hat{y}) - \phi{y} || ^{2}_{2}$

To make the content resamble the output image we simple take the Euclidian squared distance between the extracted feautres from the original image $$y$$ and the transformed image $$\hat{y}$$.

The style loss is a bit trickier. Since we want to keep the style of the style image and not its spatial features we drop spatial structure in a way that was introduced in the original neural style transfer paper the Gram matrix:

Where $$phi_j$$ is a feature map extracted at some intermediate layer from the loss network. Given these Gram-metrices from several intermediate layers we can take the Forbenius-norm of the difference between the Gram-matrix of the style image through the loss net and the transformed image thourgh the loss net.

Results

Van Gogh Picasso

To train the network with the imagenette dataset:

pipenv shell
pipenv install
pipenv run python train.py --starry_night --width=<width_int> --height=<height-int>


Examples

• Heidelberg - images
Original Van Gogh Picasso

Videos

• Outside scene with friends
Van Gogh Picasso
• Mount Vesuvio’s inside
Van Gogh Picasso

There are some burn-out probably due to the original images I’ve taken because the transfer model cannot handle the extreme pixel values. In the Van Gogh stilizied images the whites are overexposed while in the Picasso stilizied images the sky seems to mess with algorithm.

My implementation is available here

@Regards, Alex

References

[2] A residual PyTorch implementation by E. Linder-Norén

[3] The research paper that introduced fast neural style transfer by J. C. Johnson