How to create photo-realistic images from rough sketches

It doesn't seem that long since Google unleashed its DeepDream technology and the nightmarish image possibilities of deep learning neural networks, and gave birth to this harrowing designer quiz.

Since then the world of neural network-powered image trickery has been blazing ahead at a terrifying rate; a few months ago we introduced you to Neural Doodles, a way to turn simple doodles into fine art, and now there's a new technique that goes even further.

It doesn't have a catchy name just yet, but the possibilities of Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman and Alexei A. Efros' new image manipulation technology are incredibly exciting.

Their new technique, Generative Visual Manipulation on the Natural Image Manifold, tackles one of the major problems of realistic image manipulation: unless you're really skilled, it's hard to modify an image to any great extent while preserving the realism of the result.

In their terms, it's easy to fall off the natural image manifold while editing, and so they've come up with a way to learn the natural image manifold using a generative adversarial neural network.

We don't really understand that either, but thankfully the results speak for themselves. Using this new tool you can manipulate images in real time and even morph one object into the shape and colour of another, while still getting realistic results. In their video, they demonstrate this by taking shoes and handbags and morphing them into quite different shapes; it's a remarkable process to see in action.

You can use the technology to sketch new details onto existing objects - for examples, a set of laces on a shoe - and see them instantly appear in photorealistic form. And taking the technology further, you can use it to create entirely new photorealist images from rough sketches.

The video demonstrates how, with a few lines and some dashes of colour you can create a stunning mountain landscape, and then instantly adjust it until it's exactly what you want.

The possibilities are dizzying; the bottom line seems to be that it's only a matter of time before anyone will be able to create amazing artwork without any artistic skill whatsoever. And bear in mind that this project is funded and developed in part by Adobe, so you might see this technology - most likely, a much more refined and powerful version - turn up in a future edition of Photoshop CC.

For now, though, you can try out iGAN (Interactive Image Generation via Generative Adversarial Networks) for yourself, if you're feeling brave. Like so many of these cutting edge image tool, everything you need is available to download on GitHub, but you'll need to be comfortable with using Python and third-party libraries.

Let us know in the comments if you manage to create anything with it!