How to create photo-realistic images from rough sketches

It doesn't seem that long since Google unleashed its DeepDream technology and the nightmarish image possibilities of deep learning neural networks, and gave birth to this harrowing designer quiz.

Since then the world of neural network-powered image trickery has been blazing ahead at a terrifying rate; a few months ago we introduced you to Neural Doodles, a way to turn simple doodles into fine art, and now there's a new technique that goes even further.

It doesn't have a catchy name just yet, but the possibilities of Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman and Alexei A. Efros' new image manipulation technology are incredibly exciting.

Their new technique, Generative Visual Manipulation on the Natural Image Manifold, tackles one of the major problems of realistic image manipulation: unless you're really skilled, it's hard to modify an image to any great extent while preserving the realism of the result.

In their terms, it's easy to fall off the natural image manifold while editing, and so they've come up with a way to learn the natural image manifold using a generative adversarial neural network.

We don't really understand that either, but thankfully the results speak for themselves. Using this new tool you can manipulate images in real time and even morph one object into the shape and colour of another, while still getting realistic results. In their video, they demonstrate this by taking shoes and handbags and morphing them into quite different shapes; it's a remarkable process to see in action.

Here you can see how this tech can morph one object into another

Here you can see how this tech can morph one object into another

You can use the technology to sketch new details onto existing objects - for examples, a set of laces on a shoe - and see them instantly appear in photorealistic form. And taking the technology further, you can use it to create entirely new photorealist images from rough sketches.

The video demonstrates how, with a few lines and some dashes of colour you can create a stunning mountain landscape, and then instantly adjust it until it's exactly what you want.

With a few doodled lines you can create an entire landscape

With a few doodled lines you can create an entire landscape

The possibilities are dizzying; the bottom line seems to be that it's only a matter of time before anyone will be able to create amazing artwork without any artistic skill whatsoever. And bear in mind that this project is funded and developed in part by Adobe, so you might see this technology - most likely, a much more refined and powerful version - turn up in a future edition of Photoshop CC.

For now, though, you can try out iGAN (Interactive Image Generation via Generative Adversarial Networks) for yourself, if you're feeling brave. Like so many of these cutting edge image tool, everything you need is available to download on GitHub, but you'll need to be comfortable with using Python and third-party libraries.

Let us know in the comments if you manage to create anything with it!

Thank you for reading 5 articles this month* Join now for unlimited access

Enjoy your first month for just £1 / $1 / €1

*Read 5 free articles per month without a subscription

Join now for unlimited access

Try first month for just £1 / $1 / €1

Jim McCauley

Jim McCauley is a writer, performer and cat-wrangler who started writing professionally way back in 1995 on PC Format magazine, and has been covering technology-related subjects ever since, whether it's hardware, software or videogames. A chance call in 2005 led to Jim taking charge of Computer Arts' website and developing an interest in the world of graphic design, and eventually led to a move over to the freshly-launched Creative Bloq in 2012. Jim now works as a freelance writer for sites including Creative Bloq, T3 and PetsRadar, specialising in design, technology, wellness and cats, while doing the occasional pantomime and street performance in Bath and designing posters for a local drama group on the side.