It's difficult to define exactly what 'creative coding' means. In broad terms, it refers to the production of something that is expressive in nature rather than having a purely practical use. It is a chance to learn and explore how various different technologies can come together to create impressive works of art.
What form that takes is entirely up to the creator. Anything from data visualisation to image manipulation could be considered 'creative coding'. The web is flexible enough to open up opportunities in different areas of art and design.
Here we look at the different ways you can experiment with code and create stimulating visual results. Want to give it a try? Take a look around the best code editors the web has to offer and get stuck in! Want to keep things simple? Try a website builder. Either way, you'll want to get your web hosting service right.
01. Head straight for the frameworks
If you're comfortable with code and have a good idea of what you want to create, why wait? Here are the four best frameworks for creative coders; take your pick!
p5.js
Processing is the holy grail for creative coders. It provides a language and an IDE to enable designers and non-technical people to create stunning visualisations without getting buried in the technicalities.
The p5.js library takes the principles of Processing and applies them to the web. It delivers the same kinds of abstractions, which provide simplicity to the beginners and offer powerful control to experts.
The bulk of the work happens within two functions – one controls the project setup, while the other draws to the page in a loop for the life of that project. The library supplies plenty of global methods and variables to update the page within these functions. For example, calling frameRate(60) lets p5.js do the hard work of maintaining a constant 60fps.
Get the Creative Bloq Newsletter
Daily design news, reviews, how-tos and more, as picked by the editors.
sketch.js
Weighing in at under 5kb, sketch.js is a tiny framework that helps developers get up and running with creative JavaScript. It's light enough to embed in any website, but provides plenty of features along with it.
It's possible to hook into methods that react to events within the projects. While this includes the usual callbacks, such as the animation loop, sketch.js also opens up events such as keyboard input and window resizing.
Touch and mouse inputs are treated the same, which makes all projects touch-friendly by default. While it is possible to target either input type specifically, it takes the worry out of dealing with the small differences between the two event types. It can also calculate the deltas between these points automatically to make physics calculations easier.
D3
While D3 has long been the go-to library for creating graphs and charts, that isn't all it's capable of. Its data-driven approach makes it great for creating attractive visualisations – either informative or abstract. When combined with a rich data source such as the Twitter API, it makes infographics easy to create.
By manipulating DOM elements, such as SVG, D3 will work with any assets already a part of an application. It provides a structure around which they can be transformed to provide emphasis or to animate onscreen.
There are plenty of ways to access D3 through other libraries and frameworks as well. Specially created components for React, directives for Angular, and plugins for most other front-end frameworks are readily available, for example.
three.js
Working in 3D with WebGL can be complicated at the best of times. Working out how shaders and camera perspectives work in such a wide-reaching API can take away from the creative aspects of a project and slow everything down.
With three.js it's possible to skim over these tricky aspects and focus on the visuals. It is a 3D JavaScript library, which works with WebGL to easily create beautifully smooth interactions with minimum effort.
The library provides many built-in methods for creating objects in the scene. Everything can be altered just like any other JavaScript object and will update accordingly. Complicated techniques, such as texture mapping, are provided out of the box and are usually just a case of setting the appropriate option on the object.
02. Get smart with data sources and APIs
There's plenty of publicly-accessible data out there for you to play with keep your project fresh; see what you can do with these sources.
Twitter is a wealth of information. People are talking about every conceivable topic by posting pictures and video alongside their unfiltered reactions in real time. This makes it a goldmine for a creative coding project. The official API can bring back the tweets in the format needed along with other metadata that is open for analysis. This data can then be visualised in new and exciting ways.
The Twitter API provides four defined 'objects' that can be retrieved – tweets, users, locations and entities. Entities include extra information linked to those objects including hashtags and media. Values on these objects are fixed and will only ever be extended upon, which means you will never lose access.
Tweets themselves can be searched for with a query to a single endpoint. These searches can either be for a string or can be more descriptive, such as for images or tweets with a positive sentiment.
Visualising Twitter data can make a great connected design piece. Combining multiple data points such as location and sentiment can create a project that is also informative, such as graphing live tweets about a topic from around the world.
While there are plenty of opportunities to use the output from Twitter, the API also allows for applications to post messages to Twitter. These can come from any source provided that it's connected to the API.
All endpoints in the API require some kind of authentication. For most projects that is only an application key, but authenticating can be tricky when done manually. Thankfully there are a few packages that make working with the Twitter API a little easier.
The npm module 'twitter' is the catch-all client library when working through Node. It supports regular endpoints as well as streaming, which may be useful for real-time projects.
Instagram and Flickr
Creative coding projects are often visual, which makes image and video sharing APIs an attractive prospect for inspiration. Thankfully, sites such as Instagram and Flickr open up their data and allow developers access to that content, as well as the surrounding metadata.
Instagram has become one of the web's most popular image-sharing platforms. As a result, it is a rich source of images to use in creative projects (got images to save? Keep them safe in cloud storage). The API is no different. It provides access to images, videos, comments and tags, as well as ways of searching through this information to find what is needed for the project.
Access within the API will be initially limited to a small pool of accounts and images. To remove some of these restrictions, Instagram will need to review the project themselves, which can be time-consuming and fruitless for creative coding projects. Depending on the project, the initial sandbox mode may suffice.
Flickr has a more readily accessible API. It is a resource of high-quality images available at multiple resolutions. Many of the photo endpoints only require an application key to work, which makes set up and fetching much easier.
The 'flickr.photos.search' endpoint is the most commonly used and gives access to most of Flickr's content. It can be filtered and sorted by date, location and even license as necessary. Other endpoints can get more detail on the photo, such as comments or EXIF data
Getting the API to work exactly how the project needs it to can be difficult, particularly if writing information back to Flickr. Luckily, plenty of packages exist to help use it with JavaScript, including the company's own 'flickr-sdk' on npm.
It's important to note that all images remain the property of their owners. For personal projects this will not be an issue, but if it's being shared be sure to either seek permission or credit the owner, depending on the terms set out by the photographer.
Camera and microphone
The web is usually limited to pointing and clicking with a mouse, but that is not the only way of providing input. Today's browsers are equipped with new APIs to pull in data from different external sources, including cameras and microphones.
Making projects that are visually aware is a great way to get users involved. By using vision as an input over a traditional keyboard and mouse, users are able to interact in different ways, such as face tracking or image recognition. As most devices now come with a camera built in, it is no longer the barrier it once was and works out great for the web.
Listening to the user is also a great alternative to manual input. Voice recognition could control navigation, or users could provide their own audio samples to use within their experience. It could also serve as an alternative to a button press. By extracting pitch and volume, these can then be mapped to what normally would be a button press, which could be used to control anything from colour to movement.
Access to a camera and microphone is possible through the navigator object.
navigator.mediaDevices.getUserMedia({
audio: true,
video: { width: 640, height: 480 }
})
.then(function(stream) {
// Access input
})
.catch(function(error) {
// Provide fallback option
});The code has to specify exactly what is needed from the input devices. The call to getUserMedia() will trigger a permissions dialog that the user must accept before continuing. Once accepted, these features are available as streams, making them more memory efficient.
All major browsers support this API, including mobile. If the browser cannot meet the exact requirements, the promise will reject and not work. It's important to supply an alternative experience, such as an image upload form, where access is not possible.
03. Mobile device sensors
The world of mobile presents a completely different set of opportunities when it comes to creative coding. The wide variety of form factors and sensors open for use within the browser can make for some distinctive experiences.
Mobile devices are almost exclusively touch interfaces. Browsers have the ability to detect and track multiple touches at once. This means that projects do not have to be led by a single point and can be manipulated in a much more intuitive way.
window.addEventListener("touchstart", e => {
e.preventDefault();
const touches = e.changedTouches;
});All touch events use a 'changedTouches' property on the event. This holds a reference to all the touch points that changed dependent on the type of event, rather than a separate event for each touch point. By tracking these, it's possible to map gestures or paint with touch.
Accelerometers and gyroscopes are mostly used for changing from portrait to landscape mode, but these are also up for interpretation. Different methods of interaction can be created by using the device itself as a control mechanism – whether that in specific co-ordinate movements or more gesture-based, such as a shake.
window.addEventListener("deviceorientation", e => {
console.log(e.gamma);
});
window.addEventListener("devicemotion", e => {
console.log(e.acceleration);
});APIs provide access to this information, but the values returned will vary by browser as they do not all use the same coordinate system. Libraries such as p5.js provide special values and hooks like 'rotationX' or 'deviceShaken()' to help abstract away the differences.
Mobile devices also have the ability to determine their exact position using geolocation. Being able to get the exact location of a device can open up new possibilities and make for a more streamlined experience.
navigator.geolocation.
getCurrentPosition(pos => {
console.log(pos.coords); });Access is provided through the navigator object. The value returned is the lat-long co-ordinates of the device, along with other data such as altitude or speed if the device supports it. Browsers will use the fastest method of detecting location, such as the IP address. However, this may not always be the most accurate.
Combining these inputs in creative ways is the key to making something special. For example, using the device's location and orientation to create a virtual stargazing experience.
04. Multi-screen experiences
Desktop browsers provide reassuring familiarity when it comes to creative projects. But by experimenting with different displays, users can get a more personalised experience that works for them.
The principles of responsive design still hold true for creative coding projects as well. Users should be able to enjoy them regardless of device, whether that's desktop, mobile, or projected on a giant screen.
These differences can also be used to enhance the project further. By using media queries in CSS, mobile users could get an experience tailored to the smaller, handheld form factor. Since mobile users tend to be just one person, these visitors could be offered a more personalised view of a bigger platform, for example.
Paper Planes, by company Active Theory, encouraged users to create virtual planes on their phones. These could then be 'thrown' and show up on the desktop version of the site. There, visitors could watch as planes flew around the world and see where they had come from.
Interactions such as these can be made easier with WebSockets. Communication between a browser and the server is event based, avoiding the need to poll the server, which can often be wasteful and prone to delays. Projects such as Socket.io can make setting up WebSocket protocols easier.
Direct communication between devices can also have a powerful effect when users are together in the same room. Seb Lee-Delisle's PixelPhones project, for example, turned a crowd of screens into a makeshift display. Once all were connected, everybody became part of the experience.
WebRTC is a set of JavaScript APIs that makes real-time communication between browsers easier. It needs a server to set up a connection, media is sent directly between browsers, which makes interactions quick and simple to create. It's now supported by all major browsers both on desktop and mobile.
05. Self-generative art
Even the most creative of coding projects can become stale after a while. An element of randomness can keep things fresh each time a piece of code runs. While that can come from user input, it can be interesting to see what code is capable off when it's let off of the leash.
Instead of defining what the output of a block of code would be, define a set of rules for it to follow. When a random starting position is defined, the end results will vary completely.
Conway's Game of Life is a great place to start. In a defined grid, each square can either be 'on' or 'off' depending on the squares around it. By colouring the squares dependant on their state, it generates an image. Repeating the process multiple times shows how groups of squares mutate over time. Tweaking the rules on which they change can dramatically alter the outcome.
Using JavaScript with a <canvas> element makes this visual process fairly simple. By using requestAnimationFrame(), JavaScript can re-evaluate its environment each frame. From there it is a case of using the output of the last frame as input for the next frame and let the program take care of itself.
06. WebVR and Bluetooth
While browsers are getting more capable and powerful every day, there is only so much that one viewport can provide. Thankfully, browsers are also expanding out of the browser too.
Virtual Reality has the ability to immerse the viewer in ways not possible before. WebVR is an open specification that allows access to these immersive worlds to the masses through the browser. It also helps bridge the gap between different device types, such as Google Cardboard and the HTC Vive.
Frameworks, such as A-Frame, can help bypass any complications by providing ready-to-go building blocks for VR experiences. Since these are built for performance and reliability, they leaves the creator free to make a great user experience.
Bluetooth is another option and an opportunity to forego a screen entirely. Chips are readily available and can be combined with different output devices to emit sound and light – all controlled by the browser through the Web Bluetooth API. The interface is promise based, which makes asynchronous communication somewhat easier.
Browser support varies as these technologies evolve. WebVR is currently supported in various states by development builds of Edge, Firefox and Chrome. Web Bluetooth only has support from Chrome right now, but others are considering it.
This article was originally published in Web Designer, the creative web design magazine. Subscribe to Web Designer here.
Related articles:
