Published in · 13 min read · Jul 16, 2019
--
I’ve written previously about bringing D3 into web applications here, looking at how to bind D3 visuals to UI elements. The purpose was to encourage moving beyond stand-alone visuals and get people prototyping fuller applications. Real applications solicit feedback because they get used, helping us validate analyses beyond usual statistical measures. IMO if you’re not building a real product you’re not really learning/doing data science.
The previous article still stands, but D3 is changing directions towards what it calls Observable (formally known as d3.express). Observable provides a playground of sorts, allowing users to modify D3 code online inside a notebook. For those who use Jupyter Notebooks you will find the experience similar. It’s essentially a REPL for doing D3.
Observable opens D3 up to true development since it now provides the ability to download your tailored D3 visual as a standalone “package” (a tarball file) that you can embed inside your application. Observable comes with its own runtime and a standard library, which provides helpful functions for working with HTML, SVG, generators, files and promises.
Here is the Observable documentation:
…and an opinionated writeup about their approach:
You can find examples of visuals here, which you can immediately start playing around with in your browser. When you’re looking to create a new visualization visit the following site, choose a project, edit the visual as needed, and embed inside your application.
In this article we’ll look at the following topics:
- creating a quick app layout based off 2 simple mockups;
- crafting components for our app’s UI elements;
- embedding Observable inside our app;
- sending data between our app and Observable;
- using Google’s Book API.
You can view the simple application here.
Let’s get started.
Let’s make a simple app that uses the Goodreads dataset hosted on Kaggle to allow people to explore book titles. The dataset lists book titles, their authors, ISBNs, and a few simple features like ratings.
The Mockup
We’ll allow the user to see a table of the original data and provide filtering functionality so users can search the table by author, ISBN number, and language.
We’ll also fetch book attributes from Google’s Book API and showcase them in a bar chart. The API also provides an image URL of the selected book, so we will show the book cover when the user searches by ISBN:
We’ll stitch together Observable and Google’s Book API into a real application using Azle.
The Directory Structure
We first create the following directory structure for our application:
app
├── data
├── scripts
├── d3_visuals
── table
── bar_chart
├── index.htmlBold names are empty folders and the index.html file is the usual Azle starting point:
We’ll add the needed files throughout this article. For now start a simple web server inside the app folder by running the following in a terminal session:
python3 -m http.server…then point your browser to localhost:
http://localhost:8000I’ll use Azle to create the scaffolding of my application. I created Azle because it’s fast to use, easy to understand, lightweight, flexible and free, and makes stitching together libraries and frameworks easy. But you can use whatever JS tools you like.
1.1 Creating Application Layout
We create layouts using Azle’s az.add_layout function. This is how we create a grid on our page. I’ll place my layout code in Azle’s index.html file:
Read through the above code and you can easily tell how the page is being constructed. Every Azle function takes a “target_class” and target_instance to add an element to the DOM. It also takes an object with properties. If we’re adding an element it’s a content object, and if we’re styling an element it’s a style object (usual CSS styles).
The above code produces the following:
We can see how layouts will allow us to position elements by demarcating areas on the screen. Let’s color our main section so it blends with the body color. We pass in usual CSS styling as properties to the style object:
az.style_sections('my_sections', 1, {
"background": "rgb(51, 47, 47)",
"height": "auto"
})Let’s also add a dark color to the background of our visual_layout:
az.style_layout('visual_layout', 1, {
"align": "center",
"background": "rgb(39, 36, 36)",
"border-radius": "4px",
"height": "460px",
"margin-top": "10px",
"column_widths": ['65%', '35%'],
"border": 3
})Now our application looks like this:
All our layout cells are waiting for their content. That’s where components come in.
1.2 Create Application Components
Components are combined UI elements, styling, and events. It’s like packaging up all the code necessary to create a specific part of our application. For example, if we wanted to have a calendar in our app we would create a calendar component and place it in one of our layout cells.
Creating components makes our code modular, easy to reuse, easier to maintain, and enables us to pivot our application more readily when ideas change. While we will be creating our components in Azle, these could also be React components added to Azle layouts.
From our mockup above we know we need search bars, icons, a dropdown menu, an image, and the D3 visuals. Showing how we create every component for this application is beyond the scope of this article. You can view the full application code here. We’ll create a few of the main ones in this article.
We place all component code inside the az.components object:
az.components = {}
Create a file called component.js and add the following code:
Looks like a lot but it reads easily. Note how we first add and style a layout, just like we did above, this time to hold our input box and search icon. We then add and style an input element into the first layout cell, then add and style the search icon into the second layout cell. Finally we add an event to our search icon so when the user clicks it, something happens.
Don’t worry about all the things being done inside our click event, we will address those after we embed our D3 visuals. Let’s now create our components for adding D3 visuals to our app. Here’s the table example:
It looks a tad callback hellish but it’s readable. We add and style an iframe, wait until the iframe is done loading, ensure the full dataset is available, then post our message to the iframe. Message posting is how we communicate with D3. We will discuss this in the next section.
Adding our components works the same way Azle adds elements to an application:
az.components.d3_table(target_class, target_instance)The above line will thus add our iframe to one of the cells in our app scaffolding. There aren’t any visuals to show inside those frames yet, so let’s go grab our Observable visuals, then we’ll target them into their proper cells.
Embedding Observable is as simple as downloading the tarball of the desired visual, then hosting its index.html file inside an iframe. This isn’t the only way to bring Observable into an application, but it’s quick and works well for rapid prototyping.
2.1 Get Visuals from Observable
We need a table and bar chart. These are both available with a little online searching:
We download their tarballs by clicking on the 3 dots in the upper right:
Once downloaded, unzip the tarball and place inside its respective folder:
2.2 Establish Communication between App and D3
We need our application to communicate with our Observable visuals. In the previous section we alluded to the use of az.post_message_to_frame to do this. Here is the function:
Azle’s post_message_to_frame allows our application to send data and functions inside iframes (as long as everything is on the same server there should be no CORS issues).
The main.redefine comes from Observable itself, and is how we can redefine variables and data in the D3 visual. The D3 table started with its data object called “fakeData”, thus we need to replace this with our book data from the Kaggle dataset. You’ll notice we are passing in a parent function called filter_by_author, rather than the data itself. We’ll discuss this shortly.
The other half of this equation is how D3 accepts the posted message. For our hosted Observable to accept incoming messages from our application we must add an event listener to the index.html file of the Observable:
And the Azle library to the top:
<script src='https://azlejs.com/v2/azle.min.js'></script>So our D3 table index.html file should look like this:
To be clear, we are adding to this file:
The communication works because we are sending Observable’s main.redefine inside our frame using Azle’s post_message_to_frame, at which point our frame accepts that message and executes the parent.filter_by_author function (as shown in codeblock 5). The following figure depicts the concept:
Before we discuss parent functions let’s style our D3 visuals.
2.3 Add Styles to D3 Observable
Since we downloaded the entire notebook there will be notebook cells we do not want to appear in our app (they are used to tailor the visual as a REPL). We should remove those cells.
We can do that by removing the following line from Observable’s index.html file:
const main = runtime.module(define, Inspector.into(document.body));with the following:
const main = runtime.module(define, name => {
if (name == 'chart') {
return new Inspector(document.body)
}
});This way only the chart will be drawn.
We can also style the D3 visual to make it look more modern. Check out the table-with-nested-data.js file inside the table folder. Compare the original file to the one I have prepared here to see styles I’ve added.
Here is the difference:
If you inspect the bar-chart.js file inside the bar_chart folder you’ll see similar changes made.
We mentioned above the use of parent functions. In Figure 7 we see a 2-way communication between our application and Observable. Parent functions are how we call functions in our application from Observable.
We want Observable to redraw its D3 visual using new, filtered data. So we’ll filter the original dataset (that we read in previously) using functions called from inside our iframes.
Create a file called parent_functions.js and place inside the scripts folder:
app
├── data
├── scripts
── parent_functions.js
├── img
├── d3_visuals
── table
── bar_chart
├── index.htmlHere are the first 3 parent functions that return either the full dataset, data filtered by language, or data filtered by author.
Our frames will call these functions via codeblock 6 and replace the default visual dataset with the newly returned data.
Ai this point we’ve built a simple app layout based on our mockup, crafted our components, set up our click events inside our components containing main.redefine, added event handlers to our D3 index files, and created parent functions for returning full and filtered data. Also, we have our D3 Observable visuals waiting in their folders.
All that’s left for the D3 piece is to place our components into their target cells. We target our components like any other Azle function, using the target_class and target_instance of the destination layout cell.
First, we need to load our dataset into the application.
I downloaded the Kaggle Goodreads dataset as a CSV, then converted it into JSON using an online converter. I then downloaded the new JSON file and saved into the data folder of our app directory as books.json.
Now we can just use Azle’s read_local_file function to read in our dataset. Place it at the top of the index.html file:
I am saving the data as a new object called az.hold_value.full_dataset. You can use az.hold_value.[variable name] to make any variable or object globally available within the Azle namespace. Notice I am also “slicing” the data to limit the number of rows shown in the app. You can remove this to show all data (or better, make a component that allows users to control how many rows are loaded :) )
Now I will use Azle’s call_once_satisfied function to ensure the full dataset has been loaded prior to calling our components:
This will add all components to our app layout once the condition of an existing data object is defined.
Our table looks like this:
If you play with the app you’ll see users can filter the D3 table by author name and language. Many B2B applications benefit from this kind of functionality.
We still need to fulfill our mockup promise of showing a bar chart of information and the cover of the book, when a user searches by ISBN. Let’s do that now.
As mentioned earlier we can fetch book covers from ISBNs using the free Google Book API. A quick Google search points us to Stack Overflow answers showing how to use it (often faster than the usual documentation):
We want the bar chart and book cover to appear after the user searches by ISBN. If you look at the search_by_isbn function inside components.js and inspect its click event you’ll see how this is handled.
When a user pastes in an ISBN # and clicks the search icon 4 things happen:
- the fetch_book_cover component is called;
- the iframes have their displays toggled using the CSS display property inside az.style_iframe;
- a message is posted to the iframe holding the bar chart, with the data filtered by ISBN;
- the add_chart_buttons component is called to accompany the bar chart.
The fetch_book_cover component parses the data returned from Google’s Book API and retrieves the required image URL:
data.items[0].volumeInfo.imageLinks.thumbnail…which we can use as the image URL when we add the book cover to our app:
az.add_image('visual_layout_cells', 2, {
"this_class": "cover_img",
"image_path": data.items[0].volumeInfo.imageLinks.thumbnail
})If you look in component.js of the app code you will see a function for calling Google’s Book API. It uses Azle’s call_api function along with the following URL:
"https://www.googleapis.com/books/v1/volumes?q=isbn:" + az.grab_value('search_isbn_bar', 1)This simply concatenates the API url with the ISBN added by the user to the input field with class name “search_isbn_bar”.
Users can thus copy an ISBN number from the table and paste it into the search by isbn input field, which will draw the bar chart and show the book’s cover:
Users can also change the bar chart view using the thin buttons below the bar chart. Here is the app gif again:
As mentioned earlier, you can read through the full codebase to see how the full app has been constructed. If you see a function that looks unfamiliar simply refer to Azle’s documentation. This article wanted to show the core pieces needed to bring D3 Observable into a real web application.
In this article we looked at embedding D3 Observable inside a web application. We used Azle to stitch together Observable and Google’s Book API to create a real application that allows users to search and explore book titles.
Our tasks involved creating a quick layout based off 2 simple mockups, crafting and targeting components, embedding an Observable table and bar chart inside frames, communicating between app and Observable using message posting and event handlers, and finally using Google’s Book API to populate the bar chart and show the book cover after searching by ISBN.
We only covered the major parts needed to connect D3 to apps. There is more code you can explore on the GitHub project. I encourage you to create your own applications, and find ways to bring your analyses to users via real products.
If you enjoyed this article you might also like:
