Web figures: uifigures and uihtml

As I mentioned in the previous post, the ultimate goal of this blog is to craft our own library of custom-made HTML components. However, before we start to design and style those components, we should first explore their containers: uifigures and uihtml elements. That’s the idea of todays post. We will be looking at webservers, websites and how does Matlab use them to display its HTML-based figures. And although this information won’t help us design our components, it will allow us to understand some limitation we’ll face when we start creating them.

Uifigures & web windows

If you’ve read last post, you’ll probably remember that I mentioned that uifigures are actually “a browser loading a web page”. How can we get more information about this web page and the browser? When trying to explore some hidden (undocumented) functionality, the first step I usually make is to open the profiler and investigate the classes and functions that Matlab uses internally.

There’s a lot to digest in the profilers flame graph. But there are some calls to a webwindow class that sounds exactly like what we are trying to find. To get a list of all the web windows currently opened (all the uifigures and apps) we will have to use the webwindowmanager:

f = uifigure(); drawnow(); 
wwManager = matlab.internal.cef.webwindowmanager.instance();
ww = wwManager.windowList(); 
% ww = wwManager.windowListInProc(); % if the uifigure is not yet rendered

The webwindow class has many properties and methods that look interesting, and we might use them in the future. For now, I want us to focus in 2 of its properties: ChromiumVersion and URL.

ChromiumVersion

The ChromiumVersion, as the name implies, tells us that Matlab is using a Chromium browser to render the page. Chromium is the open-source browser that some bigger browsers (such as Google Chrome, Opera or Edge) use as their base. When inspecting uifigures, we will see a lot of times the acronym CEF, which stands for “Chromium Embedded Framework”. This is just the framework that Mathworks uses to attach the Chromium browser into the panel that appears when we create a uifigure.

When we start designing our components, we will need to know which JS/CSS features are supported by the browser that will be rendering our page. These languages are continuously evolving, and so do the browsers evolve to implement the new language features.

To see if a certain feature is supported for a specific browser and version, the website https://caniuse.com/ will be our go-to. For me, using Matlab R2022b, I will need to check in that page if a given feature is available for version 99 (the value of the property) of Chrome browsers.

Note: We also have to take into account that uihtml was introduced in R2019b, and the Chromium version shipped with R2019b was 69. If we want our components to be backwards compatible up to that version of Matlab, we will need to stick to using features available in that older Chromium version.

URL

An URL specifies the location from where the browser is retrieving the HTML file. If you check the URL of your web window, it should look something like this:

https://127.0.0.1:31515/toolbox/matlab/uitools/uifigureappjs/cefComponentContainer.html?channel=/uifigure/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&websocket=on&syncMode=MF0ViewModel&snc=XXXXXX

An URL is composed of many parts, and each of them tells us something relevant of our figure:

• The first part is called the scheme, and it identifies the protocol that the browser uses to access the file. For our case, the scheme is https://, which means that the browser is going to receive the file from a server. Another scheme is file://, which would mean that the browser is loading the file from a local drive. For example, if you open a .html in your computer, it should open in a new tab of your default browser with this file:// protocol.
• Then we find the host, or who is serving the file, accompanied by the port number. Our host is 127.0.0.1, which is the IP address of our own computer (it may also appear as localhost, which is just an alias of that IP address). The port number (31515) just indicates the port at which the browser will connect to server.
• Then we have the path, which is the specific location of the file in that server. For our Matlab figure, the path of the HTML file being loaded is:
  toolbox/matlab/uitools/uifigureappjs/cefComponentContainer.html
  That file exists on your computer, inside Matlabs root directory. You can navigate to it and see the HTML source code of a uifigure, and you can also try to open the other JS/CSS resources that Matlab uses when creating its other standard components (the JS code is minified, so you’ll have trouble reading it).
• The last part is the query string, and it contains information that the file can use for its own purposes. In our case, we have some channel, websocket, synccMode and snc queries. The only thing relevant for us at this moment is the channel, which is formed by a pattern of 8-4-4-4-12 random characters. This is usually called an UUID (Universally Unique Identifier), and Matlab uses it to differentiate between the different open uifigures. We can get this same identifier from the Uuid property of an uifigure handle (simply uuid = f.Uuid), and use it to find the web window from the list provided by the webwindowmanager.

You may have noticed something strange here. Why are we opening a connection to a local web server when the files we are trying to load are in our local drive? Would’t it make more sense to serve the files using the file:// protocol? I made myself this question, and I could think of a problem related to security and what is called Cross Origin Resource Sharing (CORS) and Same Origin Policy (SOP).

In a nutshell, CORS controls the interaction between different origins. An origin is just the combination of the scheme, host and port number (https://127.0.0.1:31515 in our case). By default, CORS allows embedding data from other origins into our HTML (you can for example add <script>, <audio> or <img> tags whose source comes from another origin); but blocks fetching data from another origin (using JS to read a text file from another origin would be blocked, for example). You’d think that with the file:// protocol, all files in our drive would have all the same origin, but modern browsers (including Chromium) mark their origins as opaque for security reasons. That means that they are no longer considered to be from the same origin, and the CORS policy blocks us from things like reading a stored text or pdf file.

Web servers (like our localhost) have a defined origin, so they can actually fetch resources that are served by themselves (same origin!). Since the local server is hosting the Matlabs root directory (the parent of /toolbox/matlab/... ) everything below it will be accessible to the server.

Uihtml & iframes

We have talked so far a lot about uifigures, but when we create our components we won’t be messing with them. Instead, we will be using uihtml components. At their core, uihtml elements are just websites being served from the same host (localhost) and port (31515).

To get this information, we will need to take a different approach. Instead of using the profiler and looking for specific classes or functions, we’ll be using our own JS code inside a uihtml element. Let’s first create an HTML file in Matlabs path:

<!DOCTYPE html>
<html>
<head></head>

<body>
    <div id="textDiv"></div>
    <script type="text/javascript">
        let t = document.getElementById("textDiv");
        t.textContent = ...; // We will be changing this line! 
    </script>
</body>
</html>

If you are not familiar with HTML or JS, I simply added a box (<div>) element to the website. Then I inserted a JS that gets a reference to that <div> to later change its text contents.

Anytime we want to render the HTML we’ll go to Matlab, create a uihtml element inside a new uifigure, and set its HTMLSource property to the path of that HTML file.

f = uifigure();
h = uihtml(f, 'HTMLSource', 'myHTMLdemo.html');

Getting the URL

There are a lot of properties from our HTML website that we can retrieve through the Web APIs. To get the current document URL, we can simply write in our JS script:

t.textContent = document.URL;

Now we can run the Matlab lines and check the resulting figure. If everything went well, you should be seeing an URL somehow similar to the uifigure URL.

https://127.0.0.1:31515/static/riNm7zDI/d572df8b-591f-4582-b136-b17b53d7adae/myHTMLdemo.html?mre=https%3A%2F%2F127.0.0.1%3A31515%2F

We can check that the file is served from the same host and port. The path now is different, but we don’t have to worry too much. The important thing to keep in mind is that the server is hosting the folder that contains our myHTMLdemo.html file.

Mini-exercise: what happens if you open the HTML file in your browser? Can you see which protocol is being used? We will need to be aware of this and CORS when we are tempted to debug our component HTML files by opening them in the browser!

It’s just an <iframe>

Wait… so our main uifigure website is loading another website, from the same origin, with our uihtml source? Exactly! There is an HTML element whose role is just that: embedding websites inside other websites. They are called inline frames, or iframes, and created with an <iframe> tag.

Iframes have a couple of interesting properties:

1. The content of an <iframe> cannot overflow its boundaries, meaning that elements of its rendered document cannot escape outside its borders.
2. The document rendered by an <iframe> cannot access the code of its parent, unless they are from the same origin.

Usually, <iframe> elements are used for embedding ads in websites. The ad becomes some HTML code served from another server, and we are safe because we know that it won’t be able to access our sites data or mess with the layout of the elements.

For us, however, these properties mean something different:

• Firstly, we will need to be aware of the size and position of our components. If we have popups or tooltips, they may end up being cropped if the size of the <iframe> is not big enough.
• Secondly, our frames are served from the same origin, so we should be able to access the parent code.

We can double check that last point using our previous HTML-JS code. We’ll use the Web API to escape the embedded HTML by getting its the frameElement. This should return a reference to the <iframe> element, which, remember, exists inside the uifigure HTML webpage. Instead of displaying the frames URL in the <div>, we will display the uifigures URL.

t.textContent = window.frameElement.ownerDocument.URL;

Run the uihtml in Matlab and check the text being displayed. If you find the webwindow from the uifigure, both URLs should match.

Mini-exercise: Try opening the HTML now in your browser? Do you see anything? Is there an error if you open the console? Can you explain (using the documentation of the frameElement) what caused the error?

Being able to get the uifigures document is very interesting, and feels a bit like cheating. Thanks to this, we are technically able to alter the any standard component (a uibutton, for example), by modifying their look or even their callbacks. But I wouldn’t recommend messing with Mathworks base code. I think that it is against the license agreement, and chances are you add a bug by mistake. That being said, I have found myself using this “trick” to make some components (uigridlayouts) transparent, which is something that none of the standard components support (I don’t know why and I hate not being able to do it in my apps).

Just one more limitation

When we talked about uifigures, we mentioned CORS and how by default we generally can embed files from different origins in our websites (embedding was ok, but fetching was blocked). That means that, in theory, we should be able to embed a <script> or an <iframe> inside our uihtml <iframe> from anywhere over the internet. For example, we should be able to load a library in our myHTMLdemo.html file and then use it, as in our next example.

Don’t worry if you don’t know the lodash library, it just contains some basic functions that are faster than the default JS ones. I added it just as an example and what it does is not relevant to our cause.

<!DOCTYPE html>
<html>
<head>
    <script type="text/javascript" 
       src="https://cdn.jsdelivr.net/npm/lodash@4.17.21/lodash.min.js">
    </script>
</head>

<body>
    <div id="textDiv"></div>  
    <script type="text/javascript">
        let t = document.getElementById("textDiv");
        t.textContent = _.add(10, 4); // Use lodash(_) to add 2 numbers
    </script>
</body>
</html>

When you try to run again the Matlab commands and open the uifigure, you’ll notice that nothing appears. And if you open the HTML file directly in the browser, it works! Matlab is blocking us on purpose.

We talked before about CORS and how does it specifies which origins can access a file from a server. CORS has a brother called CSP (Content Security Policy) that allows a server to specify from where can resources be loaded. It’s like a double security layer: for a server, CORS specifies who can get its resources, and CSP dictates from who can it get resources. Matlab configures our localhost CSP directives such that we can only load resources from our own origin. To inspect the CSP directives, we can attach a securitypolicyviolation event listener to our <iframe> document before we embed lodash.

<!DOCTYPE html>
<html>
<head></head>

<body>
  <div id="textDiv"></div>
  <script type="text/javascript">
      let t = document.getElementById("textDiv");
      document.addEventListener("securitypolicyviolation", (e) => {
        t.textContent = e.originalPolicy;
      });
  </script>
  <script type = "text/javascript" 
   src = "https://cdn.jsdelivr.net/npm/lodash@4.17.21/lodash.min.js">
  </script>
</body>
</html>

Now, running the commands from Matlab should result in a lot of text that is formatted like default-src ...; script-src ...; img-src ...; frame-src .... As you may have guessed, each of these lines represents what sources are allowed for every specific element.

script-src: 'self' 'unsafe-inline' 'unsafe-eval' *.mathworks.com:* *.thingspeak.com:* thingspeak.com:* *.matlab.com:* https://localhost:* https://127.0.0.1:*;

If we look at what the allowed sources for scripts can be, we see that we are actually not only limited to our own origin (the self keyword), but we could load a file from a .matlab, .mathworks and .thingspeak server, or even from another port in our localhost. The unsafe-inline and unsafe-eval are there to bypass default security limitations, allowing us to write JS inside the HTML document (just as how we have been doing so far, without needing to have the JS code in a separate file) and to use the JS eval function.

If we were allowed to access any source, CSP directive would be just a wildcard:

script-src: *;

Practically speaking, the conclusion from this limitation is that our uihtml components will only be able to embed resources that are already being hosted by the web server. Resources need to be located locally, in a directory at the same level as our HTML file or below it. So instead of embedding a script or an image from the Internet, we will need to download it and save it in the same folder as the HTMLSource from the uihtml.

Conclusion

That was a lot of information! I hope that this wasn’t too boring and that you could follow it easily. Will it be useful? Maybe. From our user perspective, most of the things I mention in the post will not have an impact when we start designing our custom components. But there is some stuff that is quite relevant, such as the SOP and us being able to access the uifigure document directly from the uihtml one. And even if it turns now not to be useful, it helps us understand the limitations that we will face when we start our component design.

I leave you one last exercise. From what we have learned, can you answer these?

• Go to the uihtml documentation page. You will find a section called limitations. Can you, after reading this post, explain what causes them?
• Why can’t I embed in my HTML an MP4 video encoded through the HEVC/H.265 standard? Matlab won’t open it, but if I open the HTML with Safari it works just fine!
• Does it make sense to debug our HTML files using console.log() and opening them in our browser? When won’t that work?

Disclaimer

I am not a front-end or a back-end application engineer, and I didn’t know about any of these topics before I started messing around with Matlab. If you notice that I said something that’s not true, please correct me in the comments and I will happily update the post.