← Back to home

Status update - May 2023

Here's what I've been doing in May.

Smaller prototypes, SQLite WASM, and more.

Single-file tools

I’ve been trying to make smaller things lately. I have a habit of getting ahold of a fun idea, then making it not fun by biting off more than I can chew. Sometimes that’s good. It’s fun to find that something is easier than I thought. Or maybe as hard as I thought but still possible. Other times it’s just a bummer to leave something unfinished because it was just too big.

Tools, scripts, and little code snippets have been them “smaller things” lately. I’ve found that a good template for keeping these things small is what I call a “single-file tool” – a single .html file with a <script> tag, some styling, and that’s about it. It looks like this:

Here’s the proof of concept HTML template I’ve been using.

<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="utf-8">
  <meta content="width=device-width, initial-scale=1, shrink-to-fit=no"
        name="viewport">
  <title>single-file tool demo, v1</title>
  <script>
    function ready(fn) {
      document.readyState !== "loading" ?
        fn() :
        document.addEventListener("DOMContentLoaded", fn);
    }
  </script>
</head>

<body>
  <div id="root">
    <h1>single-file tool</h1>
    <p>click the button</p>
    <button id="button">
      click me
    </button>
    <pre id="render-data"></pre>
    <p>
      <button id="save-as">
        Save...
      </button>
    </p>
  </div>
  <script>
    function getDocHtml() {
      const raw = document.documentElement.outerHTML;
      return "<!DOCTYPE html>\n" + raw;
    }

    async function saveAs() {
      const blob = new Blob(
        [getDocHtml()],
        {
          type: "application/xml;charset=utf-8"
        });
      if (window.showSaveFilePicker) {
        const newHandle = await window.showSaveFilePicker({
          suggestedName: `version-${window.DATA.version}.html`
        });
        const writableStream = await newHandle.createWritable();
        await writableStream.write(blob);
        await writableStream.close();
      } else {
        const a = document.createElementNS(
          'http://www.w3.org/1999/xhtml',
          'a'
        );
        a.download = `version-${window.DATA.version}.html`;
        a.rel = 'noopener';
        a.href = URL.createObjectURL(blob);
        setTimeout(() => {
          URL.revokeObjectURL(a.href);
        }, 60_000);
        setTimeout(() => {
          a.dispatchEvent(new MouseEvent('click'));
        }, 0);
      }
    }

    ready(() => {
      document.getElementById("button")
        .addEventListener("click", () => {
          window.DATA = {
            "version": window.DATA.version + 1,
            "value": Math.random()
          };
          const d = document.getElementById("data-block");
          const json = JSON.stringify(window.DATA);
          d.innerHTML = `window.DATA = ${json}`;
          render();
        });

      document.getElementById("save-as")
        .addEventListener("click", async () => saveAs());

      render();
    });

    function render() {
      const preData = document.getElementById("render-data");
      preData.innerHTML = JSON.stringify(window.DATA, null, "  ");
      document.title = `single-file tool demo, v${window.DATA.version}`;
    }
  </script>
  <script id="data-block">
    window.DATA = {
      "version": 1,
      "value": 100
    };
  </script>
</body>

</html>

Here’s a demo of the above file.

It’s nice to be able to use little tools like this by just saving them back to my file system after opening them in chrome through file://. A little personal finance calculator, a habit tracker, a scratch pad. Cool to have copies of them at a single point in time.

Script asset inlining

When I was building larger versions of the single-file tools, I was surprised to find that it’s hard to inline scripts into an html file. If you’re building/rendering with React, it comes down to using dangerouslySetInnerHTML. And with static HTML, you can replace a template string. Both of these will have issues with scripts containing xml tags, however. You can escape those characters, but it’s not the easiest thing to do. It turns out there’s no clean way to do this without invoking an external library, and then you’re just punting the problem because they’re doing much the same thing. In the end I landed on just URI-encoding the script, and unpacking it and using eval on the frontend. This is admittedly slow and clunky, but not as much as you’d think.

// Compile js, css, and index.html into single file.
const fs = require("fs");
const path = require("path");

const html = fs.readFileSync(path.join(__dirname, "src", "index.html"))
  .toString();
const rawJs = encodeURIComponent(
  fs.readFileSync(path.join(__dirname, "build", "index.js")).toString(),
);
const js = `
  const boot = performance.now();
  console.log("bootstrapping ${rawJs.length / 1000000} MB of js");
  const rawCode = "${rawJs}";
  console.log("found " + (rawCode.length/1000000) + " MB of raw code");
  const rawCodeStr = decodeURIComponent(rawCode);
  const dTime = Math.floor(performance.now() - boot);
  console.log("decoding raw code took " + dTime + "ms");
  console.log("raw code length: " + rawCodeStr.length);
  console.log("running eval() on js");
  const start = performance.now();
  eval(rawCodeStr);
  const end = performance.now();
  const eTime = Math.floor(end - start);
  const bTime = Math.floor(end - boot);
  console.log("eval() took " + eTime +"ms");
  console.log("bootstrapping took " + bTime +"ms");
`;
const css = fs.readFileSync(path.join(__dirname, "build", "styles.css"))
  .toString();
const distHtml = html
  .replace("/*SCRIPT_HERE_89709f7c7890c91f428c*/", js)
  .replace("/*STYLE_HERE_89709f7c7890c91f428c*/", css);
fs.rmSync(path.join(__dirname, "dist", "index.html"), { force: true });
fs.mkdirSync(path.join(__dirname, "dist"), { recursive: true });
fs.writeFileSync(path.join(__dirname, "dist", "index.html"), distHtml);

In terms of performance, it’s actually not that bad. Here’s an intentionally large JS build evaluating in 44ms.

bootstrapping 5.464157 MB of js
found 5.464157 MB of raw code
decoding raw code took 26ms
raw code length: 3475559
running eval() on js
eval() took 44ms
bootstrapping took 71ms

SQLite WASM

Using SQLite inside JS is the dream and it’s finally supported! Hurray! Working on a small experiment to write a VFS that I can patch into the WASM module to read and write page data directly to the DOM. It’s easy once you figure out how the VFS system works, but that takes a while.

It’s only about a dozen IO and calls for the sqlite_io_methods interface, and you’re there.

interface SqliteVfs {
  mkdir: (dirname: string) => number;
  xAccess: (filename: Pointer) => number;
  xClose: (fid: number) => number;
  xDelete: (
    filename: Pointer,
    syncDir?: number,
    recursive?: boolean,
  ) => number;
  xDeleteNoWait: (
    filename: Pointer,
    syncDir?: number,
    recursive?: boolean,
  ) => number;
  xFileSize: (fid: Pointer) => number;
  xLock: (fid: number, lockType: LockType) => number;
  xOpen: (
    fid: Pointer,
    filename: number,
    flags: number,
  ) => number;
  xRead: (
    fid: Pointer,
    n: number,
    offset64: number,
  ) => number;
  xSync: (
    fid: Pointer,
    flags: number,
  ) => number;
  xTruncate: (fid: Pointer, size: number) => number;
  xUnlock: (
    fid: Pointer,
    lockType: LockType,
  ) => number;
  xWrite: (
    fid: Pointer,
    n: number,
    offset64: number,
  ) => number;
}

interface IoSyncMethods {
  xCheckReservedLock: (pFile: number, pOut: number) => number;
  xClose: (pFile: number) => number;
  xDeviceCharacteristics: (pFile: number) => number;
  xFileControl: (pFile: number, opId: number, pArg: number) => number;
  xFileSize: (pFile: number, pSz64: number) => number;
  xLock: (pFile: number, lockType: number) => number;
  xRead: (
    pFile: number,
    pDest: number,
    n: number,
    offset64: number,
  ) => number;
  xSync: (pFile: number, flags: number) => number;
  xTruncate: (pFile: number, sz64: number) => number;
  xUnlock: (pFile: number, lockType: number) => number;
  xWrite: (
    pFile: number,
    pSrc: number,
    n: number,
    offset64: number,
  ) => number;
}

interface VfsSyncMethods {
  xAccess: (
    pVfs: number,
    zName: number,
    flags: number,
    pOut: number,
  ) => number;
  xCurrentTime: (pVfs: number, pOut: number) => number;
  xCurrentTimeInt64: (pVfs: number, pOut: number) => number;
  xDelete: (pVfs: number, zName: number, doSyncDir: number) => number;
  xFullPathname: (
    pVfs: number,
    zName: number,
    nOut: number,
    pOut: number,
  ) => number;
  xGetLastError: (pVfs: number, nOut: number, pOut: number) => number;
  xOpen: (
    pVfs: number,
    zName: number,
    pFile: number,
    flags: number,
    pOutFlags: number,
  ) => number;
  xRandomness: (pVfs: number, nOut: number, pOut: number) => number;
  xSleep: (pVfs: number, ms: number) => number;
}
type Pointer = number | string;

If you didn’t care about journaling, or didn’t want to support all the flags, you could probably get away with even fewer functions.

status-update
2023-06-05