Exploring Nix Flakes: Usable Go Plugins

Part 2: APIs and Dependencies

Previous Next

In the previous part we created a plugin with an init function so that it runs some code when compiled into the mainapp. Of course, that does give the plugin no practical possibility to interact with the mainapp. To do this, we’ll need a plugin API that defines how the main application and the plugins communicate.

To be able to define an API, we need to give our application some functionality. For this article, the functionality will be that it creates an image and serves it via HTTP; plugins can add to the image before it is served. This functionality is chosen for two reasons:

By serving via HTTP, the application can easily be delivered as an OCI image, which we will do in part 3.
We create an image via cairo, so that we have a dependency to a C library. This will enable us in part 3 to discuss handling C dependencies during cross-compilation.

The API we need cannot be part of our mainapp’s Go module since that would create a circular dependency (remember that our mainapp imports the plugin’s module). Thus, we will create a separate directory api inside our root. Inside that directory, create a Go module and setup the go-cairo dependency:

nix run nixpkgs#go mod init example.com/api
nix run nixpkgs#go -- get -d github.com/ungerik/go-cairo

We’ll give the API a Plugin type that defines the entry point of our plugin. Create a file api/plugin.go with the following content:

package api

import "github.com/ungerik/go-cairo"

// Plugin is the interface of mainapp plugins.
type Plugin interface {
	Paint(surface *cairo.Surface) error
}

In an actual application, you’d probably publish the API and thus could reference it directly in go.mod with its module path. But to keep it local – and also because we don’t have control over example.com – we’ll make this a Nix Flake just like everything else (remember how I said in part 1 that we could manage standard dependencies with Nix Flakes? That’s what we’ll be doing with the API).

Create a file api/flake.nix:

{
  inputs = {
    nix-filter.url = github:numtide/nix-filter;
  };
  outputs = {self, nix-filter}: {
    src = nix-filter.lib.filter {
      root = ./.;
      exclude = [ ./flake.nix ./flake.lock ];
    };
  };
}

This gives us a flake whose src output is simply the API module’s sources (this is a non-standard output but that’s fine for our local setup).

Now we need the main application that implements our functionality. To set it apart from our earlier iteration, create a directory image-server in the root directory. In it, create another module:

nix run nixpkgs#go mod init example.com/image-server
nix run nixpkgs#go -- get -d github.com/ungerik/go-cairo

Now let’s write our application. Put this in image-server/main.go (this is a modified version of the go-cairo example, extended with an HTTP server and plugin interaction).

package main

import (
	"log"
	"net/http"
	
	"example.com/api"
	"github.com/ungerik/go-cairo"
)

var plugins []api.Plugin

func main() {
	log.Println("serving at http://localhost:8080")
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		surface := cairo.NewSurface(cairo.FORMAT_ARGB32, 240, 80)
		surface.SelectFontFace(
			"serif", cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_BOLD)
		surface.SetFontSize(32.0)
		surface.SetSourceRGB(0.0, 0.0, 1.0)
		surface.MoveTo(10.0, 50.0)
		surface.ShowText("Hello World")
		for _, p := range plugins {
			if err := p.Paint(surface); err != nil {
				panic(err)
			}
		}
		png, _ := surface.WriteToPNGStream()
		w.Write(png)
		surface.Finish()
	})
	
	log.Fatal(http.ListenAndServe(":8080", nil))
}

The plugins variable is to hold our plugins. To populate it, we’ll extend the code we generate into plugins.go. Create a new template image-server/plugins.go.nix to do this:

lib: plugins: with builtins; ''
package main

// Code generated by Nix. DO NOT EDIT.

import (
	"log"
	${concatStringsSep "\n\t"
	  (lib.imap1 (i: p: "p${toString i} \"${p.goPlugin.goModName}\"")
		           plugins)}
)

func init() {
	${concatStringsSep "\n\t"
	  (lib.imap1
		 (i: p: "plugins = append(plugins, p${toString i}.Plugin())")
		 plugins)}
	log.Println("plugins have been initialized.")
} 
''

Compared to our previous iteration, we now give the imported plugin packages actual names. The generated code will look like this:

import (
	"log"
	p1 "<plugin 1 path>"
	p2 "<plugin 2 path>"
)

func init() {
	plugins = append(plugins, p1.Plugin())
	plugins = append(plugins, p2.Plugin())
}

We generate package names p1, p2 etc to avoid any possibility for name collisions. The code assumes that the root package of any plugin provides a function Plugin() which returns an api.Plugin. This is akin to the entry point of a classical C dynamic-library-plugin.

Now we need the flake of our new application at image-server/flake.nix:

{
  description = "demo image server";
  inputs = {
    nixpkgs.url = github:NixOS/nixpkgs/nixos-21.11;
    utils.url = github:numtide/flake-utils;
    nix-filter.url = github:numtide/nix-filter;
    api.url = path:../api;
  };
  outputs = {self, nixpkgs, utils, nix-filter, api}:
  let
    buildApp = { system, vendorSha256, plugins ? [] }:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        requireFlake = modName: ''
          require ${modName} v0.0.0
          replace ${modName} => ./vendor-nix/${modName}
        '';
        vendorFlake = modName: src: ''
          mkdir -p $(dirname vendor-nix/${modName})
          cp -r ${src} vendor-nix/${modName}
        '';
        sources = pkgs.stdenvNoCC.mkDerivation {
          name = "image-server-with-plugins-source";
          src = nix-filter.lib.filter {
            root = ./.;
            exclude = [ ./flake.nix ./flake.lock ];
          };
          nativeBuildInputs = plugins;
          phases = [ "unpackPhase" "buildPhase" "installPhase" ];
          PLUGINS_GO = import ./plugins.go.nix nixpkgs.lib plugins;
          GO_MOD_APPEND = builtins.concatStringsSep "\n"
            ((builtins.map (p: requireFlake p.goPlugin.goModName)
             plugins) ++ [(requireFlake "example.com/api")]);
          buildPhase = ''
            mkdir vendor-nix
            ${builtins.concatStringsSep "\n"
              ((builtins.map (p: vendorFlake p.goPlugin.goModName
                              "${p}/src") plugins)
              ++ [(vendorFlake "example.com/api" api.src)])}
            printenv PLUGINS_GO >plugins.go
            echo "" >>go.mod # newline
            printenv GO_MOD_APPEND >>go.mod
          '';
          installPhase = ''
            mkdir -p $out/src
            cp -r -t $out/src *
          '';
        };
      in pkgs.buildGoModule rec {
        name = "image-server";
        src = builtins.trace "sources at ${sources}" sources;
        modRoot = "src";
        subPackages = [ "." ];
        inherit vendorSha256;
        nativeBuildInputs = [ pkgs.pkg-config ];
        buildInputs = [ pkgs.cairo ];
        preBuild = ''
          export PATH=$PATH:${pkgs.lib.makeBinPath buildInputs}
        '';
      };
  in (utils.lib.eachDefaultSystem (system: rec {
    packages.app = buildApp {
      inherit system;
      vendorSha256 = "yII94225qx8EAMizoPA9BSRP9lz0JL/UoPDNYROcvNw=";
    };
    defaultPackage = packages.app;
  })) // {
    lib = {
      inherit buildApp;
      pluginMetadata = goModFile: {
        goModName = with builtins; head
          (match "module ([^[:space:]]+).*" (readFile goModFile));
      };
    };
  };
}

Compared to our previous iteration, we integrated the API code and give nixpkgs.lib to plugins.go.nix. We also added pkg-config, made it available in the PATH, and added cairo as dependency. go-cairo is configured to use pkg-config to discover how to link to cairo, which is why we need those dependencies.

As always, in image-server, check in everything and run:

git add . ../api
nix flake update
git add flake.lock
git commit -a -m "image server"
nix run

After we see our log line, visit http://localhost:8080 in your browser to query the image we generate. Stop the server with ^C. This sums up our updated main application.

Count Plugin

It’s the plugin that counts!

Our current implementation always creates the same image. We will now write a plugin that adds to the image the number of times the image has been queried. Create a directory count-plugin and do the usual initialization in it:

nix run nixpkgs#go mod init example.com/count-plugin
nix run nixpkgs#go -- get -d github.com/ungerik/go-cairo

Write the plugin’s implementation into count-plugin/plugin.go:

package count

import (
	"example.com/api"
	"github.com/ungerik/go-cairo"
	"log"
	"strconv"
)

type CountPlugin struct {
	count int
}

func (cp *CountPlugin) Paint(surface *cairo.Surface) error {
	cp.count += 1
	surface.SetFontSize(24.0)
	surface.MoveTo(200.0, 20.0)
	surface.ShowText(strconv.Itoa(cp.count))
	return nil
}

func Plugin() api.Plugin {
	log.Println("initializing count-plugin")
	return &CountPlugin{count: 0}
}

We provide a func Plugin() that is the plugin’s entry point. We define a type CountPlugin that implements api.Plugin. In Paint, we add a running number to the image. That’s it.

Now we need count-plugin/flake.nix, which is very similar to our previous plugin’s Flake, apart from the inclusion of the API:

{
  description = "count plugin for image-server";
  inputs = {
    nixpkgs.url = github:NixOS/nixpkgs/nixos-21.11;
    utils.url = github:numtide/flake-utils;
    nix-filter.url = github:numtide/nix-filter;
    image-server.url = path:../image-server;
    image-server.inputs = {
      nixpkgs.follows = "nixpkgs";
      utils.follows = "utils";
      nix-filter.follows = "nix-filter";
    };
  };
  outputs = {self, nixpkgs, utils, nix-filter, image-server}:
  utils.lib.eachDefaultSystem (system:
    let
      pkgs = nixpkgs.legacyPackages.${system};
    in rec {
      packages = rec {
        plugin = pkgs.stdenvNoCC.mkDerivation {
          pname = "image-server-count-plugin";
          version = "0.1.0";
          src = nix-filter.lib.filter {
            root = ./.;
            exclude = [ ./flake.nix ./flake.lock ];
          };
          passthru.goPlugin = image-server.lib.pluginMetadata ./go.mod;
          phases = [ "unpackPhase" "buildPhase" "installPhase" ];
          buildPhase = ''
            echo "\nrequire example.com/api v0.0.0" >>go.mod
          '';
          installPhase = ''
            mkdir -p $out/src
            cp -r -t $out/src *
          '';
        };
        app = image-server.lib.buildApp {
          inherit system;
          vendorSha256 = "US38BDmwhrrMxvZVzEq1ch65DGDS6Mq/IO4NvgyHsQU=";
          plugins = [ plugin ];
        };
      };
      defaultPackage = packages.app;
    }
  );
}

For this go.mod we only need the require directive – the replace in the main application’s go.mod will be honored. Let me stress again that we only refer to the API via flake to keep everything local and in actual code you most probably want to have a publicly available standard Go module as API.

Let’s finalize the plugin and run it:

git add .
nix flake update
git add flake.lock
git commit -a -m "count plugin"
nix run

Now when you access http://localhost:8080/ again, you will have a running number as part of the image, which increases when you reload. Browsers do background prefetching and caching, so you might not see every number.

Multiple Plugins

At this point, we have seen that we can create plugin-based applications and write usable plugins for them. What we haven’t talked about is what to do when we want to have multiple plugins active – currently, a plugin Flake simply defines an application where exactly this plugin is active. We did this mostly for convenience.

A tailored setup with a defined set of plugins would be written in an own Flake or as part of a system configuration, like this:

let
  refPlugin = url: {
    inherit url;
    inputs = {
      nixpkgs.follows = "nixpkgs";
      utils.follows = "utils";
      image-server.follows = "image-server";
    };
  };
in {
  description = "Tailored image-server";
  inputs = {
    nixpkgs.url = github:NixOS/nixpkgs/nixos-21.11;
    utils.url = github:numtide/flake-utils;
    image-server = {
      url = path:../image-server;
      inputs.nixpkgs.follows = "nixpkgs";
      inputs.utils.follows = "utils";
    };
    first = refPlugin path:../first-plugin;
    second = refPlugin path:../second-plugin;
  };
  outputs = {self, nixpkgs, utils, image-server, first, second}:
    utils.lib.eachDefaultSystem (system: {
      defaultPackage = image-server.lib.buildApp {
        inherit system;
        vendorSha256 = nixpkgs.lib.fakeSha256;
        plugins = [ first.packages.${system}.plugin
                    second.packages.${system}.plugin ];
      };
    });
}

Is it necessary to have platform-dependent packages for our plugins if they merely contain sources? For what we did here it isn’t, but it might be the case for other applications. The alternative would be to put them into some non-standard Flake output.

We now have explored how to make our main application and our plugins communicate via an API. We have also seen how to incorporate C dependencies with Nix. What remains now is to be able to target systems that don’t support Nix, most importantly Windows. This is what we’ll be doing in the next and final part.

Previous Next