What is OpenTelemetry

The OpenTelemetry project is a collection of standards, libraries, and applications that come together to form a vendor-neutral open source observability ecosystem.

When looking at the overview diagram of the OpenTelemetry ecosystem, we see "OTel Auto. Inst." in the top left. This is what we will be focussing on in this article.

OpenTelemetry Zero-Code Instrumentation

As part of its suite of libraries, OpenTelemetry provides zero-code instrumentation for .NET, Go, Java, JavaScript, PHP, and Python.

The OpenTelemtry documentation the goals of this zero-code instrumentation as succinctly as possible:

Zero-code instrumentation adds the OpenTelemetry API and SDK capabilities to your application typically as an agent or agent-like installation. The specific mechanisms involved may differ by language, ranging from bytecode manipulation, monkey patching, or eBPF to inject calls to the OpenTelemetry API and SDK into your application.

Typically, zero-code instrumentation adds instrumentation for the libraries you’re using. This means that requests and responses, database calls, message queue calls, and so forth are what are instrumented. Your application’s code, however, is not typically instrumented. To instrument your code, you’ll need to use code-based instrumentation.

We will be installing the Java agent in a Docker container to monitor our application.

The Approach

A long time ago I stumbled upon the following problems:

1. Managing the Dockerfile can be hard.
2. The syntax for enabling the agent in the Dockerfile is easily broken.
3. Knowing when a new version of the agent is released can be considered difficult.
4. The agent is not easily cached across CI builds, this is annoying.

I solved all of these problems by using the Gradle distribution plugin and Jib, a tool built by Google to build optimized Docker and OCI images using a Gradle (or Maven) plugin.

All examples will use the Gradle Kotlin syntax. This has been the default since 2023.

First, we have to install the distribution and Jib plugins:

plugins {
    id("java")
    id("distribution") // distribution-base also works >= Gradle 8.13
    id("com.google.cloud.tools.jib") version "3.4.4"
}

Before jumping ahead, we will do something unexpected. We will create a Gradle configuration to house our agent:

val openTelemetryAgent: Configuration by configurations.creating

After this, we can use the configuration to define a dependency on the OpenTelemetry Java agent:

dependencies {
    // ...
    openTelemetryAgent("io.opentelemetry.javaagent:opentelemetry-javaagent:2.13.3")
    // ...
}

Luckily for us, the OpenTelemetry team publishes a Maven artifact for us.

The custom configuration makes sure that we do not include our agent in our final application JAR by accident. It also makes bots like Dependabot and Renovate aware of the dependency. The dependency is also exposed to the ben-manes/gradle-versions-plugin, if you use it.

Now, define a distribution that contains our agent. Gradle automatically manages build caching based on the from method.

distributions {
    create("openTelemetryAgent") {
        distributionBaseName = "otel-agent"
        contents {
            from(openTelemetryAgent)
            rename("opentelemetry-javaagent-.*.jar", "otel-javaagent.jar")
        }
    }
}

Notice that we strip the version number of the Maven artifact. The agent JAR is stored in the build/install/otel-agent/otel-javaagent.jar directory (when building or calling the distribution installation Gradle task).

We will now configure Jib (make sure not to forget to import the Jib Gradle plugin):

jib {
    container {
        ports = listOf("8080/tcp") // Configure the port(s) exposed by your app
        // === This is optional-language env vars are usually defined by your base image
        environment = mapOf(
            "SERVER_PORT" to "8080", // Works on Spring
            "LANG" to "en_US.UTF-8",
            "LANGUAGE" to "en_US:en",
            "LC_ALL" to "en_US.UTF-8"
        )
        // === Above here is optional
        jvmFlags = listOf("-javaagent:/agent/otel/otel-javaagent.jar")
    }
    extraDirectories {
        paths {
            path {
                setFrom(layout.buildDirectory.file("./install/otel-agent"))
                into = "/agent/otel"
            }
        }
    }
    from {
        image = "eclipse-temurin:21"
    }
    to {
        image = "ghcr.io/my_repository/my_image"
        tags = listOf("latest") // Or use an environment variable in your CI
    }
}

For all options, see the Jib Gradle plugin configuration reference.

As you can see, we copy the otel-javaagent.jar from the build directory into the /agent/otel directory of the image during built time. We then activate it using -javaagent:/agent/otel/otel-javaagent.jar.

However, we are missing one final step. It is crucial that we let Gradle know that we need the distribution to run our :jib tasks:

tasks {
    // ...
    this@tasks.jib {
        dependsOn(getByName("installOpenTelemetryAgentDist"))
    }
    // ...
}

Don't focus too much on the syntax. Without the this@tasks. the wrong thing is referenced.

We are finished! Run the :jib or :jibDockerBuild tasks to create a container image on your local machine and test it out!

Please refer to the OpenTelemetry documentation to determine what environment variables you can use.

Note that this also works with other Java agents, like the AppDynamics Java Agent or New Relic Java Agent.

Why not use the Spring Boot Starter?

The Spring Boot starter can only instrument libraries that Spring manages itself.

For example, when using Log4j2 with Spring, you need to configure the OpenTelemetry appender yourself. The agent would pick up its usage automatically.

However, if the agent bytecode scanning and manipulation are taking too long, feel free to drop the agent altogether and use the manual instrumentation.

It is also worth nothing that Java agents are not supported by GraalVM native images.

Of course, the approach mentioned in this article also works for non-Spring applications.

Closing Remarks

Installing the OpenTelemetry Java agent and keeping it up to date can be hard when applying a traditional approach, mainly because developers are used to editing Dockerfiles by hand

Using a Gradle and Jib makes it easy to integrate the Java agent with your application. It gives you the added benefit of automatically getting update notifications when using dependency management software like Dependabot or Renovate.

How have you experienced OpenTelemetry? Let me know!

Some time ago I discovered the open-source file sharing service LocalSend whilst browsing the GitHub explore page.

As my devices (phone, laptop, etc.) run on different operating systems, it's difficult to share files between them. Some approaches that I've used are uploading them to OneDrive, sending myself e-mails, and creating a Bitwarden Send link. This always felt quite inefficient, and only the last one is truly secure.

Installing LocalSend

Installing LocalSend is super easy. On the downloads page you can download clients for Windows, macOS, Linux, Android, and iOS devices. All you need to do now is open the LocalSend application.

Configuring Tailscale

Using Tailscale is probably the easiest way to connect devices together that I've ever experienced. Because of it's advanced NAT traversal techniques, it can directly connect your devices together without anyone being able to perform a MITM attack using the WireGuard protocol. It offers a generous free tier for up to 100 devices. For the purposes of this article, I will assume that you already have Tailscale installed and connected to all of your devices.

Why not use Taildrop?

Recently Tailscale announced their Taildrop file sharing functionality. Whilst this is a cool feature, it does not support sending files to and from nodes that have ACL tags, a common use case. The user interface of LocalSend is also better at the moment.

Unfortunately, LocalSend restricts the multicast mask to /24, this is not enforced by Tailscale by default. Your device addresses assigned by Tailscale could be 100.123.142.91 and 100.108.230.16, these devices will thus not be able to find each other.

It's possible to create an IP pool on the access control page that makes sure that future devices added to your tailnet stay in a certain range:

Note: This is a beta feature, the syntax might have changed when you are reading this.

{
  "acls": ["..."],
  "nodeAttrs": [
    {
      "target": ["autogroup:member"],
      "ipPool": ["100.x.y.0/24"]
    }
  ]
}

You can choose any 64 <= x <= 127 and any 0 <= y <= 255 (any address in the range 100.64.0.0/10).

If you have already added your devices, you can manually edit the IP of each device that you want LocalSend to automatically detect:

After reconnecting Tailscale, your devices will pop up in the LocalSend application:

Transferring Data

The LocalSend applications are really intuitive. They all follow the same design, as they are made using the amazing Flutter framework.

In the following example I've sent a text message to my phone. My laptop is connected to Wi-Fi and my phone is connected to 4G. Normally they would not be able to see each other, but because they are both connected to Tailscale, they can!

Fun fact: I actually sent the screenshot of my phone to my computer using the exact same setup. It only took a couple of seconds!

Configuring ACL Policies

Imagine being a household with multiple devices connected to Tailscale. You might want to limit who can send files to each other using LocalSend. Using access policies you can limit who can send and receive files:

{
  "tagOwners": {
    "tag:localsend": ["autogroup:admin"]
  },
  "acls": [
    {
      "action": "accept",
      "src": ["tag:localsend"],
      "dst": ["tag:localsend:53317"]
    }
  ],
  "nodeAttrs": [
    {
      "target": ["autogroup:member"],
      "ipPool": ["100.91.232.0/24"]
    }
  ]
}

Note: I am using IP addresses in the range 100.91.232.?/24 in my example.

Now, configure the ACL tags in the machine overview:

Restart Tailscale on all of your devices (and possibly LocalSend), and they will show up!

Alternatives

If you are fully bought into the Apple ecosystem, AirDrop is of course an obvious alternative (although this might expand to other devices in the near future). The difference with our solution is that we can also send files to family members or coworkers that might not be in our Bluetooth range.

Another popular open-source alternative is PairDrop. It works a bit differently in the sense that you have to deploy it to a server. If you ever want to share files with users outside your tailnet, this can be a great hosted alternative. The benefit of LocalSend is that it's a simple desktop application/app that works without needing any hosting infrastructure or pairing codes.

Let me know if you use any other alternatives 😀.

Conclusion

In this article I showed you how to configure Tailscale and LocalSend such that you can securely send files (such as photos or documents) to devices connected to different networks.

This is great if your phone is having trouble connecting to the Wi-Fi (and you have mobile reception). It is also useful when you want to quickly send a file to a family members' laptop down the hallway. In small business environments you can even set up LocalSend for your distributed team!

I've already used LocalSend a bunch, and it's a great product. I'm excited to hear about your use cases!

This guide will teach you how to set up GraalVM and GraalPy on your Linux or macOS machine (I am using WSL). After this, we create a Gradle project with the Native Build Tools plugin. When the setup is completed, we will install the Beancount package using pip and call a (Python) library function from our Java code.

Installing GraalVM and GraalPy

There exist multiple ways to install the GraalVM runtime environment. Because of the otherwise complicated setup process, I have chosen to use SDKMAN! and pyenv. Please follow the installation instructions on their respective pages.

First, we want to install GraalVM itself. We can do this using SDKMAN!. Please ensure that you install the distribution that conforms to your licence requirements. I will be referring to the community editions in this article.

sdk install java 21.0.1-graalce

After this is completed, we can move onto installing GraalPy. This Python 3.10 compliant runtime makes polyglot programming possible.

pyenv install graalpy-community-23.1.0

To access the GraalPy environment we can use pyenv shell graalpy-community-23.1.0.

You can verify the installations by running the following commands:

$ java --version
java 21.0.1 2023-10-17
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.1+12.1 (build 21.0.1+12-jvmci-23.1-b19)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.1+12.1 (build 21.0.1+12-jvmci-23.1-b19, mixed mode, sharing)
$ pyenv shell graalpy-community-23.1.0
$ graalpy --version
GraalPy 3.10.8 (GraalVM CE Native 23.1.0)

Creating a Virtual Environment

We can now create a virtual environment to store our dependencies:

graalpy -m venv venv

In this example our virtual environment is simply called venv.

To use pip and other commands in this virtual environment, we need to "activate" it. This can be done by sourcing the correct shell script for your platform, in our case: source ./venv/bin/activate.

Instantiating Your Gradle Project

You can instantiate your Gradle anyway you like, or reuse an existing project.

This can be done using gradle init.

The tutorial will assume a Java application project using the Kotlin DSL.

First of all, we want to make sure we have the application and Native Build Tools plugins installed:

plugins {
  application
  id("org.graalvm.buildtools.native") version "0.9.28"
}

We also need to define the following dependencies¹:

dependencies {
  implementation("org.graalvm.polyglot:polyglot:23.1.0")
  runtimeOnly("org.graalvm.polyglot:python-community:23.1.0")
  runtimeOnly("org.graalvm.polyglot:llvm-community:23.1.0")
}

We can define our main class using the application plugin:

application {
  mainClass.set("org.example.Main")
}

Installing Beancount

When I first tried to install Beancount using pip install beancount==2.3.6 I got the following error:

Compile failed: command 'cc' failed: No such file or directory
    ...
    creating tmp
    cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitsdjoqzb0.c -o tmp/xmlXPathInitsdjoqzb0.o
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    *********************************************************************************
    error: command 'cc' failed: No such file or directory
    [end of output]
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lxml
Failed to build lxml
ERROR: Could not build wheels for lxml, which is required to install pyproject.toml-based projects

In this case I was missing cc. On Ubuntu this is included with the build-essential package. For good measures, I made sure to install all packages that might be necessary to install lxml:

sudo apt-get install libxml2-dev libxslt-dev pyhton3-dev build-essential

Different system dependencies might be necessary, depending on your operating system and the package you are trying to install. Always carefully read the error messages produced by pip install, and check the package documentation to see if you are missing any prerequisites.

It is important to note that installing packages in your virtual environment might take longer than you're used to. This is because the environment also checks if any patches need to be made to the package or its (sub)dependencies to make it cooperate correctly with GraalPy.

The Testcase

To keep it simple, in Beancount, a ledger is the combination of a set of files that contain certain directives.

If this concept is something you want to investigate further, you might be interested in the Getting Started with Beancount article by the creator of Beancount. Another good starting point is plaintextaccounting.org.

One of these directives is the commodity directive. This directive defines a commodity (e.g., currency or stock) that can be referenced in the rest of your ledger.

In our example, we want to read a ledger, extract its directives, and find all commodity symbols (e.g, EUR).

2023-12-12 commodity EUR
2023-12-12 commodity GBP
2023-12-12 commodity USD
2023-12-12 commodity CHF

In this example we define the commodities EUR, GBP, USD, and CHF.

Our goal is to create a String[] with these symbols.

In Python these symbols can be found programmatically using the following script:

from beancount import loader
from beancount.core.data import Commodity

# In this tuple, `entries` is a list of directives
(entries, errors, option_map) = loader.load_file('test.beancount')

symbols = [entry.currency for entry in entries if isinstance(entry, Commodity)]

print(symbols) # Prints ['EUR', 'GBP', 'USD', 'CHF']

Folder Structure for Our Example

Our project structure should now look somewhat similar to the following:

.
├── build.gradle.kts
├── gradle
│   └── wrapper
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
├── settings.gradle.kts
├── src
│   ├── main
│   │   ├── java
│   │   │   └── org
│   │   │       └── example
│   │   │           └── Main.java
├── test.beancount
└── venv
    ├── bin
    │   ├── activate
    │   ├── bean-check
    │   ├── pip
    │   └── ...
    ├── include
    ├── lib
    │   └── python3.10
    │       └── site-packages
    │           ├── __pycache__
    │           ├── beancount
    │           │   ├── __init__.py
    │           │   └── ...
    │           └── ...
    └── ...

If this is the case, we can start work on implementing the final solution.

Implementing Our logic in Java

When everything is set up correctly, it is trivial to call this exact code in Java. We are going to use the GraalVM Polyglot API to run our Python code in the JVM.

We start by creating a Context object, building these objects can be done by using the Context.Builder builder class:

final Context.Builder graal = Context.newBuilder("python", "llvm")
  .option("python.Executable", "venv/bin/graalpython")
  .option("python.ForceImportSite", "true")
  .allowIO(IOAccess.ALL)
  .allowNativeAccess(true);
try (Context ctx = graal.build()) {
  // Your code
}

Within the try-catch block we can use the Context to interact with our guest languages (in our case these are Python and LLVM).

We need to allow native access because the Beancount package internally uses the struct (and thus pack) builtins. These builtins are implemented as a C module, which requires native access on top of IO access. Otherwise, we will get the following error: ImportError: cannot import name 'pack' from 'struct'.

To make calls to our guest language (Python), we can use the Context#eval(String, String) method:

// First, import the required packages
ctx.eval("python", """
    from beancount import loader
    from beancount.core.data import Commodity
    """);
// Then, load the ledger
ctx.eval("python", "(entries, errors, option_map) = loader.load_file('test.beancount')");
// Find all symbols
final Value pySymbols = ctx.eval("python", "[entry.currency for entry in entries if isinstance(entry, Commodity)]");
// Convert them to their Java representation
final String[] symbols = pySymbols.as(String[].class);
// Finally, print the array (in the Java world)
System.out.println(Arrays.toString(symbols));

Our full solution now looks like this:

package org.example;

import org.graalvm.polyglot.Context;
import org.graalvm.polyglot.Value;
import org.graalvm.polyglot.io.IOAccess;

import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;

public class Main {
  public static void main(String[] args) {
    final Context.Builder graal = Context.newBuilder("python", "llvm")
          .option("python.Executable", "venv/bin/graalpython")
          .option("python.ForceImportSite", "true")
          .allowIO(IOAccess.ALL)
          .allowNativeAccess(true);
    try (Context ctx = graal.build()) {
      ctx.eval("python", """
            from beancount import loader
            from beancount.core.data import Commodity
            """);
      ctx.eval("python", "(entries, errors, option_map) = loader.load_file('test.beancount')");

      final Value pySymbols = ctx.eval("python", "[entry.currency for entry in entries if isinstance(entry, Commodity)]");
      final String[] symbols = pySymbols.as(String[].class);

      System.out.println(Arrays.toString(symbols));
    }
  }
}

This prints [EUR, GBP, USD, CHF], exactly what we want.

Verify that you are running the application using the executable built with the nativeBuild task, or that it is run using the nativeRun task. The Native Image Tools plugin offers an assortment of tasks. If you use different tasks (e.g., run), you will get an error because you are running the application like a "normal" Java application. This does not offer the polyglot programming capabilities offered by GraalVM.

Build Time

On my relatively powerful machine, it takes almost 8 minutes to compile this example.

When needing to interface with Python fast and securely, there can be advantages to using GraalPy and GraalVM, but techniques should be applied to make sure development time is not impacted by these long compilation times.

Conclusion

In this post, we learned how to set up GraalVM and its Python interface (GraalPy) with the Gradle build tool.

We looked at how to install packages using the pip package manager, and identified some ways to resolve issues as they occur during package installation in the context of GraalVM.

The polyglot programming capabilities of GraalVM are very powerful when used correctly. It is, however, sometimes challenging to find the right way to set up projects that make use of certain features.

What can these polyglot capabilities and powerful interfacing techniques bring to your business or project?

The example project is available as a repository on GitHub. The repository also includes a Docker image that shows how we can leverage GraalVM, GraalPy, and pip in containerized environments.