8 minute read

Categories:
Tags:

GitHub Packages is a natural extension of a CI/CD pipeline created in GitHub Action. It currently offers repositories for Java (Maven), .Net (NuGet), Ruby (Gems), JavaScript (npm), and Docker images.
For a lot of users this can be a free private service if you can squeeze under the size limitation and are okay using OAuth keys managed in GitHub.

Scala artifacts are usually stored in a Maven compatible repository, so while GitHub Packages doesn’t advertise Scala support explicitly it can still be a great fit for your project. Maven publishing is supported by SBT in its sbt publish task, so let’s give it a go.

HTTP 422 Errors

There are a few nuances with configuring GitHub Packages in SBT, they can be done manually in your build.sbt or more easily using a purpose built SBT plugin like sbt-github-packages. It looks like there are a lot of happy users, but sadly I couldn’t get it to work.

[error] java.io.IOException: Error writing to server
[error] 	at java.base/sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:718)
[error] 	at java.base/sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:730)
[error] 	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1613)
[error] 	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
[error] 	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
[error] 	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
[error] 	at org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:284)
[error] 	at org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)

For an actively maintained GitHub project, I was sad to see an identical 4-month-old open issue Without change the SBT file, I get a “java.io.IOException: Error writing to server” exception. Even manually configuring SBT without the plugin couldn’t resolve the issues, there might be an incompatibility with how my project is named or versioned that I just couldn’t resolve.

Über Jar

aka: fat jar, uber jar, or executable jar

Steps to publish regular library jars is included as an appendix

To complicate the issue further, I didn’t just want to privately publish my standard artifacts, I want to publish an über jar. Similar to how Docker creates an easy deploy with just 1 file, an über jar is similar. It is also similar to how a Java War file is a deployable package. Normal Jar files are lean, they only contain your compiled code and publish their dependencies in a POM file. This is great for libraries, but in the case where the jar is meant to be a standalone executable all dependencies will need to be included. This is an über jar, it is like a regular jar but includes the .class or .jar of all of your code’s dependencies. It doesn’t make sense to publish an über jar like it was a library, since the extra code it includes will likely have conflicts or overlaps with other libraries used in a linking project.

It does however to maintain a set of compiled releases, and here we will be using GitHub Packages.

There is an SBT plugin called sbt/sbt-assembly that will create an über jar, and allow it to be published to Maven. It is a single-line add to project/plugins.sbt and takes zero configuration

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "1.2.0")

It should be as simple as running sbt assembly to create, and sbt publish to publish. But like before, GitHub Packages is still returning HTTP 422 on any attempt, and with this new non-standard artifact it just gets more complex.

SBT is compiling, and Maven is publishing

We have 2 tasks, compile and publish.

It looks like sbt-assembly compiles fine, but SBT is failing at publishing the artifacts.

An alternative approach would be to set up Maven to compile, and also use it to publish the Scala project like it was Java (which I would assume works fine since it is directly supported by GitHub).

A second alternative approach would be to keep SBT, but create an SBT task to publish using an external Maven call. Looking over the Maven documentation there is a deploy:deploy-file command in Maven that can publish any file.

The command for our über jar will be something like:

mvn deploy:deploy-file \
  -Durl=https://maven.pkg.github.com/user/repo \
  -DrepositoryId=github \
  -Dfile=target/scala-2.13/project_2.13-version.jar \
  -DgroupId=com.yourcompany \
  -DartifactId=yourproject \
  -Dversion=1.0.0

There is an implicit file that the repositoryId=github parameter refers to, it expects your repository credentials for the github repository id to be stored in your ~/.m2/settings.xml file. Pleasantly there is way to manually specify where this file is located using --settings=, which will be important for us because our repository credentials for our GitHub Action are stored in GitHub.

⚠️ Since we are creating an über jar, the artifact generated by sbt-assembly is named project-assembly instead of project

⚠️ Locally we would use a GitHub OAuth token for credentials, we could also use it in the GitHub Action but they should ideally use the token provided to the action.

Creating a custom SBT task

SBT plugins and customization in .sbt and /project/* files are Scala code. Nothing stops you from running insanely complicated code in an SBT task, but here we have a pretty simple two-step workflow. We want to create a new task to:

  • create an über jar using sbt-assembly
  • create a settings.xml file
  • publish the jar using a call to an external mvn

Since this is drop in task, put it either in build.sbt or its own file called publishAssembyToGitHubPackages.sbt. Defining a manually executed task is pretty simple in SBT:

lazy val publishAssembyToGitHubPackages = taskKey[Unit]("Publish Über Jar to GitHub Packages")

publishAssembyToGitHubPackages := {
  your scala code goes here
}

The body of the task can be any Scala. Normally SBT is complicated by different configurations existing in different scopes, here everything we need is in the Global scope. The inputs for this task, whether run locally or inside GitHub Actions will be populated from environmental variables. This is a secure way to handle OAuth tokens and prevent them from being committed to your code repo.

Create an Über Jar Using sbt-assembly

This is simple, call assembly in SBT. The task returns a java.io.File which is useful in the next steps.

Create a settings.xml File

This file can be created in /target to keep things tidy when running this task locally. The secure contents of this file will also be populated by Maven from ENV variables, so it would be safe to commit this file into source code and skip this step. To dynamically create this file from a String defined in our .sbt file we can use SBT’s IO.write.

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <activeProfiles>
        <activeProfile>github</activeProfile>
    </activeProfiles>
    <profiles>
        <profile>
            <id>github</id>
            <repositories>
                <repository>
                    <id>github</id>
                    <url>https://maven.pkg.github.com/${GITHUB_REPOSITORY}</url>
                    <snapshots>
                        <enabled>true</enabled>
                    </snapshots>
                </repository>
            </repositories>
        </profile>
    </profiles>
    <servers>
        <server>
            <id>github</id>
            <username>${GITHUB_REPOSITORY_OWNER}</username>
            <password>${GITHUB_TOKEN}</password>
        </server>
    </servers>
</settings>

The file references 3 ENV variables, which according to GitHub Action documentation are populated during execution:

  • GitHub owner GITHUB_REPOSITORY_OWNER
  • GitHub repository GITHUB_REPOSITORY
  • GitHub OAuth token GITHUB_TOKEN

Once settings.xml exists, the first bit of Scala code should confirm these 3 ENV variables have been set:

val githubRepository = sys.env.get("GITHUB_REPOSITORY").getOrElse {
  throw new Exception("You must set environmental variable GITHUB_REPOSITORY, eg: owner/repository")
}
if (!sys.env.keySet.contains("GITHUB_REPOSITORY_OWNER")) {
  throw new Exception("You must set environmental variable GITHUB_REPOSITORY_OWNER, eg: your username")
}
if (!sys.env.keySet.contains("GITHUB_TOKEN")) {
  throw new Exception("You must set environmental variable GITHUB_TOKEN")
}

Publish the Über Jar Using a Call to mvn

This last step will assume mvn is installed can can be called from the command line. This is true for GitHub Actions, when running this task locally ensure that mvn is available in your $PATH.

The name, organization and version keys defined every build.sbt are suitable for most scenarios to populate the mvn parameters. The sbt-assembly task returns the file we want to publish. In SBT, it is necessary to call .value on setting keys, since they need to be resolved to their current value. Calling assembly.value on the assembly task automatically runs it ensuring the file is in our /target folder.

val exe =
  s"""mvn deploy:deploy-file
  -Durl=https://maven.pkg.github.com/$githubRepository
  -DrepositoryId=github -Dfile=${assembly.value}
  -DgroupId=${organization.value}
  -DartifactId=${name.value}-assembly
  -Dversion=${version.value}
  --settings=target/settings.xml
""".stripLineEnd

println(s"Executing shell command $exe")

import scala.sys.process._

if (exe.! != 0) throw new Exception("publishAssembyToGitHubPackages failed")

GitHub Workflow

We have populated our publishAssembyToGitHubPackages code added it to a publishAssembyToGitHubPackages.sbt file in the base of our project. The next step is to create a new Action in GitHub Actions. It can be created through the website, or manually by creating a new file /.github/workflows/publish-uber-assembly-to-github.yml.

⚠️ You may need to adjust your Personal Access Token permissions to check in this file using git since it requires workflow permissions._

The action YAML starts with the basics, a name:

name: Publish Über Assembly to GitHub Packages

Make it a manual execution for now, this could optionally be tied into git tag events in the future:

on:
  workflow_dispatch:

The GITHUB_TOKEN isn’t populated into ENV by default, so do this now. It will also need write privileges to packages to push to GitHub Packages.

env:
  GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
permissions:
  contents: read
  packages: write

The actual job is a simple setup of checkout, java, and then calling our publishAssembyToGitHubPackages task in SBT.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up JDK 11
        uses: actions/setup-java@v3
        with:
          java-version: '11'
          distribution: 'temurin'
      - name: Publish Über Jar to GitHub Packages
        run: sbt publishAssembyToGitHubPackages

And now we have working GitHub Package publishing of an über jar by adding only 2 files to our project.

Downloading using wget

A separate article Downloading from GitHub Packages Using HTTP and Maven has instructions on how to browse and download artifacts from GitHub Packages using HTTP and Maven.

Publishing non-Über Jars

This article is a little easier since there is only 1 über jar to publish. A typical library would also want to publish a dependency pom, javadoc jar and sources jar, plus those for any subprojects.

The necessary changes for a monolithic project are very small, we will use other SBT packaging tasks instead of the assembly task in the plugin. There is are mvn deploy:deploy-file parameters to specify the pom file, and optionally source and JavaDoc jars.

val exe =
  s"""mvn deploy:deploy-file
  -Durl=https://maven.pkg.github.com/$githubRepository
  -DrepositoryId=github
  -Dfile=${(Compile / packageBin).value}
  -DpomFile=${(Compile / makePom).value}
  -Dsources=${(Compile / packageSrc).value}
  -Djavadoc=${(Compile / packageDoc).value}
  --settings=target/settings.xml
""".stripLineEnd

For monolithic libraries include publishToGitHubPackages.sbt instead of publishAssemblyToGitHubPackages.sbt and call publishToGitHubPackages instead of publishAssemblyToGitHubPackages in your GitHub Action.

Full Sources:

Categories:
Tags:
Updated:

You May Also Enjoy

Job Queue Execution Management using ZIO Scopes

8 minute read

, ,

Job Queues are critical parts of Enterprise workloads. Complex queues use distributed nodes, state machines, and complex scheduling to trigger and track running jobs. But when simplicity allows the best approach is to create small idempotent jobs. The smaller the unit of work the easier progress can be tracked, jobs can be restarted or rerun with minimal waste, composability and reuse are increased, and logic is easier to reason about. These are the same arguments for Functional Programming and their Effect Systems, such as ZIO. Effect systems are congruent to the enterprise job queue, with ZIO fibers performing work and ZIO Resource Management forming the scheduling and supervision backbone. An efficient job queue can be written using ZIO constructs using surprisingly minimal amount of code. Table of Contents ZIO Resources and Scope Queue Features Class Interface ZIO Concurrency uses Fibers, Not Threads ...

Realtime Client Database using gRPC Bi-Directional Streams and ZIO Hub

1 minute read

, ,

Realtime pushed-based databases such as Google Firebase are a convenient way to ensure clients have the most recent data locally. Data updates are automatically streamed to clients immediately as they happen, or in the case of a client disconnect, immediately after reconnecting. gRPC server streaming and ZIO Hub allow this functionality to be easily replicated and customized beyond what expensive paid-for services such as Firebase can do.

Http4s Streams and Multipart Form-Data File Uploads

10 minute read

, ,

Streaming is the primary mechanism to reduce memory requirements for processing large datasets. The approach is to view only a small window of data at a time, allowing data to stream through in manageable amounts matching the data window size to the amount of RAM available. A practical example is a file-upload, where multi-GBs file streams can be handled by MBs of server RAM. However, enforcing streaming in software code is prone to errors, and misuse or incompatible method implementations will lead to breaking stream semantics, and ultimately to OOM exceptions. This article focuses on streams within the context of file uploads, using the Http4s library for examples.

Compiling Scala Native in a GitHub Action; Alternatives to GraalVM

11 minute read

, ,

Scala Native is a compiler and JDK written in Scala with the goal of removing Scala’s dependency on the JVM. This isn’t meant to achieve a higher performance such as with JDKs, and it is targeting a specialized use-case not considered to be today’s typical Scala development. Its competitors are Rust and Go, not GraalVM, Java or Kotlin. This article goes through common steps and challenges encountered when compiling Scala Native for linux with a GitHub Action.