gRPC versus JavaScript libraries

Cloud providers have libraries with support for uploading and downloading from cloud storage. These libraries approach transfers as a large number of HTTP requests, breaking up large files into smaller transfer requests. As requests move from pending, in progress, to complete, the current status of the transfer can be calculated.

The gRPC approach is similar, however instead of using separate HTTP requests each file chunk is sequentially sent over an HTTP/2 stream. The performance of both approaches is comparable, as multiple requests are multiplexed over a single HTTP connection closely mirroring how HTTP/2 streaming operates over a single connection. The only real difference is encountered when moving from sequential to concurrent operation, ie: sending multiple chunks at a time. Request multiplexing directly support this, however multiple stream requests need to be used to implement this using gRPC as each stream is inherently sequential and ordered.

pubspec.yaml Dependencies

This sample client requires 2 Flutter libraries:

• grpc core implementation for networking.
• image_picker implements an image picker widget to easily select upload files.

Flutter Workflow

Adding the complexity of streaming file transfers only benefits larger file transfers. Small transfers that can fit into the 4MB maximum message size for GRPC would be well served with the more simple approach.

In our demo, we are using a 35MB image originated by the James Webb Space Telescope, MACS J0417.5-1154 Wide Field , 4623 X 4623, PNG (35.14 MB).

Flutter Code

This implementation will use the state management built into Flutter:

• ChangeNotifier and ListenableBuilder
• ValueNotifier and ValueListenableBuilder

These work in the same way, with ValueNotifier being an implementation inheriting from ChangeNotifier suitable for simplified immutable state. The ChangeNotifier and ValueNotifier are data classes storing an internal state, which are rendered by ListenableBuilder and ValueListenableBuilder widgets, with the ability to trigger widget rendering on state changes.

There are 5 states to watch in this workflow:

• ValueNotifier<XFile?> representing a selected upload image (XFile) on the client, chosen by ImagePicker.
• ValueNotifier<FileSendChangeNotifier?> represented by either null when there isn’t an upload, or non-null for an upload.
• FileSendChangeNotifier represents an upload process, either actively uploading or completed.
• ValueNotifier<FileReceiveChangeNotifier?> represented by either null when there isn’t a download, or non-null for a download.
• FileReceiveChangeNotifier represents a download process, either actively downloading or completed.

Code: file_transfer_change_notifier.dart

Both uploads and downloads have been modeled using an abstract class FileTransferChangeNotifier. This class will track the transfer process, with update being called as each FileChunk of the stream is processed.

A close method should be called after a successful transfer has completed.

The implementations for upload and download require different implementations because of how stream processing has been implemented: uploads process stream elements before they are uploaded (ie: transfer will happen), and downloads process stream elements after they are downloaded (ie: transfer has happened).

abstract class FileTransferChangeNotifier with ChangeNotifier {
  /// State
  FileTransferProgress _progress;
  DateTime _lastUpdate;

  /// Used to render UI Widgets
  FileTransferProgress get progress => _progress;
  
  /// `fileChunk` is about to be sent on send, or has been received on receive.
  void update(FileChunk fileChunk);

  /// Mark the transfer as successful and complete.
  /// `filename` argument should be the server reference on send, 
  ///  or the local file path if received.
  void close(String filename);
}

FileSendChangeNotifier

This class implements the upload process, each update call will mark the current FileChunk as in progress and all previous as completed, with close marking the current FileChunk as completed.

FileReceiveChangeNotifier

This class implements the download process, each update call will mark the current FileChunk as completed, with close unnecessary but for performing the client-server filename mapping should it exist.

Code: file_transfer_progress.dart

This class is the immutable data model used by FileSendChangeNotifier and FileReceiveChangeNotifier.

class FileTransferProgress {

  //Populated when process starts.
  // null only for an upload that hasn't processed its first FileChunk
  final DateTime? startTimestamp;
  
  //Populated by calling `close`
  final DateTime? endTimestamp;
  
  //Populated from first FileChunk
  final int fileSizeInBytes;
  final int chunkSizeInBytes;
  
  //Calculated when each FileChunk is processed
  final int transferredBytes;
  final int bytesPerSecond;
  final int secondsRemaining;
  
  //Populated when process starts
  //Updated by calling `close`
  final String? filename;
}

Code: file_transfer_grpc_client.dart

This class implements the GRPC client; initiating GRPC connections and requests, as well as handling Stream creation to and from the local filesystem.

class FileTransferGrpcClient {
  /// Code-generated client
  final FileServiceClient _fileServiceClient;
  
  /// Requests
  Future<FileSendChangeNotifier> upload(XFile xFile)
  FileReceiveChangeNotifier download(String serverFilename, File output)
  
  /// Static Helpers
  static Stream<(Uint8List, int)> _calcOffset<T>(Stream<Uint8List> input)
  static Stream<Uint8List> _rechunkStream(Stream<Uint8List> input)
  static Iterable<Uint8List> _rechunkUint8List(Uint8List list, int length)
  static Future<Stream<FileChunk>> _readXFile(XFile xFile)
}

Flutter Widget: file_transfer_progress_bar_widget.dart

Flutter Widget: file_transfer_widget.dart

class FileTransferProgressBarWidget extends StatelessWidget {
  final ValueNotifier<FileTransferChangeNotifier?> fileTransferChangeNotifierNotifier;

  /// Returns % progress, display text: line1, line 2
  /// action = 'upload' or 'download'
  static (double, String, String) calculateTextAndProgress(
    FileTransferProgress fileTransferProgress, 
    String action,
  )
  
  @override
  Widget build(BuildContext context) {
    /// Displays a Column([ProgressBar, Text(line1), Text(line2)]);
    return ValueListenableBuilder();
  }
}

Comparison of gRPC to HTTP/2

Because gRPC is built directly on top of HTTP/2 it is understandable that for simple file transfers gRPC can be viewed as HTTP/2 with unnecessary overhead. For certain simple tasks, gRPC can never reach the resource efficiency of a pure HTTP implementation.

For this reason, use-cases with significant volume or large file transfers will see noticeably reduced server resource demands using HTTP instead of gRPC. However, this comes with the burden of maintaining another server cluster for HTTP, or sideloading an HTTP server onto the gRPC server. The HTTP performance gain becomes a trade-off against system complexity.

Most gRPC overhead comes from the intentional copying of in-memory data models. The Java gRPC implementation will recopy userspace data to yet another array simply as a precaution: ensuring no code references and data immutability. This to enable assertions about internal state in order to optimize serialization code paths.

On the other hand, HTTP servers can be optimized for their simpler code paths without extra work; reading and writing data directly from storage to network with zero or minimal memory buffering and CPU processing.

HTTP/2 Backpressure

The gRPC server implementations leverage the flow control in HTTP/2 streams to apply backpressure. The particular settings are implementation dependent, but can usually be modified when initializing the server implementation. Performance tuning the gRPC implementation is beyond the scope of this article, with the takeaway being that how much data the server will receive from a client before blocking the client stream will also impact transfer performance, memory usage, and code settings such as chunk_size.

Protobuf Definition

Feature Requirements

The primary requirements of a file transfer mechanism are:

• No additional encoding overhead
• Ability to determine progress

Other feature concerns related to client and server implementation will be discussed later, but compatibility should still be verified when architecting the proto models. These features would include:

• the ability to handle partial-file resumption,
• multiplexed / concurrent segment transfers, and
• pre transfer actions, such as authentication, permissions, collision detection and quotas
• post actions, such as file renaming, name sanitation, or moving completed from temp to final directories.

The gRPC stream has the least amount of network overhead for an indeterminate amount of data. Each gRPC message is an identical type, in our case we will define a stream of FileChunk messages.

gRPC supports call metadata, which are directly like an HTTP header it is possible to send additional data which would not be part of each FileChunk in the stream. Perhaps, fields which do not vary for each message such as filename and file_size would more efficiently be sent as call metadata, but this is typically

message FileChunk {
  //Name of file
  string filename = 1;
  //Total size of file
  uint64 file_size = 2;
  //Starting offset of current chunk
  uint64 offset = 3;
  //Binary data of chunk
  bytes body = 4;
}

Server Definition

service FileService {
  rpc GetFile (GetFileRequest) returns (stream FileChunk);
  rpc SetFile (stream FileChunk) returns (SetFileResponse);
}

Server Implementation

Implementation Assumptions

Before getting into the server code, a set of assumptions have been made for simplicity.

First, the grpc connection is assumed to have been authenticated, and clients have full access to read/write to the server’s filesDirectory directory. The value of filesDirectory might depend on the authenticated user of the client, with each separate user having access to only a home directory.

The sample javaFile function maps a filename request parameter to a server java.io.File, and implements no sanitation on the value. Obviously allowing clients to enter .. and / characters within the filename will result in security vulnerabilities.

private def javaFile(unsafeFilename: String): File = {
  File(s"$filesDirectory/$unsafeFilename")
}

Put requests where filename already exists will overwrite the existing file. Concurrent requests to read/write the same file will result in corruption. How to solve this depends on requirements, but a common first recommended change would be to write to a temporary server directory and move completed files to a readable directory only after the upload has completed successfully.

Client GetFile (Download)

A GetFileRequest client request will result in the server streaming filename back to the client in a chunk size set by the server. The request could easily be expanded with an offset field to allow partial file resumption, or both offset and end_offset fields to allow concurrent download streams.

message GetFileRequest {
  string filename = 1;
}

The ZIO server implementation for the generated Protobuf is:

def getFile(request: GetFileRequest): Stream[StatusException, FileChunk]

The body is fairly simple after creating 2 private helper functions around the ZIO NIO file library.

One function to return the file_size of a file using ZIO Files.size.

private def readFileSize(file: File): IO[IOException, Long] = {
  val path = Path.fromJava(file.toPath)
  Files.exists(path)
    .filterOrFail(_ == true)(FileNotFoundException(file.getName))
    .flatMap(_ => Files.size(path))
}

And another to create a read stream of the file using ZStream.fromPath, and luckily ZIO will chunk the stream to a specified chunkSize.

private def readFile(file: File): UStream[ByteString] = {
  ZStream.fromPath(file.toPath, chunkSize = chunkSize)
    .chunks.map(chunk => ByteString.copyFrom(chunk.toArray))
    .catchAll { 
      ex => 
        ZStream.fromZIO {
          ZIO.logErrorCause(s"Error reading file ${file.getName}", Cause.fail(ex))
        }.drain
    }
}

The only logic left will be to convert the ZIO NIO file stream to a FileChunk stream. The only complexity here is that each chunk will depend on the previous chunk. The offset will simply be a running total of the body size of all previous FileChunk. ZIO mapAccum implements a stateful stream mapping, the state being the count of sentBytes.

override def getFile(request: GetFileRequest): Stream[StatusException, FileChunk] = {
  val file = javaFile(request.filename)
  ZStream.fromZIO(readFileSize(file))
    .flatMap { fileSize =>
      readFile(file).mapAccum(0L)((sentBytes, byteString) {
        val fileChunk = FileChunk.of(
          filename = file.getName,
          fileSize = fileSize,
          offset = sentBytes,
          body = byteString,
        )
        (sentBytes + byteString.size, fileChunk)
      })
    }
    .catchAll { ex =>
      ZStream.fromZIO(ZIO.fail(StatusException(io.grpc.Status.fromThrowable(ex))))
    }
}

There are many types of IO related errors which can happen accessing local files, our implementation will opt to return the default GRPC error status on all failures.

Client SetFile (Upload)

As mentioned at the head of the article, GRPC allows metadata content to be part of call headers. Storing upload parameters as headers would simplify a streaming approach by removing the requirement to inspect the head element of the stream for parameters such as the filename and file_size.

Alternative GRPC Metadata Headers Approach

Standard GRPC practices are to use the call metadata to store call agnostic data, such as authentication, tracing, and other information which will apply to all calls. The .proto file specification doesn’t include the ability to define call message headers, so associating headers with calls will make .proto files an incomplete documentation of the call.

Moreover, GRPC services are generated code creating an inflexibility to specifying individual call signatures. Each signature within a service will contain the same additional parameter. So a modified setFile will need to choose between having a generically typed header field or residing in a separate service class.

ZIO Approach

The setFile request has the Scala signature:

def setFile(request: Stream[StatusException, FileChunk]): IO[StatusException, SetFileResponse]

The SetFileResponse response returns a filename, useful in the case where the input filename has been modified, such as stripping out illegal characters, adding a version identifier or translating to a UUID or URI. Our implementation will mirror the input.

message SetFileResponse {
  string filename = 1;
}

The implementation will use a helper class, as the stream is stateful. The head FileChunk of the stream will create a new state by opening an AsynchronousFileChannel on the server that subsequent stream elements will append to. The SaveFileAccum state will also continue to update its offset field, which while unnecessary to function it will continue to be verified against the offset of the incoming stream elements.

case class SaveFileAccum(
  asynchronousFileChannel: AsynchronousFileChannel,
  file: File,
  totalSize: Long,
  offset: Long,
)

The code for the setFile function will be center around creating a ZSink, a ZIO Stream class that processes a stream and returns a final output value. Our sink will have the signature:

ZSink[Scope, StatusException | IOException, FileChunk, Nothing, Option[SaveFileAccum]]

The type parameters can be a bit intimidating at first, but are straight forward. Our ZSink needs a Scope to run in because it contains an open AsynchronousFileChannel which will need to be closed. It will throw both StatusException and IOException, but we could reduce this to just GRPC StatusException if made our sink a little more complicated by handling all IO errors internally. It processes a stream of FileChunk items and will process all of them so it will have Nothing remaining elements, and its output will be an Option[SaveFileAccum].

ZSink Output

The choice to return an Option[SaveFileAccum] allows us to externalize more code from the sink than if it returned a SetFileResponse directly.

After the sink runs, the value of Option[SaveFileAccum] is either:

• If Some then a file was created. Either:
  • The client’s filesize matches our file size means the upload was successful, or
  • The file size doesn’t match, meaning the upload was incomplete.
• If None, then no file was created.

The ZIO code for this is:

ZIO.scoped {
  request.run(sink).flatMap {
    case Some(SaveFileAccum(_, file, expectedSize, actualSize)) if expectedSize == actualSize =>
      val response = SetFileResponse.of(file.getName)
      ZIO.succeed(response)

    case Some(SaveFileAccum(_, file, expectedSize, actualSize)) =>
      ZIO.logError(s"Upload ended at $actualSize of $expectedSize") *> ZIO.fail(StatusException(CANCELLED))

    case _ =>
      ZIO.logError("Could not create file") *> ZIO.fail(StatusException(UNKNOWN))
  }
}

ZSink Processing

The processing done by the sink to the stream of FileChunk has a few responsibilities. First, it will need to inspect the first element of the stream to look for the upload parameters. The upload parameters will include the filename which will be used to create the AsynchronousFileChannel, and an expected filesize, which will be used to verify a successfully completed stream. (Additionally we could include an MD5 or other hash to verify content as well as file size.)

Our code will include a verification that the expected upload file size is below a maximum, as we don’t have unlimited storage resources. Moreover, the processing of each FileChunk will continue to verify that the running total of bytes uploaded is below our server maximum and equal to the expected file size.

The sink will process the stream using a foldLeftZIO since we will need to maintain a state (Option[SaveFileAccum]) updating it during each chunk processing.

ZSink.foldLeftZIO(None)((saveFileAccum, fileChunk) {
  //case 1: invalid chunk size, throw INVALID_ARGUMENT
  //case 2: file channel not open => open file channel and write first chunk
  //case 3: file channel open, but chunk offset != expected throw INVALID_ARGUMENT
  //case 4: file channel open, but chunk greater than remaining bytes
  //case 5: channel open, chunk valid => append to file channel
})

Case 1: Invalid Chunk Size

Separate from maxFileSize it is also important to limit how large individual chunks can be. Noted earlier, gRPC will perform multiple copies of the body data so allowing too large of chunks will result in more memory churn and GC pressure. For example, allowing 100MB chunk sizes may require > 300MB of memory to properly process the request.

case _  if fileChunk.body.size() > maxChunkSize =>
  ZIO.logError(s"Chunk size ${fileChunk.body.size()} exceeds maximum $maxChunkSize") 
    *> ZIO.fail(StatusException(INVALID_ARGUMENT))

Case 2: File Channel Not Open

This should only run on the first chunk. It will either create an open AsynchronousFileChannel or determine that the file upload is invalid and throw an exception.

val file = javaFile(fileChunk.filename)
val path = Path.fromJava(file.toPath)
val chunk = Chunk.from(fileChunk.body.asScala.map(Byte.unbox))
if (fileChunk.fileSize > maxFileSize) {
  ZIO.logError(s"File too large, attempted ${fileChunk.fileSize} bytes")
    *> ZIO.fail(StatusException(OUT_OF_RANGE))
} else if (fileChunk.fileSize < chunk.length) {
  ZIO.logError(s"Invalid chunk ${chunk.length} exceeds total size ${fileChunk.fileSize}")
    *> ZIO.fail(StatusException(INVALID_ARGUMENT))
} else {
  for {
    _ <- ZIO.log(s"Uploading file ${path.toString}, size ${fileChunk.fileSize}")
    channel <- AsynchronousFileChannel.open(
        path,
        StandardOpenOption.WRITE,
        StandardOpenOption.TRUNCATE_EXISTING,
        StandardOpenOption.CREATE
      ) 
    _ <- channel.writeChunk(chunk, 0)
  } yield {
    Some(SaveFileAccum(channel, file, fileChunk.fileSize, chunk.length))
  }
}

Case 3: File Channel Open But Invalid Offset

Verifies that this FileChunk is at the expected offset.

case Some(SaveFileAccum(_, _, _, offset)) if fileChunk.offset != offset =>
  ZIO.logError(s"Invalid chunk offset ${fileChunk.offset}, expected $offset")
    *> ZIO.fail(StatusException(INVALID_ARGUMENT))

Case 4: File Channel Open, But Chunk Greater Than Remaining Bytes

Verifies that this FileChunk body doesn’t exceed the remaining number of bytes left in the stored file.

case Some(SaveFileAccum(_, _, totalSize, offset)) if fileChunk.body.size() > totalSize - offset =>
  ZIO.logError(s"Invalid chunk ${fileChunk.offset} exceeds total size $totalSize")
    *> ZIO.fail(StatusException(OUT_OF_RANGE))

Case 5: File Channel Open, Write to Channel

All invalid FileChunk cases have been checked, the body should be appended to the open file channel and the sink state updated with the new expected offset for the next element of the stream.

val chunk = Chunk.from(fileChunk.body.asScala.map(Byte.unbox))
val saveFileAccum = SaveFileAccum(
  asynchronousFileChannel = asyncFileChannel,
  file = file,
  totalSize = totalSize,
  offset = offset + chunk.length,
)
asyncFileChannel.writeChunk(chunk, offset).as(Some(saveFileAccum))

Conclusion

The next post in this series will create a Flutter client for this server component.

pubspec.yaml Dependencies

This sample client requires 2 Flutter libraries:

• grpc core implementation for networking.
• flutter_bloc …
• built_collection …
• path_provider …

The testing client for a performance test can be the same as one created for local unit tests.

Performance Testing Implementation

A sample implementation for performance testing would timestamp on Data element updates allowing clients to compare time lag between server updates and their notification of it.

Performance testing implementation and results are covered in Realtime Client Database Performance Testing.

class ClockExternalDataLayer private(clock: Clock, refreshSchedule: Schedule[Any, Any, Any])
  extends ExternalDataLayer(refreshSchedule) {

  override protected def externalData(
                                       chunk: NonEmptyChunk[Either[DataId, DataRecord]]
                                     ): UIO[Chunk[Data]] = {
    clock.instant.map {
      now =>
        val dataId = either.fold(identity, _.data.id)
        chunk.map(either => Data.of(dataId, now.toString))
    }
  }
}

case class GrpcClient(
                       userId: UserId,
                       requests: Queue[SyncRequest],
                       responses: Stream[Throwable, (UserId, SyncResponse)],
                       grpcLayer: ZLayer[Any, Throwable, SyncServiceClient],
                     ) {

  def update(request: UpdateRequest): IO[Throwable, UpdateResponse] = {
    SyncServiceClient.update(request).provideLayer(grpcLayer)
  }
}

object GrpcClient {
  def launch(
              userId: UserId,
              serverAddress: String,
              serverPort: Int,
            ): ZIO[Scope, Nothing, GrpcClient] = ZIO.scoped(Queue.unbounded[SyncRequest]).map {

    requests =>

      val grpcLayer: ZLayer[Any, Throwable, SyncServiceClient] = SyncServiceClient.live(
        ZManagedChannel(ManagedChannelBuilder.forAddress(serverAddress, serverPort).usePlaintext()),
        options = CallOptions.DEFAULT,
        metadata = SafeMetadata.make((AuthenticatedUser.MetadataUserIdKey, userId.toString)),
      )

      val responses = SyncServiceClient
        .bidirectionalStream(ZStream.fromQueue(requests))
        .provideLayer(grpcLayer)
        .map((userId, _))

      GrpcClient(userId, requests, responses, grpcLayer)
  }
}

Data ids: 1 - 600 are second markers. 601 - 1000 are random

Clients: Subscribe 1 - 600, subscribe i mod clientid

External Data using a ZLayer

The ZLayer mechanism in ZIO conveniently defines environment dependencies, and will be used to create an external data service implementation. The implementation details will depend on how and where the external data is stored, but it will be exposed with a simple interface mapping data ids to new data elements.

The ZLayer trait will contain the related methods used to transform inputs and outputs, as well as scheduling updates. With protected visibility implementations can be overriden accomidating non-standard operation.

/**
 * Data layer that will refresh all subscribed data an external datasource based on a schedule.
 *
 * @param refreshSchedule
 */
abstract class ExternalDataLayer(refreshSchedule: Schedule[Any, Any, Any]) {

  /**
   * Implementation details for resolving elements in queue to external `Data`.
   *
   * @param chunk Will contain the `DataRecord` for records existing in `databaseRecordsRef`
   * @return `Data` from external source when available.
   */
  protected def externalData(chunk: NonEmptyChunk[Either[DataId, DataRecord]]): UIO[Chunk[Data]]

  /**
   * All DataId from this queue are fetched from external datasource.
   * Updates are commited to `databaseRecordsRef` and emitted to `journal`.
   * Creation of queue will start the automatic data refresh using `refreshSchedule`.
   * Closing queue will stop the scheduled refresh.
   * Calling multiple times will execute schedules in parallel, 
   * useful for manual refresh using `Schedule.once`.
   */
  def createFetchQueue(
                        journal: Hub[DataRecord],
                        databaseRecordsRef: Ref[Map[DataId, DataRecord]],
                        globalSubscribersRef: Ref[Set[Ref[HashSet[DataId]]]],
                      ): UIO[Enqueue[DataId]]

  /**
   * Whenever DataId are queued for refresh, call `externalData` to get data,
   * Update `databaseRecordsRef` when new data available without conflicts,
   * Emit data updates to `journal`.
   */
  protected def attachFetchQueueListener(
                                          queue: Dequeue[DataId],
                                          journal: Hub[DataRecord],
                                          databaseRecordsRef: Ref[Map[DataId, DataRecord]]
                                        ): UIO[Unit]

  /**
   * On schedule, attempt to refresh all subscribed data.
   */
  protected def attachRefreshScheduler(
                                        queue: Enqueue[DataId],
                                        globalSubscribersRef: Ref[Set[Ref[HashSet[DataId]]]]
                                      ): UIO[Unit]
}

External Data Sample Implementations

Hardcoded Data

A sample implementation useful for testing will allow data to be specified as a construction parameter. The data will be part of the ZLayer instance with it consumed by calls from the server. This will allow subsequent calls to return different data values allowing for state-based unit testing. However this requires strict ordering, so calls to externalData requires single concurrency. Java threading uses syncronize for serialize execution, in ZIO fiber synchronization is through Ref.Synchronized.

class HardcodedExternalDataLayer private(
                                          hardcodedData: Ref.Synchronized[Seq[Data]],
                                          refreshSchedule: Schedule[Any, Any, Any]
                                        )
  extends ExternalDataLayer(refreshSchedule) {

  override protected def externalData(
                                       chunk: NonEmptyChunk[Either[DataId, DataRecord]]
                                     ): UIO[Chunk[Data]] = {
    hardcodedData.modify {
      fetchData =>
        chunk.foldLeft((Chunk.empty[Data], fetchData))({
          case ((foldData, foldFetchData), either) =>
            val dataId = either.fold(identity, _.data.id)
            foldFetchData.find(_.id == dataId).map {
              data => (foldData :+ data, foldFetchData.filterNot(_ eq data))
            }.getOrElse {
              (foldData, foldFetchData)
            }
        })
    }
  }
}

ZIO Resources and Scope

ZIO Resources form a contract preventing resource leaks and ensuring proper finalization of a closure. A job queue which maintains observability of in-progress jobs has the same concern. Active jobs are open resources and need to properly finalization after being consumed, in the same way an open file needs to be properly closed of after use.

The approach is release queue items with an attached ZIO Scope. Work can be performed within this scope, and the scope can be responsible for marking queue items either as consumed or to be returned back into the queue due to processing failure.

Queue Features

• Maintain a distinct list of queue entries.
  If objects are added multiple times the queue will only contain the first object, in its correct queue position.
• Automatically remove popped queue items after their work has been completed This allows work in-progress to count towards the item uniqueness. Re-adding work that is already in-progress will be rejected by the queue.
• Popping from the queue is a blocking operation There is no need to poll the queue for new items, all consumers can stream items and fetch batches using thread-safe operations.
• For simplicity, there is no dead-letter output Items which cannot be processed are returned to the queue by the Scope finalizer.

Class Interface

class DistinctZioJobQueue[A] {

  //Jobs queued for execution.
  def queued: ZIO[Any, Nothing, Seq[A]]

  //Jobs currently executing.
  def inProgress: ZIO[Any, Nothing, Seq[A]]

  //Add job to queue, will return `true` if successful. Jobs already in queue will return `false`.
  def add(elem: A): ZIO[Any, Nothing, Boolean]

  //Add jobs to queue. Will return all jobs that failed to be added.
  def addAll(elems: Seq[A]): ZIO[Any, Nothing, Seq[A]]

  //Blocks until returning at least one, but no more than N, queued jobs.
  def takeUpToNQueued(max: Int): ZIO[Scope, Nothing, Seq[A]]

}

ZIO Concurrency uses Fibers, Not Threads

Using Semaphore for Concurrency

The common approach to create concurrent collections in Java is using the JDK provided wrapper Collections.synchronizedSet(). This is a great mechanism for handling thread-safety, however ZIO concurrency operates with ZIO fibers making this approach untenable. It is incorrect to block threads at any time in ZIO outside a ZIO.blocking scope because this will block all fibers using that thread. Fibers are the independent workers in ZIO, not threads, and blocking the system thread will cause performance degradation and possible deadlocks.

The ZIO Semaphore is the ZIO equivalent mechanism to provide synchronization. It operates on the fiber level making it distinctly different from a JDK semaphore. The same concurrency concerns are still valid while using fibers to access LinkedHashSet methods. All write operations must be synchronized, and all read operations can only parallelize with other read operations. Any read operation occurring during a write operation is vulnerable to a ConcurrentModificationException if it its iterator encounters stale state.

Ref and Ref.Synchronized

In addition to semaphore, ZIO provides other concurrency mechanisms such as Ref and STM. Software Transactional Memory is a powerful construct however requiring specialized implementations of common classes, making it worthy of its own external discussion. The Ref construct is a very accessible mechanism in ZIO comparable to the AtomicReference class in the JDK, but at a higher level. A noticeable downside is it is only suitable for immutable references. Our implementation uses a mutable LinkedHashSet.

Ref and Hash Codes

There are fundamental differences between the Java and Scala library implementations of LinkedHashSet.

java.util.LinkedHashSet

The Java implementation of LinkedHashSet is a mutable implementation, and modifying it will not change its hashCode. This means that wrapping it within either a Ref or Synchronized will cause all atomic guarantees to brake. The atomicity is implemented using hashCode verification to detect write conflicts rather than thread synchronization to the memory address. This is better for performance, but in this case it will effectivily behave as if there were no write management at all.

scala.collection.mutable.LinkedHashSet

The Scala implementation of LinkedHashSet is a mutable implementation, however it has a dynamically computed hashCode. This will allow it to function correctly within a Ref under certain conditions. By incurring a performance overhead during all writes it will allow the AtomicReference.compareAndSet method to work correctly.

override def hashCode: Int = {
  val setIterator = this.iterator
  val hashIterator: Iterator[Any] =
    if (setIterator.isEmpty) setIterator
    else new HashSetIterator[Any] {
      var hash: Int = 0

      override def hashCode: Int = hash

      override protected[this] def extract(nd: Node[A]): Any = {
        hash = unimproveHash(nd.hash)
        this
      }
    }
  MurmurHash3.unorderedHash(hashIterator, MurmurHash3.setSeed)
}

There still remains the inability of LinkedHashSet to handle reads during writes. The primary benefit of immutability is data never changes preventing all data instability during a read operations.

The motivation of the dynamic hashCode for the mutable LinkedHashSet in the Scala Collections Library is to allow equality between immutable/mutable variants. Comparing a mutable Set to a immutable Set with the same internal elements will be successful, creating a more powerful Set interface abstraction.

To allow the Scala LinkedHashSet to work within a Ref we will need to accept a hashCode calculation penalty on all writes, as well as implement a gate around simultaneous read/write operations. This may be suitable in some situations, but optimizing for fast reads is a more generally acceptable approach, and for these reasons our queue implementation is better off using a semaphore directly.

Blocking in ZIO

The primary mechanism to block ZIO fibers is by mapping from await on a Promise. When a second fiber completes the promise using succeed the first fiber will unblock and resume execution. Within our queue, all consumers can await the same activity promise. Whenever the queue state changes making queued elements available we will call notifyActivity to complete the promise. The unblocked consumers are in a FIFO priority and will execute in order until the queue is empty, initiating another blocked state.

private def notifyActivity: UIO[Unit] =
  for {
    resetPromise <- Promise.make[Nothing, Unit]
    _ <- promise.succeed(())
  } yield promise = resetPromise

The notifyActivity method will complete the current promise, and replace it with a new uncompleted promise. All calls to notifyActivity are within the semaphore write permit so access is guaranteed to be exclusive to the fiber, allowing this to be a simple var instead of a Ref.

Queue Write Operations

The queue can be modified in 3 ways: elements added, elements removed, and elements undergoing status change. These map to add/addAll calls and Scope creation / finalization. Exclusive access to the queue is enforced by reserving all permits via semaphore.withPermits(MaxReadFibers). This call will block until any outstanding permits reserved by read-only operations have been returned, and will block any new read-only permits from being obtained.

def add(elem: A):
//- obtain permits to LinkedHashSet
//- try to add to queue
//- if added successfully call notifyActivity
//- release permits

def addAll(elems: Seq[A]):
//- obtain permits to LinkedHashSet
//- try to add all to queue
//- if any added successfully call notifyActivity
//- release permits

def takeUpToNQueued(max: Int):
//- obtain permits to LinkedHashSet
//- return items with Scope
//- release permits
//- Scope finalizer:
//    - obtain permits to LinkedHashSet
//    - returns items to queue on exception
//    - calls notifyActivity
//    - release permits
//  or
//    - obtain permits to LinkedHashSet
//    - remove items from queue
//    - calls notifyActivity
//    - release permits

Implementation details of add / addAll are straight-forward queue enqueue operations. The return values of these may be immaterial for many use-cases. The example code emits enqueue outcomes to the server➤client stream, but for network efficiency, these can be omitted.

The takeUpToNQueue implementation is best examined in 3 parts:

• taking from the queue
• handling zero queue elements
• creating a scope with finalizer

Creating Scope

Using acquireReleaseExit

The Scope is a trait and not typically defined as a named class. It should normally be anonymously constructed using one of the acquireRelease methods. There are variations, the simplest being acquireRelease where the acquire is an action to get an A and release is an action to perform on A to close it.

This queue will require the more advanced acquireReleaseExit method, which has the same acquire but the A as well as Exit are available to the release as a tuple. ZIO Exit is the functional equivalent of the Scala Try, resolving into either a Success or Failure. Failures can be the result of either exceptions or interruptions.

Scope Interruption

The other Scope creators acquireReleaseInterruptible and acquireReleaseInterruptibleExit need to be avoided here. They lack the ability to determine the queue elements which were part of the scope being closed. This ability is critical in the correct operation of this queue (without very advanced logic being added). Because of this reason, the takeUpToNQueue has been broken into 3 stages:

Taking N from Queue

//TODO:

Handling Zero Queue Elements

This is separated

Creating Scope with Finalizer

This is covered in the Creating Scope using acquireReleaseExit section below.

Defining Scope acquire and release Finalizer

def acquire: ZIO[?, ?, Seq]
//- obtain permits to LinkedHashSet
//- iterate elements in queue collecting up to N unflagged
//- flag elements as being taken
//- if any elements unflagged call notifyActivity
//- release permits

A line-by-line implementation of the Scope acquire would be:

semaphore.withPermits(MaxReadFibers) {
  val flagged = linkedHashSet.iterator
    .filter(_.status == Status.Queued)
    .take(max)
    .map {
      jobStatus =>
        jobStatus.status = Status.InProgress
        jobStatus.a
    }
  if (flagged.nonEmpty) notifyActivity.as(flagged) else ZIO.unit
}

The finalizer will be outcome dependent, with a path for successful Scope closure and one for failure.

def release: (Seq, Exit[Any, Any]) => ZIO[?, Nothing, Any] = {
  //if Seq has elements from the Queue:
  //- obtain permits to LinkedHashSet
  //- loop through all elements in Seq
  //- if Exit was success:
  //  - remove element from Queue
  //- if Exit was failure:
  //  - unflag element as taken in Queue
  //- if any elements unflagged call notifyActivity
  //- release permits
}

A line-by-line implementation of the Scope release finalizer would be:

(takeUpToNOption, exit) =>
  takeUpToNOption.fold(false)(seq =>
    semaphore.withPermits(MaxReadFibers) {
      val activity = seq.foldLeft(false)((hasActivity, a) =>
        exit match {
          case Exit.Success(_) =>
            val _ = linkedHashSet.remove(JobStatus(a, Status.InProgress))
            hasActivity
          case Exit.Failure(_) =>
            val _ = linkedHashSet.find(_.a == a).foreach(_.status = Status.Queued)
            true
        })
      if (activity) notifyActivity else ZIO.unit
    }
  )

Closing Scope

The mechanism to close the scope will depend on the actions performed by the consumer. The Scope is part of the ZIO effect’s environment and appears in the type signature until it is removed. Explicitly defining a scope boundary can be done using the ZIO.scope partial function. This is a closure around effect code, and drops it from the environment type:

ZIO.scope(zio: ZIO[Scope, ?, ?]): ZIO[?, ?, ?]

Streaming Consumers

Typical consumers would adopt a stream pattern. As the queue releases NonEmptyChunk elements within a Scope, consumers should opt to complete the scope as soon as possible, however long-running scope have very minimal impact on the queue as it is already optimized minimal memory consumption and internal iteration performance.

val consumer: ZStream[Any, Throwable, T] = ZStream.repeatZIO {
  ZIO.scoped {
    queue.takeUpToNQueued(?).map {
      nonEmptyChunk => ??? //function to create Ts
    }
  }
}

//TODO:

Element Uniqueness using equals / hashCode

The basic requirement for this queue will be able to flag entries as either queued, or having been released within a scope by a consumer.

private enum Status {
  case Queued, InProgress
}

To attach a status field to each entry, we can wrap entries with a JobStatus class, but overwrite the equals and hashCode fields such that they will only consider the queued item, and not the current status. This ensures that trying to add a duplicate entry will be blocked regardless of the existing item’s status.

Feature Extensions using JobStatus Fields

The JobStatus class could be expanded to support additional queue functionality. Queue extensions typically include performance metrics: queued time, in-progress time, and operational metrics: add-collision count and failure-count from any failed consumer executions.

private class JobStatus[A](val a: A, var status: Status) {
  override def hashCode(): Int = a.hashCode()

  override def equals(obj: Any): Boolean = obj match {
    case o: JobStatus[?] => o.a.equals(a)
    case _ => false
  }
}

Queue Class and Private Members

There are 3 private variables in the queue (these are constructor params to avoid Unsafe construction). They are all intended to be internal references only.

class UniqueJobQueue[A](
                         private val semaphore: Semaphore,
                         private var promise: Promise[Nothing, Unit],
                       ) {
  val linkedHashSet = new mutable.LinkedHashSet[JobStatus[A]]
}

The semaphore will be the read/write synchronization for the linkedHashSet. The promise will be used to signal consumers to retry the queue because elements have been added. The promise is a mutable var but will only be modified behind the semaphore write guard, ensuring no write conflicts.

Conclusion: Putting It All Together

The example code utilizes type alias, such as type URIO[-R, +A] = ZIO[R, Nothing, A], as well as using Chunk and NonEmptyChunk instead of Seq.

class SynchronizedUniqueJobQueue[A](
                         private val semaphore: Semaphore,
                         private var promise: Promise[Nothing, Unit],
                       ) {
  
  private val linkedHashSet = new mutable.LinkedHashSet[JobStatus[A]]

  def add(elem: A): ZIO[Any, Nothing, Boolean]

  def addAll(elems: Seq[A]): ZIO[Any, Nothing, Seq[A]]

  def takeUpToNQueued(max: Int): ZIO[Scope, Nothing, Seq[A]]
  
  private def takeUpToQueuedAllowEmpty(max: Int): ZIO[Scope, Nothing, Option[Seq[A]]]

  private def notifyActivity: ZIO[Any, Nothing, Unit]
}

gRPC server streaming and ZIO Hub can implement this functionality replicating an expensive paid Firebase service while allowing greater extensibility.

Evolving User Demands for Data Streaming

Typical web-based client-server communication is to have clients initiate all requests to the server. This minimizes load on the server as very minimal processing needs to be performed between requests. As server technology and hardware performance have increased, more robust paradigms have evolved to cater to rising user expectations. To reduce notification latencies, it was imperative to allow server request initiation. Immediately reaction by the server to
external changes can be set to clients without waiting for manual user initiations or for a polling delay to elapse.

Bi-Directional gRPC Streaming using HTTP/2

Two technology standards for bi-directional web communications have become standards: WebSockets and HTTP/2 streams.

Today’s Complementary Web Standards

WebSockets

WebSockets were created first of the two as an extension to the HTTP/1.1 standard. By allowing clients to issue an Upgrade request on an established HTTP/1.1 connection, capable servers can switch the connection to use the websocket network protocol, supporting bi-directional streaming. Since this is an upgrade mechanism, backwards compatibility and fallback support for older clients has less complication to implement. The WebSocket protocol has a common JavaScript API across all browsers, making it today’s preferred bidirectional protocol within browser use-cases. Despite websocket being an unversioned specification it has seen improvements over time by way of it being built on top of an HTTP protocol. HTTP/2 significantly improved connection performance through connection multiplexing, allowing websockets to benefit when clients use multiple websocket connections simultaneously as they are now being multiplexed over the single upgraded HTTP/2 connection.

HTTP/2 Streaming

For non-browser communications such as by mobile apps or server-server communications, WebSockets is an unnecessary layer. WebSockets is an upgraded HTTP connection, implemented by creating an additional layer in the networking protocol. The changes in HTTP/2 directly addressed bidirectional communication streams, so when the WebSocket API isn’t beneficial, it is optimal to use the HTTP protocol capabilities directly. HTTP/2 streaming is the preferred bidirectional mechanism for all use-cases without a web browser client or JavaScript dependency.

gRPC: A Remote Procedure Call (RPC) framework

gRPC is a high-performance networking framework built on top of HTTP/2, supporting multiple programming languages for both client and server implementation. It encodes network traffic using Protocol Buffers using proto files and syntax, which also are used to define the server API exposed to clients.

A gRPC server which will be used for the realtime database will be called SyncService, and expose a bidirectional stream on the Bidirectional endpoint, receiving Request objects and emitting Response objects to their respective streams.

A Update endpoint has also been defined, this is the typical client-initiated handler, which will receive a UpdateRequest object from the client and return a UpdateResponse object to the client.

service SyncService {
  rpc Bidirectional (stream Request) returns (stream Response);
  rpc Update(UpdateRequest) returns (UpdateResponse);
}

This maps to the Scala interface:

  /**
 * Requests will subscribe/unsubscribe to `Data`.
 * Data updates of subscribed elements is streamed in realtime.
 */
def bidirectionalStream(request: Stream[StatusException, SyncRequest]): Stream[StatusException, SyncResponse]

/**
 * Creation / Update of `Data`. Response will indicate success or failure due to write conflict.
 * Conflicts are detected based on the ETag in the request.
 */
def update(request: UpdateRequest): IO[StatusException, UpdateResponse]

Subscription Request and Response Objects

A basic subscription mechanism will allow clients to subscribe and unsubscribe to object updates based on the object id. The protobuf definition for requests is:

message SyncRequest {
  repeated Subscribe subscribes = 1;
  repeated Unsubscribe unsubscribes = 2;
}

The definition for responses is the Data object itself:

message SyncResponse {
  Data data = 1;
}

Within the server implementation, we will define a Subscription Manager which will remember subscriptions for a specific client. ZIO has a primitive called ZIO Hub which allows objects such as these Subscription Manager to subscribe to a singular, central message queue. The queue will receive notifications to all Data updates, and each client Subscription Manager will subscribe to the Hub and filter for events of the ids its client has requested. Because this Hub will queue the stream of database changes, we will name it journal since database journals have similar behaviour.

A helpful feature is to return the current Data object back in the client response when initiating a subscription. This operation has been indicated by a yellow dashed line in the function diagram.

Data

The Data class will represent an arbitrary data record class, code will rely on the presence of an id field, here represented as an uint32. While this type doesn’t exist in Java, it adds clarity to the API, but as the Protocol Buffers Documentation API Best Practices indicates, limits to even the int64 addressable range may make a string id preferable. The field1 field represents an arbitrary field, it could be extrapolated to have Data contain additional fields (field2,field3 etc.).

message Data {
  uint32 id = 1;
  string field1 = 2;
}

Adding Data Updates

ETag and Timestamp

ETags are part of the HTTP Specification and exist to reduce network transfer. The HTTP If-None-Match header, when implemented signals that should the response have the same generated ETag that the server should respond with a HTTP 304 Not Modified instead of a 200 Success with a populated body.

The usefulness of an ETag depends on server support: APIs may implement ETag support similar to HTTP Specification and use it to omit a response body, others may use it internally to return a previous response from its cache, while others solely include it as a convenience for clients.

Our API will use an ETag to have our server only return a full Data object on subscription if the client either doesn’t have a previous copy (ie: no ETag available) or has a stale version (ie: conflicting ETag).

To support this functionality, as well as many others which may depend on fetch/cache durations, we’ll associate an ETag and last updated time to all Data elements by wrapping them in a new DataRecord class:

case class DataRecord(data: Data, lastUpdate: Instant, etag: ETag)

/**
 * Update `databaseRecordsRef` with data in `request`, rejecting any conflicts.
 * Conflicts are based on the included ETag:
 * If the `previousEtag` doesn't match the ETag in the database it is a conflict. 
 * New item creation ignores the `previousEtag` field.
 * All database item creation / updates are emitting to `journal`.
 */
def updateDatabaseRecords(
                           request: UpdateRequest,
                           journal: Hub[Data],
                           databaseRecordsRef: Ref[Map[Int, Data]]
                         ): UIO[UpdateResponse]

We will need to call an effect to connect the Subscription Manager to the journal Hub:

/**
 * Create Stream from database `journal`
 */
def userSubscriptionStream(
                            userSubscriptionsRef: Ref[HashSet[Int]],
                            journal: Hub[Data]
                          ): ZIO[Scope, Nothing, Stream[StatusException, SyncResponse]]

We will need to connect the

/**
 * Update `userSubscriptionsRef` with subscription changes.
 */
def modifyUserSubscriptions(
                             syncRequest: SyncRequest,
                             userSubscriptionsRef: Ref[HashSet[Int]],
                             databaseRecords: Map[Int, Data]
                           ): UIO[Seq[SyncResponse]]

//TODO:

The context: AuthenticatedUser parameter is for a feature exposed by ZIO to handle gRPC metadata. Every gRPC request is able to provide headers, similar to how HTTP allows request headers. ZIO generates 2 interface

Collections versus Streams

Pragmatic programming moved complexity from projects to external libraries and frameworks. This allowed even small improvements in libraries to have over-sized benefit scaling proportionally to their popularity. However, this has the unintended consequence of developers never learning underlying concepts, both limiting their work within project implementations directly and through misusing or under-using available libraries. From personal observation, streams falls into this category, their performance benefits often passed-over until they are absolutely required by software to function correctly.

HTTP File Transfers

It is inappropriate to write ETLs assuming the entire dataset will fit into RAM. The OS will attempt to manage if given enough virtual memory, but it will be blind to program implementation details and unable to perform optimally. Beyond this, by not streaming data directly from source to destination the additional steps can inefficiently be required to copy data multiple times to different locations, noticeably decreasing throughput based on available I/O.

Intermediate Nodes and Proxies

When data is transferred from source to destination, any proxies buffering substantial amounts of data can cause performance degradation if using the wrong implementation. Proxies should use streams when possible, unless the buffering is serving an explicit purpose, such as increasing data durability, smoothing irregular network flow or providing batch/windowing.

File Uploads without Streams

A common scenario where streaming is required, rather than optional, is handling file uploads. An typical example encountered is to fully buffer a file upload within a proxy. In the following example, the user is allowed to upload a file within the Salesforce UI, with the destination of the file to be stored in SharePoint.

There are 3 different approaches to this:

• File is uploaded to Salesforce, then copied to SharePoint
• File is buffered in Salesforce memory, then copied to SharePoint
• File streams to SharePoint, either directly or proxies through Salesforce servers

The first 2 are without streaming and suffer from transfers A and B being sequential rather than concurrent. The user must wait for B after A has completed. This also allows new modes of failure: what happens when B fails, will A have to be repeated? Will the user know B will fail before starting A?

Not only is the streaming solution faster, it has fewer modes of failure.

Backpressure

The idea of backpressure is that consumers can dictate throughput to the producer. The consumer essentially applies pressure backwards against the path of flow towards the producer. Streams have different implementations and not all streams allow for backpressure. Naive buffering can prevent backpressure signals from traveling back towards the source. But when buffering is limited or non-existent, if the consumer is slower than the producer then there is nowhere for the data to build up. For file transfers, the backpressure solution is obvious such that the producer isn’t allowed to upload faster than the consumer can process.

This is ideal for file transfers because we want the outcome of the consumer to be relayed to the producer. If the file upload to SharePoint B doesn’t complete, we don’t benefit from A completing successfully.

Strategies for HTTP File Transfers

Large File Transfers

Large file transfers have additional concerns over smaller files:

• Ability to track progress,
• Ability to pause/resume over difference sessions,
• Ability to parallelize,
• Ability to support unknown or unlimited final file size

Content-Range Requests

The simplest strategy is to stream file contents within a single HTTP request body. A chunked transfer encoding can be specifying using a Transfer-Encoding: chunked header allowing unknown or unlimited request size, however the request has limited recoverability since there is no mechanism to pause, resume or parallelize for performance. The Content-Range header is a common approach to expand into these features by replacing a single large request with multiple smaller requests. Each request will specify a byte range to be transmitted, allowing the requests to be parallelized, retried, and monitored.

Alternative SDK Approaches

While the Content-Range approach is a common standard built into all browsers, alternatives exist such as within the AWS SDK. The AWS SDK builds out additional file features:

• Ability to list complete/incomplete segments
• Management of in-progress transfers
• Support a file lifecycle

AWS has chosen not to use content-range headers, instead utilizing a custom part identification scheme. Each request will include the part identifier within the URL query parameters instead of within a header.

Not covered within the content-range approach is how to manage in-progress transfers. Are each range tied to a specific HTTP session? Can they be deleted? How can we list incomplete or range failures? By exposing additional URLs to list transfers in progress, list transfer part statuses, and manage/delete transfers, AWS has both increased the durability of the file transfer process and removed state management complexity from client agents.

The final feature of a file lifecycle exists to mark transfers as complete, such that they can be transferred from temporary locations used during the upload to final destinations. This is important because of the granularity of destinations, some being referred to as cold or offline storage, which would normally not be directly accessible to clients.

HTTP Multipart Requests

A variation to a singular HTTP request body is the multipart request. It is indicated by way of a Content-Type: multipart/form-data header. As can be inferred from the form-data name, the primary use-case was the expansion of the HTTP specification to directly support form data transfers without dependency on external encoding schema such as JSON.

The multipart mechanism separates the request body into separate parts, each with its own headers and body. Each of the parts is free to specify their Content-Type thereby allowing each part to be a binary file transfer.

The intended use of a multipart upload is to handle multiple, small data fields, typically being user text input. From a high level, the HTTP request is broken up into parts delimited by an arbitrary boundary sequence specified within the Content-Type header.

There are multiple reasons this is unsuited to transferring large files:

• Content needs to be inspected for boundary occurrences,
• Unable to know name, count, or content-type of parts without parsing previous parts,
• No additional features supporting large files beyond those of the single request format.

The multipart form-data request should be considered a strictly worse version of putting the file content within the request body. The cost to support multiple files within the request introduces the overhead of the data comparison against the boundary. While algorithms exist to minimize overhead, it should generally be considered to be O(n) where n is the file content length. Expansions of HTTP with HTTP/2 and HTTP/3 have removed request overhead so there is reason to use form-data to transfer anything but trivially small files.

Real-World Examples: Amazon Alexa API

The Amazon Alexa device makes use of multipart form-data requests to communicate with the Alexa Voice Service (AVS) servers. Every request includes JSON metadata and binary audio data as separate parts. There is an inherent incompatibility between JSON and binary data: there are no efficient ways to embed binary data into the text format, and embedding JSON into binary data requires an additional encoding mechanism.

The binary encoding selected was multipart form-data, which would reduce complexity by being a well-supported part of HTTP and requiring no additional dependencies. While more efficient, binary encodings such as gRPC/protobuf, Thrift, and JSONB are often overlooked because of the need to add library dependencies for parsing, inspection, and debugging of the over-the-wire data.

As mentioned earlier, the boundary mechanic creates a CPU bottleneck working against higher throughput. It should be noted that this inefficient processing can be omitted when the multipart format is of a particular type. When the only binary part is at the end of the request, there would be no need to inspect the binary data for boundaries. With this relaxation of the format, the multipart form-data encoding is just as performant as the binary encodings mentioned while retaining the human-readable over-the-write representation of the JSON part.

Http4s EntityDecoder

The org.http4s.EntityDecoder trait includes the documentation:

A type that can be used to decode a Message EntityDecoder is used to attempt to decode a Message returning the entire resulting A. If an error occurs it will result in a failed effect. The default decoders provided here are not streaming, but one could implement a streaming decoder by having the value of A be some kind of streaming construct.

It is also readily apparent from the org.http4s.multipart.Multipart implementation:

final case class Multipart[+F[_]](
       parts: Vector[Part[F]],
       boundary: Boundary,
     )

As mentioned earlier, multipart form-data requests are not a good mechanism to handle large file uploads. There has been no effort to optimize the streaming capabilities of the Http4s Multipart handler, opting to represent the parts as an immutable Vector. To construct this instance, the entire request will need to be parsed, negating any request streaming ability.

It is key to note that Out-Of-Memory issues are circumvented by way of temporary files, the implementation of this decoder has chosen to create and manage filesystem objects as a way to offload large requests from system RAM. Http4s implicits can be explicitly created to modify default values for this mechanism, by creating a decoder using the MultipartDecoder static method mixedMultipartResource:

def mixedMultipartResource[F[_] : Concurrent : Files](
       headerLimit: Int = 1024,
       maxSizeBeforeWrite: Int = 52428800,
       maxParts: Int = 50,
       failOnLimit: Boolean = false,
       chunkSize: Int = 8192,
     ): Resource[F, EntityDecoder[F, Multipart[F]]]

Streaming Multipart File Uploads with Http4s

There is no streaming issues when directly using the request.body exposed by Http4s. It is a Stream[F, Byte], the issues mentioned in this post are concerning the included body decoders breaking streaming semantics. This shouldn’t be seen as criticism or oversight, as the direct parsing of the body is a cleaner, more direct and the recommended approach to handle streaming request data. Instead of the inefficient use of multipart form-data, the same effect is better achieved by moving all text-based form-data to HTTP headers. But for the stubborn and niche use-cases such as with Alexa above, can the Http4s multipart decoder be implemented to support streaming?

Problems converting a Stream to Stream-of-Streams

An HTTP request body should be viewed as a single stream.

//TODO: finish

a single stream, how can a multipart body be represented using a stream? Conceptually it maps to a Stream[Part] since parts will need to be accessed sequentially. But pragmatically, each part could represent a large file which would also need to be a stream. A single Stream cannot map to a Stream[Stream[_]] since this wouldn’t allow each part, which can represent a large file, ecause we need to , but then what is a Part? It cannot be a s to be as efficient as possible? multipart request streaming be properly implemented? With HTTP/2 multiplexing, is there even a plausible use-case to send multiple fields in a single request?

//TODO: implementation details for streaming Multipart decoder

Testing Streams: Memory Use

Streams are programming abstractions which are difficult to test directly, as their output is identical to their collection counterparts. It is insufficient to test the final output is being streamed, since any intermediate transformation could have easily buffered results only to stream them again. The absolute test would be to measure memory use of the system, as any buffering will have a measurable effect on heap use. For the purpose of our testing, restricting the JVM heap size to be smaller than the stream data would indicate no intermediate buffering. (This cannot detect buffering to the filesystem, which would require additional code inspection).

Running the JVM with a 128Mb memory allocation can use the Xmx parameter:

java -Xmx128m -jar build-action-file-receiver-assembly-1.0.25.jar 

Implementation

//TODO: Scala implementation

Conclusion

//TODO: conclusion

After 7 years Scala Native is at pre-production maturity version 0.4.17.

Different versions of OpenJDK

To avoid confusion between Scala Native, and Scala compiled natively it’s important to be clear on the goals of alternative OpenJDK implementations. Initially Oracle released Java and Java’s JDK/JRE but licencing has changed causing open source, free to use, alternatives to emerge that are compatible with the Java SE Specification.

From a high level, whichjdk.com highlights the JDK products supported by various cloud computing entities; AWS, Azure, IBM, SAP, and RedHat all have JDKs. There are minor differences such as in their Garbage Collector tuning, but for the most part they are unremarkable in their similarity.

Eclipse OpenJ9

Eclipse OpenJ9 is a cloud optimized JDK which rethinks OpenJDK implementation while offering full compatibility. With focus on JVM startup times and memory management, the notable new features include:

• Checkpoint/Restore in Userspace (CRUI),
• Class Data Sharing,
• partial Ahead-of-Time (AOT) compilation, and
• aggressive GC tuning.

GraalVM

The goal of GraalVM is of Java using strictly Ahead-of-Time (AOT) compilation, rather than using Java’s Just-in-Time Compilation. This is a breaking change but allows compilation to a native executable rather than Java’s class bytecode.

Scala has 2 SBT plugins with GraalVM support:

• SBT Native Image
• SBT Native Packager

The benefits of GraalVM native executable is removal of the JVM initialization time and the removal of any unreachable code, resulting in smaller packages which boot quickly. Unfortunately the largest downside to the removal of JIT is removal of any runtime performance optimization based on use. The AOT is able to optimize but without any visibility into common paths and hotspots it is from a strictly static perspective. In addition, the optimizations come from Java bytecode which has already removed some key ingredients for full static analysis. In real-world benchmarks, GraalVM performance typically lags behind JIT competitors for long-running tasks.

Another breaking change in GraalVM use is the inability to use Java reflection and runtime code generation. This is more common in Java than in languages such as Scala, but any program can be affected. There are 2 approaches to work around this limitation:

Quarkus and optimized libraries written without JIT requirements

The popular Kubernetes focused framework Quarkus avoids JIT with specialized versions of standard libraries. The changes necessary can be minor, allowing for a well-supported ecosystem of popular libraries available as extensions to Quarkus, without notable lag to official library releases. However, support for any library not available as an extension will rely on the second JIT alternative.

Reflection Metadata Collection and Code Hints

The GraalVM compiler can perform static analysis of code to learn about certain types of JIT and reflection, but runtime observation with the GraalVM Tracing Agent is required. Code needs to be run under all execution paths, and the Tracing Agent will record all usages of the Java Native Interface (JNI), Java Reflection, Dynamic Proxy objects, or class path resources. These will then be supplied as a JSON document to the AOT on the next compilation. The obvious problem is triggering all executable code paths during the debugging cycle, making this (theoretically) a trial and error process.

Motivations for Scala Native

ScalaJS, the Scala to JavaScript Transcompiler

The focus of Scala Native isn’t explicitly native execution, it is in the removal of the restrictive JDK dependency. Scala Native has 2 supported outputs:

• ScalaJS JavaScript, and
• LLVM executables.

ScalaJS can realistically only transcompile to JavaScript if it can remove any unused overhead imposed by the JDK. Typical JavaScript use cases are web oriented, where code sizes dictate latency and observed performance.

The JDK and Java bytecode are opinionated in their interactions with other native code. The Java Native Interface (JNI) is an unnecessary and in often a performance restricting abstraction that is only necessary to bridge between the JRE and external native code. The LLVM linking for native-native interactions offers low level primitives such as pointers and unsafe memory access that can be necessary to achieve top performance. Scala Native includes additional Scala support for the low-level C/C++ primitives directly which cannot be represented in Java.

Executables and Jars

Java can be run directly in the JRE using java -jar. It is essentially just as easy to run a Jar as a native executable (assuming a JRE is installed). As noted previously, AOT executables may initialize quicker but often bested by JIT on long-running performance. This leaves short-lived container applications such as: AWS Lambda, serverless functions, and Kubernetes the primary target for GraalVM, with standard JDK or OpenJ9 remaining the best choice for server instance deployments or longer running serverless. So where exactly does Scala Native fit in?

CLI Tools and JNI-heavy applications

The benefit of a Scala Native LLVM executable is it is just another C/C++ program. In the same way Rust, Zig and Go have emerged as replacements to C/C++, Scala Native offers the same compatibility and performance with a familiar and powerful Scala language. It is the native compatibility and low-level features brought to Scala via Scala Native that make Scala Native a compelling choice with applications which require low-level management or direct hardware interactions.

Compiling Scala Native in a GitHub Action

As outlined previously, the use-case of Scala Native is typically a low-level CLI application. This remainder of this article deviates from this by working through a Scala application which will not benefit from Scala Native.

This is not a recommendation by the author to apply Scala Native to a similar project, merely a discover exercise

GitHub Action Runners

The linux server configurations outlining OS and installed software is available, and includes suitable CLang and LLVM versions. For local development the Scala Native install guide covers for macOS (using brew) and linux using apt.

Compiling additional C libraries with CMake

It is very likely that a Scala Native will have a library dependency not installed on the GitHub Runner. In our application the FS2 functional library requires the Amazon AWS TLS/SSL library S2N. Security and cryptography libraries are likely to be native dependencies both because they likely already exist as high-performance native implementations, and secondly because Scala Native JDK may omit them due to security concerns around implementation.

Compiling a C application with CMake is a straight-forward addition to the GitHub Action YML:

- name: Compile and Install AWS S2N-TLS
  run: |
    # clone s2n-tls
    git clone --depth 1 https://github.com/aws/s2n-tls.git
    cd s2n-tls
    # build s2n-tls
    cmake . -Bbuild \
        -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_INSTALL_PREFIX=./s2n-tls-install
    cmake --build build -j $(nproc)
    CTEST_PARALLEL_LEVEL=$(nproc) ctest --test-dir build
    cmake --install build

Linking C library paths in SBT

When linking to a GitHub Action compiled C libraries such as S2N, the library needs to be in a path scanned by LD. Because LD is being called by CMake and not GCC, typical environmental variables such as LD_LIBRARY_PATH and LIBRARY_PATH do not work.

TODO: CMAKE_EXE_LINKER_FLAGS LDFLAGS

The easiest option is to include the path in build.sbt though this path will differ between local development and within the GitHub Action.

For example, within the build-action-file-receiver project on GitHub, the runner installed S2N in the following path:

nativeLinkingOptions += s"-L/home/runner/work/build-action-file-receiver/build-action-file-receiver/s2n-tls/s2n-tls-install/lib"

Scala Native library dependencies in SBT

The build-action-file-receiver is a typical Scala web server application, but with compatibility to run Scala Native making it suitable for this article. Scala Native compatibility is dictated by the availability of Scala Native versions for all library dependencies. These are exist as separate artifacts:

• https://repo1.maven.org/maven2/co/fs2/fs2-core_3/3.9.4/fs2-core_3-3.9.4.jar (1.5MB)
• https://repo1.maven.org/maven2/co/fs2/fs2-core_native0.4_3/3.9.4/fs2-core_native0.4_3-3.9.4.jar (3.7MB)

This is indicated in build.sbt by using the %%% operator instead of % / %%:

Typical Scala:

libraryDependencies ++= Seq(
  "org.http4s" %% "http4s-ember-client" % http4sVersion,
  "org.http4s" %% "http4s-ember-server" % http4sVersion,
  "org.http4s" %% "http4s-dsl" % http4sVersion,
  "org.typelevel" %% "log4cats-core" % "2.6.0",
  "co.fs2" %% "fs2-io" % "3.9.4",
  "org.scala-lang.modules" %% "scala-xml" % "2.2.0",
  "org.scalatest" %% "scalatest" % "3.3.0-alpha.1" % Test,
  "org.typelevel" %% "cats-effect-testing-scalatest" % "1.5.0" % Test,
)

Scala Native:

libraryDependencies ++= Seq(
  "com.armanbilge" %%% "epollcat" % "0.1.4",
  "org.http4s" %%% "http4s-ember-client" % http4sVersion,
  "org.http4s" %%% "http4s-ember-server" % http4sVersion,
  "org.http4s" %%% "http4s-dsl" % http4sVersion,
  "org.typelevel" %%% "log4cats-core" % "2.6.0",
  "co.fs2" %%% "fs2-io" % "3.9.4",
  "org.scala-lang.modules" %%% "scala-xml" % "2.2.0",
  "org.scalatest" %%% "scalatest" % "3.3.0-alpha.1" % Test
)

The %%% operator and SBT nativeLink task to compile the native executables/libraries are added by the SBT Scala Native plugin, and additional configuration options are straight forward parameters covered in the setup documentation.

Code Changes Required to Support Scala Native

Cross Compiling

There exists the SBT-CrossProject plugin that will hopefully alleviate issues with compiling to Scala 3 Jar and Scala 3 Native.

TODO: Examples on this working.

Partial JVM Implementations

Multithreading and Runtime Environment

The Scala Native runtime environment is a very basic implementation lacking a threading model and thread synchronization primitives. The current recommendation is to import and use necessary C libraries. To achieve basic functionality libraries such as FS2 require an additional library to operate in Scala Native, such as the I/O runtime epollcat.

Scala libraries are typically built around event-loops and/or monads, so the use of Node.js’s I/O event library libuv has been made available as Scala Native Loop.

Cryptography and java.security package

While the Scala Native lists many JDK packages as implemented, some packages such as java.security have intentionally been left incomplete. This can have unexpected touch points, so while a TLS/SSL implementation was a predictably omitted, the dummy implementation of all hash functions, including MD5 and SHA1 in java.security.MessageDigest was not.

object MessageDigest {

  def isEqual(digestA: Array[Byte], digestB: Array[Byte]): Boolean =
    true

  def getInstance(algorithm: String): MessageDigest =
    new DummyMessageDigest(algorithm)
}

private class DummyMessageDigest(algorithm: String)
  extends MessageDigest(algorithm) {

  override protected def engineDigest(): Array[Byte] = Array.empty

  override protected def engineReset(): Unit = ()

  override protected def engineUpdate(input: Byte): Unit = ()

  override protected def engineUpdate(
                                       input: Array[Byte],
                                       offset: Int,
                                       len: Int
                                     ): Unit = ()
}

Scala-XML

The incompleteness of the Scala-XML library was a little puzzling. What is the point of publishing an XML library when it can’t pass the most simple, obvious use-case.

[error] Found 21 missing definitions while linking
[error] Not found Top(javax.xml.parsers.SAXParser)
[error] 	at file:/home/runner/work/scala-xml/scala-xml/shared/src/main/scala/scala/xml/factory/XMLLoader.scala:51

The problem with "org.scala-lang.modules" %%% "scala-xml" % "2.2.0" is that javax library hasn’t been implemented.

Runtime Observations

Performance and Memory

For a cross-compiled project, the http-maven-reciever project compiled to an 27.9MB executable “Fat Jar”. The debug-mode Scala Native produced a 41.4MB linux executable.

[info] done compiling
[info] Linking (2666 ms)
[info] Checking intermediate code (quick) (194 ms)
[info] Discovered 6982 classes and 47495 methods
[info] Optimizing (debug mode) (3964 ms)
[info] Generating intermediate code (5599 ms)
[info] Produced 10 files
[info] Compiling to native code (17076 ms)
[info] Total (29698 ms)
[success] Total time: 32 s, completed Feb. 7, 2024, 10:29:39 a.m.

35MB

[error] ld: Assertion failed: (aliasSectionNum == sectionNum && "alias and its target must be located in the same section"), function assignAliasAtomOffsetInSection, file Layout.cpp, line 3324.
[error] clang: error: linker command failed with exit code 1 (use -v to see invocation)

TODO: optimized release mode file size TODO: startup with Scala 3 Jar TODO: load benchmarks with Scala 3 Jar TODO: memory footprint with Scala 3 Jar

Unit Tests

Scala Native officially supports the major testing libraries. TODO: running tests

Debugging

The debugging of native executables and libraries is not as developer friendly as JIT code. There is less debugging information available within the executables for IDEs to interpret. A typical debugging experience using LLDB is dramatically different from JVM tooling; meaning cross compilation has practical utility within the entire software development lifecycle.

Conclusions

Pros

• Fast and direct calling of native dependencies

Cons

• Painful debugging
• Long compile/linking times
• Needs Scala Native to compatible dependencies
• Subtle code changes
• Hard to use resource and configuration files
• Usually need to make JVM version work

Scala Native is a welcome addition to the C/C++ alternatives for Scala developers. This ecosystem has room to grow as performance and latency bottlenecks are no longer determined by hardware, but by the minuscule overheads determined at compile time. Latency can’t be solved by more parallelism or faster CPUs, but by slimming the instruction and cache footprint of code. Rust and Go have taken off in popularity, and Scala Native, if done right has all the same benefits.

But to make this a compelling choice, Scala Native shouldn’t be confused with Scala’s JVM presence. Longtime users of ScalaJS are familiar with the restrictions to the lihaoyi libraries, and Scala Native have the same burden. Projects using Scala Native should be selective, and primarily oriented to using C native libraries or staying within the small pockets of JVM libraries such as lihaoyi or typelevel which have been early native adopters.

The advice to use Scala Native for only CLI tools is on point. While it will be a long time before this changes, it doesn’t need to for Scala Native to be successful. The JDK dominated space doesn’t need native via Scala Native, advancements in alternative JDKs such as GraalVM are already offering native support in a more suitable and easier to apply route. As it’s always been, select the right tool for the right job.

GitHub Packages as a Maven repository

In an earlier article, Downloading from GitHub Packages using HTTP and Maven we investigated GitHub Packages as a Maven repository for Java artifacts. Standard practices with any network service is evaluating the benefit of implementing a local cache. A local cache can speed up downloads, allow customized permissions, and increased resiliency against network failures. Even if a local Maven repository proxy cache such as Artifactory or Nexus is selected, the question remains of how to get artifacts into the local cache if GitHub Packages transfer limits are being hit.

GitHub Actions have Unlimited Egress (Transfer-Out)

In a GitHub CI/CD pipeline, where compilation occurs within a GitHub Action, a solution is to utilize the unlettered egress bandwidth available during post-compilation actions. Compiled artifacts can be stored both in GitHub Packages and transferred to a Nexus or Artifactory proxy cache when bandwidth is available for free, avoiding metered egress being made from the proxy to GitHub Packages.

Security and Implementation of a push solution

Maven Repository Checksums

The question is now of how to securely transfer files out of a GitHub Actions to remote endpoints. All receivers should accept only authenticated requests from GitHub Actions. Does this require receivers to implement token authentication? Practically speaking no, taking a step back this isn’t about user authentication, it is about file authentication. For a file to be authentic it needs to exist within GitHub Packages. If the receiver is uploaded a file, that file is authentic and secure if and only if it has a corresponding Maven checksum in our GitHub Packages repository.

Authentication Flow

An authentication mechanism beyond GitHub is unwanted and unnecessary. HTTP requests to any external receiver can be authenticated by verifying the request includes a valid GitHub auth token. The files the requests are uploading can be validated by accessing GitHub Packages using the GitHub auth token and comparing checksums.

HTTP Server for receiving and validating GitHub Action upload requests

This project is basically server-side deployment scripts written in Scala, with Akka HTTP receiving builds from GitHub, so it can easily be integrated as a Route of existing Akka HTTP / Play deployments. User Permissions

Upload permissions are limited to the ability to publish to GitHub Packages Maven.

Server-side permissions are completely internal to your server. Two Deployment Parts

SBT build tasks

publishAssemblyToGitHubPackages: pushes compiled code to GitHub Packages (Maven)
uploadAssemblyByPut: pushes compiled code to your server (HTTP PUT)

HTTP Upload Server

built on Akka, handles HTTP PUT
validates upload is latest version in Maven, and has correct MD5 checksum
performs any custom server-side tasks, such as deployment and restarting