musings

Serializing JSON in Scala with uPickle

Introduction

While working on a local Scala 3 project to track my music library, I quickly ran into the question of which JSON library to use.

In the past, I usually checked if the given project's Scala framework included a JSON library or referenced one in its documentation (e.g., Play JSON for Play Framework, jsoniter for smithy4s) and just started off with that if it worked well enough. I also used weePickle and circe at previous jobs. The point being, there are many options for working with JSON in Scala.

But as you can guess from the title, I chose to embrace the unknown and try yet another new (to me) library: uPickle. It seemed fitting because the project started as an excuse for me to learn some new Scala tools.

Setup

While my real project uses a Pekko HTTP server, these examples will use Cask instead for brevity.

Let's start with one endpoint, GET localhost:8080/releases, that returns a list of albums:

val releases: Seq[Release] = ??? // TODO: The response data

object App extends cask.MainRoutes {
  @cask.get("/releases")
  def getReleases() = {
    val json = upickle.default.write(releases)
    json
  }

  initialize()
  println("Server running on localhost:8080")
}

We'll build on this in the following sections.

Case class serialization

The first step is defining a Release case class so we can instantiate the val releases: Seq[Release] = ???:

case class Release(title: String, format: String)

val releases = Seq(
  Release(title = "Undoing Ruin", format = "album"),
  Release(title = "Blackout!", format = "album"),
)

Compiling the project now yields the following error:

[error] No given instance of type upickle.default.Writer[Seq[Release]] was found.

What's happening here is that the endpoint wants to serialize the releases with upickle.default.write(releases), but uPickle doesn't recognize Release as a serializable type without a pickler (ReadWriter[T] that can read and write JSON) in scope.

We can provide a ReadWriter[Release] in one of two ways:

  1. With upickle.default.macroRW, as an implicit value in the Release companion object:

    import upickle.default.{ReadWriter, macroRW}

case class Release(title: String, format: String)
object Release {
  given ReadWriter[Release] = macroRW
}

  2. With the derives keyword:

    import upickle.default.ReadWriter

case class Release(title: String, format: String) derives ReadWriter

Calling the endpoint now returns the Release objects as JSON:

$ curl localhost:8080/releases
[
  { "format": "album", "title": "Undoing Ruin" },
  { "format": "album", "title": "Blackout!" }
]

Nested serialization

The API is working, but the response data isn't very useful as-is. Let's add to it by including artist information with each release.

First, create an Artist case class and add it as a field on Release:

import upickle.default.ReadWriter

case class Release(
    title: String,
    format: String,
    artists: Seq[Artist],
) derives ReadWriter

case class Artist(name: String, country: String)

And then add artists to the releases collection:

val releases = Seq(
  Release(
    title = "Undoing Ruin",
    format = "album",
    artists = Seq(Artist("Darkest Hour", "USA")),
  ),
  Release(
    title = "Blackout!",
    format = "album",
    artists = Seq(
      Artist("Method Man", "USA"),
      Artist("Redman", "USA")
    ),
  ),
)

But we get the same error we saw earlier:

[error] No given instance of type ReadersVersionSpecific_this.Reader[Seq[Artist]] was found.

What went wrong?

The existing pickler knows how to handle every member of Release except for the new artists: Seq[Artist] because Artist is a case class. Every case class encountered while traversing fields during serialization needs its own pickler to tell uPickle how to handle it.

So we derive a ReadWriter[Artist]:

import upickle.default.ReadWriter

case class Artist(name: String, country: String) derives ReadWriter

And now the endpoint response includes the artist data:

$ curl localhost:8080/releases
[
  {
    "artists": [
      { "name": "Darkest Hour", "country": "USA" }
    ],
    "format": "album",
    "title": "Undoing Ruin"
  },
  {
    "artists": [
      { "name": "Method Man", "country": "USA" },
      { "name": "Redman", "country": "USA" }
    ],
    "format": "album",
    "title": "Blackout!"
  }
]

Custom serialization

Everything looks good so far, but we can make the response even more useful.

Let's include release dates to eventually support chronological sorting. And for good measure, they'll be optional to support old demo tapes and such whose release dates were lost to time and even the artists themselves can't remember.

Working with LocalDate

If we add a java.time.LocalDate field to Release:

import java.time.LocalDate
import upickle.default.ReadWriter

case class Release(
    title: String,
    format: String,
    artists: Seq[Artist],
    releaseDate: Option[LocalDate] = None,
) derives ReadWriter

case class Artist(name: String, country: String) derives ReadWriter

And then add dates to the releases collection:

val releases = Seq(
  Release(
    title = "Undoing Ruin",
    format = "album",
    artists = Seq(Artist("Darkest Hour", "USA")),
    releaseDate = Some(LocalDate.parse("2005-06-28")),
  ),
  Release(
    title = "Blackout!",
    format = "album",
    artists = Seq(
      Artist("Method Man", "USA"),
      Artist("Redman", "USA")
    ),
    releaseDate = Some(LocalDate.parse("1999-09-28")),
  ),
)

We get the same error once again:

[error] No given instance of type ReadersVersionSpecific_this.Reader[Option[java.time.LocalDate]] was found.

So we try to derive a ReadWriter[LocalDate]:

LocalDate derives ReadWriter
[error] value derives is not a member of object java.time.LocalDate
[error] LocalDate derives ReadWriter
[error] ^^^^^^^^^^^^^^^^^

But that doesn't work! And the problem isn't super obvious from the error message.

Debugging the pickler derivation

What's different this time?

For one, LocalDate comes from java.time. That means it can't be a case class like Artist or Release:

case class A1() extends java.time.LocalDate
[error] class A1 cannot extend final class LocalDate
[error]   case class A1() extends java.time.LocalDate
[error]               ^

Okay, so not only is LocalDate not a case class, it's a final class (i.e. can't be extended). Let's find out if either fact is relevant.

What happens if we try to derive a ReadWriter for a regular class?

class A1() derives upickle.default.ReadWriter
[error] No given instance of type scala.deriving.Mirror.Of[A1] was found for parameter x$1 of method derived in class ReadWriterExtension. Failed to synthesize an instance of type scala.deriving.Mirror.Of[A1]:
[error] 	* class A1 is not a generic product because it is not a case class
[error] 	* class A1 is not a generic sum because it is not a sealed class
[error]   class A1() derives ReadWriter
[error]                      ^

How about for a final class?

final class A1() derives upickle.default.ReadWriter
[error] No given instance of type scala.deriving.Mirror.Of[A1] was found for parameter x$1 of method derived in class ReadWriterExtension. Failed to synthesize an instance of type scala.deriving.Mirror.Of[A1]:
[error] 	* class A1 is not a generic product because it is not a case class
[error] 	* class A1 is not a generic sum because it is not a sealed class
[error]   final class A1() derives upickle.default.ReadWriter
[error]                                            ^

The errors are the same, with talk of a missing Mirror instance because the class is neither case nor sealed. This could be promising. Can deriving a ReadWriter work for a case class or sealed class version of LocalDate?

Well, two problems come to mind:

  1. We can't just recreate the class with the case or sealed keywords in front of it
  2. The class is final, so we can't extend children and make them case or sealed classes either

At this point, it really doesn't look like we can derive a ReadWriter[LocalDate]. It's time for a different approach.

Creating a custom pickler

The alternative to deriving a pickler (and letting that handle everything for us) is creating a custom pickler with its own logic for converting LocalDate to a type that uPickle can write to JSON, and vice versa.

We'll use String as the conversion type because:

import java.time.LocalDate
import upickle.default._

given ReadWriter[LocalDate] = readwriter[String].bimap[LocalDate](
  date => date.toString,        // write operation
  str  => LocalDate.parse(str), // read operation
)

After a quick test to make sure the ReadWriter[LocalDate] works properly:

val dateString = "1999-12-31"
val date = LocalDate.parse(dateString)

val json: String = write(date)
// returns a JSON string with double quotes on the inside, so '"1999-12-31"', not '1999-12-31'

read(json) == date
// true

read(s"\"$dateString\"") == date
// true

We can call the endpoint and see it finally returning releases with both artists and release dates:

$ curl localhost:8080/releases
[
  {
    "artists": [
      { "name": "Darkest Hour", "country": "USA" }
    ],
    "format": "album",
    "releaseDate": "2005-06-28",
    "title": "Undoing Ruin"
  },
  {
    "artists": [
      { "name": "Method Man", "country": "USA" },
      { "name": "Redman", "country": "USA" }
    ],
    "format": "album",
    "releaseDate": "1999-09-28",
    "title": "Blackout!"
  }
]

Conclusion

My first impression of uPickle is very positive. It's easy to get started with (no boilerplate), doesn't have an illegible DSL or complicated API, and allows customization for more control over the serialization/deserialization process. You'll feel right at home if you've used any of the other com.lihaoyi libraries before and there really is something to be said about their "Easy, Boring, and Fast" approach to Scala (e.g., the API code I wrote with Cask at the beginning).

tl;dr: There's a good chance that uPickle will be my go-to JSON library for future Scala projects.

Source Code

This is a self-contained example with all of the work above.

NOTE: There's no explicit uPickle dependency because it comes bundled with Cask

If you have Scala CLI, you can start the server with scala-cli <FILENAME>.scala.

//> using dependency com.lihaoyi::cask:0.10.2

import java.time.LocalDate
import upickle.default._

// Models

given ReadWriter[LocalDate] = readwriter[String].bimap[LocalDate](
  date => date.toString,
  str  => LocalDate.parse(str),
)

case class Release(
    title: String,
    format: String,
    artists: Seq[Artist],
    releaseDate: Option[LocalDate] = None,
) derives ReadWriter

case class Artist(name: String, country: String) derives ReadWriter

// Data

val releases = Seq(
  Release(
    title = "Undoing Ruin",
    format = "album",
    artists = Seq(Artist("Darkest Hour", "USA")),
    releaseDate = Some(LocalDate.parse("2005-06-28")),
  ),
  Release(
    title = "Blackout!",
    format = "album",
    artists = Seq(
      Artist("Method Man", "USA"),
      Artist("Redman", "USA")
    ),
    releaseDate = Some(LocalDate.parse("1999-09-28")),
  ),
)

// Server

object App extends cask.MainRoutes {
  @cask.get("/releases")
  def get() = {
    write(releases)
  }

  initialize()
  println("Server running on localhost:8080")
}

#json #scala #scala:upickle