Serializing data in C

I've started work on a new project! The project in question is one that I have wanted to tackle for a long time but did not have the courage to do so. However at long last it is time...

time to get funky...

I'm writing a music player, and not one that simply plays music like spotify. This music player falls more inline with MPD in which it's a daemon running in the background playing the music and clients may communicate with it to tell it what to play when and the likes.

Currently I'm at that very important part in which the daemon needs to communicate to clients via a protocol *fancy*, and to do so I have decided to go with a socket that way in theory I can control my daemon from other devices on the network. With the communication method decided I now have to define what data goes between devices. At the current moment I've settled on an enum for different signals.

As seen here:

enum libmoo_signal {
  /* core signals are in the < 100 range */
  LIBMOO_SIGNAL_ESTCON = 1,
  LIBMOO_SIGNAL_ESTDCON = 2,
  LIBMOO_SIGNAL_INCOMPATABLE_VERSION = 3,
  LIBMOO_SIGNAL_OK = 50, /* previous message received was ok */
  LIBMOO_SIGNAL_NOK = 51, /* previous message received was not ok */
  /* setting/adding data happens in the 100 range */
  LIBMOO_SIGNAL_ADD_SONG = 101,
  LIBMOO_SIGNAL_SET_PLAY = 103,
  LIBMOO_SIGNAL_SET_PAUSE = 104,
  LIBMOO_SIGNAL_SET_TOGGLE_PAUSE = 105,
  LIBMOO_SIGNAL_SET_SKIP_NEXT = 110,
  LIBMOO_SIGNAL_SET_SKIP_PREV = 111,
  /* removing data happens in the 200 range */
  LIBMOO_SIGNAL_REM_SONG = 201,
  /* getting data happens in the 300 range */
  LIBMOO_SIGNAL_GET_CUR_SONG = 301,
  LIBMOO_SIGNAL_GET_PAUSE = 303,
  LIBMOO_SIGNAL_GET_PROGRESS = 310,
};
    

As for how I actually transmit the signals, I've settled on a format which defaults to 42 bytes of data being sent with the option to send more by setting the 41st-42nd bytes to a uint16 containing the additional number of bytes being sent.

/* byte - data
 *
 * 0-7  - libmoo protocol version
 *
 * 8-31  - reserved for the future
 *
 * 32-40 - libmoo_signal
 *
 * 41-42 - uint16 more_data
 *
 * 43-8191 - char *data
 */
typedef char * libmoo_payload;
    

I'm not sure how this compares to other protocols used to transmit data, but I'm quite proud of how I've laid this out.

Moving on to actual the point of this article: serialization. For the sake of useablilty I've defined a datatype which models our payload which you can see here:

typedef struct {
  enum libmoo_signal signal; /* the signal */
  uint16_t           more_data; /* the number bytes of the additional data */
  char              *data; /* additional data */
} libmoo_data;
    

And a function which allows you to easily create libmoo_data:

/**
 * @brief create a new libmoo_data
 *
 * @param signal the signal to set
 * @param data the data to set
 * @return the data object
 */
libmoo_data *libmoo_create_data(enum libmoo_signal signal, char *data);
    

I've provided this so that you don't have to deal with manually setting the size of the data that you pass into the object, although (if you so choose) you may easily override it.

Serializing

Now that we've gone over the interface for interacting with the payload let's get to the intersting stage in which we get the data ready for launch.

/**
 * @brief convert data into a payload.
 * The payload must be allocated prior to calling this function it should be
 * LIBMOO_PAYLOAD_SIZE chars long unless you think you know better.
 *
 * @param payload pointer to the payload
 * @param data data which will be serialized
 * @return the size of the resulting payload
 */
size_t
libmoo_serialize_data(libmoo_payload payload, libmoo_data *data)
{
  void *ptr;
  ptr = payload;
  /* add LIBMOO_VERSION */
  mempcpy(ptr, LIBMOO_VERSION, strlen(LIBMOO_VERSION));
  /* skip the reserved space */
  ptr += 24;
  /* add the signal type */
  mempcpy(ptr, &data->signal, sizeof(data->signal));
  /* add the size of the more data */
  mempcpy(ptr, &data->more_data, sizeof(data->more_data));
  /* the more data */
  if (data->more_data) {
    mempcpy(ptr, data->data, strlen(data->data));
  }
  return LIBMOO_HEADER_SIZE + sizeof(data->more_data) + data->more_data;
}
    

As you may have noticed in the code snippet above libmoo_serialize_data is a function that takes in a pre-allocated payload, and the data to serialize. I've opted to allow the user to allocate space for the payload if they determine that they know what they're doing incase my dumb function is too generous. Moving onto the body of the function you'll see an unfamiliar function called mempcpy which is a macro for copying data into the payload. This macro takes in the same arguments as memcpy (dest, src, size), but in addition it increments dest by size so I can append data to the payload without making this code horrible to read.

In earlier stages of development I didn't actually return the size of the resulting payload, thinking that I could just strlen it. This was a rookie mistake. Because I've included extra space (bytes 8-31) when you run strlen on any payload it results in 5 (the length of the version string). After finding this out I decided that I do actually want to send more than the version information, and so now libmoo_serialize_data returns a size_t continaing the number of bytes that have been serialized.

Deserializing

Now that we've serialized and presumably sent the data to a client we need to do the magic part which completes this transaction: deserialization.

Below is my method of deserializing the data. It's pretty much the opposite of the serialization function with the key exception that we need to do some validation. Which I haven't implemented yet.

libmoo_data
*libmoo_deserialize_data(libmoo_payload payload, unsigned int payload_size)
{
  libmoo_data *data;
  char libmoo_version[LIBMOO_VERSION_LEN + 1];
  /* ensure that there's enough data to comply to spec */
  if (payload_size < LIBMOO_HEADER_SIZE) {
    errno = EBADE;
    return NULL;
  }
  /* attempt to allocate space and if we fail return NULL errno contains the
   * error
   */
  data = calloc(1, sizeof(libmoo_data));
  if (data == NULL) {
    errno = ENOMEM;
    return NULL;
  }
  /* get the libmoo_version */
  memscpy(libmoo_version, payload, LIBMOO_VERSION_LEN);
  libmoo_version[LIBMOO_VERSION_LEN] = '';
  /* TODO: check that the version of the library we're talking to and our
   * version are compatable
   */
  /* gotta tell the client that we're incompatible */
  if (strcmp(libmoo_version, LIBMOO_VERSION) != 0) {
    errno = EINVAL;
    free(data);
    return NULL;
  }
  /* skip the reserved space */
  payload += 24;
  /* get the signal */
  memscpy(&data->signal, payload, sizeof(data->signal));
  /* get the more data marker */
  memscpy(&data->more_data, payload, sizeof(data->more_data));
  /* get the data */
  if (data->more_data) {
    data->data = malloc((data->more_data + 1) * sizeof(char));
    memscpy(data->data, payload, data->more_data * sizeof(char));
    data->data[data->more_data] = ''; /* null terminate this string, yo */
  }
  return data;
}
    

And that's kinda it just like with the mempcpy macro I've made a memscpy macro which increments the src by the size of the data to make reading the data easy.

See you next year o/ _{I might have my project done by then, who knows}

When it comes to learning how to program there are a few things you can do:

1. Read a textbook
2. Watch videos
3. Read some source code

Of these options I find the best way to truly learn how to program is to read someone else's program and try and understand it. For example recently I've been working on my own dmenu clone for Wayland. Throughout working on it instead of looking for tutorials on how to render a square using pixman I decided to take a look at dtao which is a clone of dzen for Wayland. By just reading the code and messing around with the program I was able to get an understanding for how rendering is done in pixman.

Now you may be asking yourself something like: "But what does this have to do with suckless software?". The answer to that is in their philosophy which is about: "keeping things simple, minimal and usable". The idea of keeping things minimal and useable allows them to create wonderful programs that not only work, but also showcase how to do things without extra fluff that something like i3 might have.

Even if you don't like suckless software it still serves as a great place to learn how to do the bare minimum. And for those who do enjoy using it, it can serve as a great starting place to hack upon until you get the software of your dreams.

Recently, a few people have been asking me: "what is a squibid?" or "where did your name come from?". In this blog post I will answer those questions.

A few years ago I came up with a drawing of an animal reason to do anything with it, but regardless I chose to name it a squibid. Eventually, when trying to find a good username I chose squibid because that would cover both the username and profile picture.