Introduction to GRPC and protobufs in Go
Protocol Buffers + gRPC in Go.
Every programmer has at some point written a basic HTTP/REST/JSON application. Even if it’s just a simple server that responds to /hello, and returns {"message": "Hello, World!"}, it’s a worthwhile exercise. HTTP is simple, flexible, and available in every language, as is JSON serialization and deserialization libraries. The flexibility and ubiquity of HTTP and JSON libraries makes the pair an obvious choice for cross-machine communication. It’s easy to make changes to your models, adding new fields and removing old ones when they are no longer needed by the client. But the HTTP/JSON/REST trio has it’s downsides; serialization/deserialization hell, performance impacts of HTTP connections, and server/client APIs being out of sync, just to name a few. There are solutions to these problems, but they’re limited in their effectiveness. You can write custom serializers or use annotations to tell your deserializers how to interpret your JSON. You can version your APIs. In the early stages of development at your company or organization, these are all aspects that allow you to build flexible services at scale. But at a certain point, these benefits become hindrances. Developers add features on features, making what were once readable models unreadable, and difficult to maintain. Teams forget to notify other teams of API changes, and services break and become incompatible. While there’s no solution for people being unable to predict and manage the future, Protocol Buffers and GRPC are two tools that make it easier for people to manage features, models, and services.
Protocol Buffers, Google’s solution for serializing structured data, solves serialization/deserialization issues by letting you to define your model once, regardless of how many languages or frameworks you need to read the model in. This solves performance issues by removing your message attribute names, replacing them with numbered tags, and compressing messages.
GRPC, the remote procedure call library also developed by Google, works along side protocol buffers to let you define services in the same location that you define your model, so it’s easier to keep the two in sync. You also get the features of other RPC libraries such as streaming, allowing you to avoid the overhead of HTTP. It has libraries for Go, Python, Java, C++, Ruby, Node.js, Objective-C, and PHP.
In this post I’m going to introduce just the basics of Protocol Buffers, and how they can be used in concert with GRPC. As you can probably tell from my recent posts, I like Go, so that’s the language I’ll be using for this post. The code repo for this post can be found at https://github.com/vogtb/writing-grpc-proto.
Defining a .proto message
A JSON representation of an email might looks something like this:
{
"to": "[email protected]",
"from": "[email protected]",
"subject": "protocol buffers",
"body": "they are super cool",
"timestamp": 1479528508574
}
To represent this email message as a .proto, the first thing is to define our email.proto file.
message Email {
string to = 1;
string from = 2;
string subject = 3;
string body = 4;
int64 timestamp = 5;
}
Protocol buffers require you to use one of several standard types of fields, the most basic being string, int32, int64, enum, etc. You can even create your own types and nest them, but for now we’ll keep it simple. Each one also requires a numbered tag which allows us to recognize which field is which during deserialization. This means we can’t change the type or tag of an field. You can remove fields, and add new ones, but you can’t change the numbering or type. Protocol Buffers strongly enforce idea of backwards compatibility. It is a principle part of why protocol buffers help you manage your models. Old messages should always be able to be parsed by new code. You can always ignore new fields or old fields in your code, but your services should not read messages differently based on the existence or non-existence of certain fields.
Now that we’ve defined our message, we can use the protoc command to generate Go code. Now this is where some developers get turned off by protocol buffers: using them involves generating code. Yuck, right? Well, the protoc command only generates some basic models and interfaces, but not much more than that, so it seems like a small price to pay for a pretty cool way to tie your services together.
To generate the Go code, you’ll need to install the protoc command. You can do that by downloading the correct package from https://developers.google.com/protocol-buffers/docs/downloads and installing it in /usr/local/bin if you’re running Linux or macOS.
protoc –go_out=tools/src/cool/tool/writing-rpc-proto/email/ tools/src/cool/tool/writing-rpc-proto/email/email.proto
This command uses the email.proto file to create email.pb.go, which is where your model will live. You’ll notice it generates setters and getters automatically. Just to prove that we can initialize an Email message as a Go class instance, you can write a main.go that looks something like this:
package main
import (
"fmt"
"time"
pb "email"
)
func main() {
email := pb.Email{
To: "[email protected]",
From: "[email protected]",
Body: "they are super cool",
Subject: "protocol buffers",
Timestamp: int64(time.Now().Unix() * 1000),
}
fmt.Println("Email Message: " + email.String())
out, _ := proto.Marshal(&email)
fmt.Println(out)
}
And the out put will be:
to:"you" from:"me" subject:"Daily reminder" body:"Hey there, protobufs are cool!" timestamp:1479530886000
[10 3 121 111 117 18 2 109 101 26 14 68 97 105 108 121 32 114 101 109 105 110 100 101 114 34 30 72 101 121 32 116 104 101 114 101 44 32 112 114 111 116 111 98 117 102 115 32 97 114 101 32 99 111 111 108 33 40 240 230 166 215 135 43]
The first line is the string representation of the email, and the second is the encoded bytes of the message. Now that we’ve defined our message and we can initialize it, and marshal it to bytes to send over the wire, let’s create an RPC service definition using Protocol Buffers and GRPC.
Defining the service
Our email message will be exchanged between a client and a server, both of which will implement a service. To define a service let’s add the following to our email.proto file.
service EmailService {
rpc ReceiveEmail (Email) returns (EmailAck) {}
}
message EmailAck {
string status = 1;
}
Like the protocol buffer message definition, there’s a lot we could go into here, but the import part is that we’re defining a service, and an interface for that service. We’re also defining an acknowledgement message, just so we have something that the server can send back to the client to let it know that it has received the message.
Now when we compile the proto, we need to tell it that we’re not just creating messages, we’re creating services. So we add `plugins=grpc:` to the front of our `go-out` argument.
protoc -I=tools/src/cool/tool/writing-rpc-proto/email/ --go_out=plugins=grpc:tools/src/cool/tool/writing-rpc-proto/email/ tools/src/cool/tool/writing-rpc-proto/email/email.proto
This is telling protobuf what executable we’re going to be using for defining our RPC, in this case GRPC. This leaves open the possibility of using a different RPC library with protobufs. Using protobufs doesn’t marry you to using GRPC. It also allows you to wirte your own plugins. For example, if you wanted a different .String() function on your Go model, you could write a plugin that adds that to the generated .pb.go file.
If you scroll through email.pb.go now you’ll see that we have all the boilerplate code for implementing the “ReceiveEmail” service and binding it. Let’s do that now by creating a new package called “server” and giving it a main.go, and implementing an instance of the EmailService.
package main
import (
"log"
"net"
pb "cool/tool/email"
"golang.org/x/net/context"
"google.golang.org/grpc"
)
const (
port = ":50051"
)
// Used to implement email.EmailServer
type server struct{}
// ReceiveEmail implements email.EmailServer
func (s *server) ReceiveEmail(ctx context.Context, in *pb.Email) (*pb.EmailAck, error) {
log.Println(in.String())
return &pb.EmailAck{Status: "OK"}, nil
}
func main() {
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterEmailServiceServer(s, &server{})
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
First we create a server struct, which will implement our EmailServer, and a method on it to receive an Email, log it out, and respond with an EmailAck. We then can pass this interface to our server, and register it, and start serving. If you run this server though, nothing will happen, because we’re not receiving anything. So let’s create a simple client using the same generated go code in email.pb.go, and send a simple Email message.
package main
import (
"fmt"
"log"
"time"
pb "cool/tool/email"
"golang.org/x/net/context"
"google.golang.org/grpc"
"github.com/golang/protobuf/proto"
)
func main() {
fmt.Println("Starting client.")
email := pb.Email{
To: "you",
From: "me",
Body: "Hey there, protobufs are cool!",
Subject: "Daily reminder",
Timestamp: int64(time.Now().Unix() * 1000),
}
fmt.Println(email.String())
out, _ := proto.Marshal(&email)
fmt.Println(out)
send(email)
}
func send(email pb.Email) {
conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewEmailServiceClient(conn)
r, err := c.ReceiveEmail(context.Background(), &email)
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", r.Status)
}
Here we start up, initialize our message, log it out, and send it off with a client. We create the client by calling `pb.NewEmailServiceClient()`. This illustrates what’s really cool about using GRPC and Protocol Buffers; we were able to define a service in one place, define a message in once place, and implement our servers and clients using interfaces. If we ever want to alter the message we’re sending, we can add or remove fields at will inside the .proto file, and serialization/deserialization are taken care of.
Wrapping up
This is just an introduction, so I’ve not gone over the more advanced features of protobufs and GRPC, such as bi-directional streaming, authentication, and more complex service definitions. But you can probably guess what those look like and how useful they are. You can also probably guess that, since GRPC and Protocol Buffers are libraries, rather than standards like HTTP/REST, they’ll take some getting used to. By choosing the two over HTTP/REST/JSON you’re making a commitment to a library that new developers might be unfamiliar and uncomfortable with, and is more abstract than the latter. But if you’ve reached a point where the majority of your data can be streamed, or your client-server connections are not with a user or browser, or your HTTP overhead is causing performance degradation; the protobuf/grpc duo might be a more melodious choice than the HTTP/REST/JSON trio.
Github repo: https://github.com/vogtb/writing-grpc-proto
Header image generated using https://ondras.github.io/primitive.js.
This blog post was edited for clarity on Nov 29, 2016, and again on Aug 6, 2019.