Protobuf basics cheat-sheet

The basics  that you need to know about protobuf in a a single post

Why Protobuf?

1. Fully type safe.
2. Data is auto compressed which reduces cpu  and network bandwidth usage.
3. Provides backward and forward compatibility.
4. 3-10x smaller, 20-100x faster than xml.
5. Provides ability to  auto-generate client code in multiple languages, schema(.proto file) is used to generate  code and read the data.
6. RPC frameworks like gRPC uses Protocol Buffers by default, which provides  much better performance than JSON.
7. Easy to learn.

Disadvantages

1. Support for some languages  might be lacking(most mainstream languages are fine).
2. Cant open the serialised data with a text editor.
3. Every field in a Protobuf message is optional and has a default value, it is impossible to differentiate a field that is missing in a protocol buffer from one that was assigned with the default value. ex; Int is defaulted to Zero, strings and arrays are defaulted to empty!

What is a .Proto file?

Protocol buffers has it's own style guide and it is important to follow that, as the code generation depends on it.

Proto compiler converts the .proto file into a language specific implementation, you can find the data types supported by protocol buffers and it's language specific conversions here

Here is an example of generating java code from proto file, you can specify the language options that you like.

protoc -I=proto --java_out==java proto/simple_proto.proto

This command takes a .proto file named simple_proto.proto which is located in directory named proto and converts into SimpleProto.java and puts it in another directory called java, the generated java file is quite verbose, you can see it here

Working with Protobuf

Field tags/Field numbers:

1. Field tags uniquely identify a field in protobuf, it has to be an unique integer.
2. Smallest tag is 1, largest is 2^29-1.
3. Remember tag number 1-15 use 1 byte in space use them for frequently populated fields, tags from 16-2047 takes 2 bytes and so on.
4. You can't use the tags from 19,000-19,999 these are the tags reserved by google.
5. Field tags should never be changed once your message type is in use.
6. You cannot use any previously reserved field numbers.

Default values

If the value is not specified for any field in protobuf it always gets a default value

Advanced data types

oneof:

1. oneof fields are like regular fields except all the fields in a oneof share memory, and at most one field can be set at the same time. Setting any member of the oneof automatically clears all the other members
2. If the parser encounters multiple members of the same oneof on the wire, only the last member seen is used in the parsed message.

2. Map

3. Timestamp

Represents a point in time independent of any time zone or local calendar, contains seconds and nano seconds .

4. Duration

Represents timespan between two timestamps, contains seconds and nano seconds.

Rules for protobuf schema evolution

1. Never change the field tags for any existing fields.
2. If you add new fields all clients using old .protofile will just ignore the new field.
3. Like wise if a new code gets old data, default values will be used.
4. Fields can be removed but the removed tag number should never be reused, adding obsolete or reserved is the better way to deprecate the fields.
5. Renaming of fields is allowed, Protobuf uses key tags for serializing /      deserializing.
6. some compatible data type changes is allowed, use it with extreme precaution
7. Never remove any reserved tags.
8. when removing a field always reserve both field tag and field name,  reserve tags to prevent new fields from using it, reserve names to prevent code bugs.