Protocol Buffers & gRPC
From .proto files to production gRPC services in Go — the complete guide to high-performance RPC.
01 What is Protocol Buffers
.proto file, then use the protoc compiler to generate data-access classes in your target language (Go, Java, Python, C++, etc.).- Size: Protobuf encodes data in a compact binary format — typically 3-10x smaller than JSON.
- Speed: Parsing protobuf is 20-100x faster than parsing JSON because there’s no text parsing, no field-name lookups.
- Schema enforcement: The
.protofile IS the contract — if the message doesn’t match the schema, it fails at compile time, not at runtime. - Code generation: You get strongly-typed structs with getters, setters, and serialization methods for free.
- Backward/forward compatibility: You can evolve your schema without breaking old clients (if you follow the rules).
1:John|2:30|3:john@example.com. Same data, fraction of the size, and both sides know exactly what each number means.Imagine you and your friend want to pass notes in class. JSON is like writing full sentences: "The answer to question 1 is Paris, the answer to question 2 is 42." Protobuf is like using a secret code you both agreed on before class: "1:Paris, 2:42." Both carry the same info, but the coded version is way shorter and faster to read.
In programming, when two computers (servers) talk to each other, they need to agree on a format. JSON uses readable text like {"name": "Alice"}. Protobuf uses compact binary data that's invisible to humans but much faster for computers to read and write. You define the "secret codebook" in a .proto file, and a tool called protoc automatically generates code that both sides use.
JSON vs Protobuf — Size Comparison
The same user object in JSON vs Protobuf wire format:
// JSON — 74 bytes (human readable, lots of overhead)
{
"name": "John Doe",
"age": 30,
"email": "john@example.com",
"active": true
}
// Protobuf binary — ~32 bytes (not human readable, minimal overhead)
// Field 1 (string): "John Doe"
// Field 2 (varint): 30
// Field 3 (string): "john@example.com"
// Field 4 (varint): 1
// 0a 08 4a 6f 68 6e 20 44 6f 65 10 1e 1a 10 6a 6f ...
Use Protobuf: microservice-to-microservice communication, high-throughput systems, mobile apps (bandwidth matters), gRPC services, data storage where schema evolution matters.
Use JSON: browser-facing REST APIs (browsers speak JSON natively), config files, debugging/logging (human readable), quick prototypes, public APIs where simplicity matters.
02 Proto File Syntax
A .proto file is like a blueprint or contract. Just like an architect draws a blueprint before building a house, you write a .proto file before writing any code. It says:
- "Here's what our data looks like" — a User has a name (text), age (number), and email (text)
- "Here's what operations we support" — you can CreateUser, GetUser, DeleteUser
Then a tool called protoc reads this blueprint and automatically writes Go code (structs, functions, interfaces) for you. You never write serialization code by hand.
Every .proto file follows a specific structure. Let's break down every element:
// user.proto
syntax = "proto3"; // MUST be first non-comment line
package user.v1; // prevents naming conflicts between projects
option go_package = "github.com/yourorg/project/gen/user/v1;userv1";
import "google/protobuf/timestamp.proto"; // import well-known types
// A message is like a struct in Go or a class in Java
message User {
string id = 1; // field_number = 1 (NOT the value, it's the tag)
string name = 2;
string email = 3;
int32 age = 4;
bool active = 5;
google.protobuf.Timestamp created_at = 6;
}
// A service defines RPC methods
service UserService {
rpc GetUser(GetUserRequest) returns (GetUserResponse);
rpc ListUsers(ListUsersRequest) returns (stream User);
}
message GetUserRequest {
string id = 1;
}
message GetUserResponse {
User user = 1;
}
A Yes! proto2 was the original version. proto3 (released 2016) simplified the language significantly:
- proto2: fields are
required,optional, orrepeated. Required fields turned out to be a bad idea (breaks backward compat). - proto3: all fields are optional by default. No
requiredkeyword. Addedmaptype, JSON mapping,Anytype. - proto3 optional: Since protobuf 3.15+, you can use
optionalkeyword in proto3 to get "field presence" (distinguish between "not set" and "set to default").
Always use proto3 for new projects. proto2 is legacy.
A go_package tells protoc where to put the generated Go code. The format is "import_path;package_name":
github.com/yourorg/project/gen/user/v1— the full Go import pathuserv1— the Go package name (what you use inimportstatements)
Without the semicolon part, protoc uses the last path segment as the package name (which would be v1 — too generic).
Naming Conventions
| Element | Convention | Example |
|---|---|---|
| File names | lowercase_snake.proto | user_service.proto |
| Message names | PascalCase | UserProfile |
| Field names | snake_case | first_name |
| Enum names | PascalCase | UserStatus |
| Enum values | UPPER_SNAKE_CASE | USER_STATUS_ACTIVE |
| Service names | PascalCase + "Service" | UserService |
| RPC methods | PascalCase verb | GetUser, CreateUser |
| Package | lowercase dot-separated + version | myorg.user.v1 |
03 Scalar Types
Scalar just means "simple, single value" — a number, a piece of text, or true/false. Think of them as the basic building blocks, like LEGO bricks. You combine these simple types to build complex messages (like combining bricks to build a house).
In JavaScript you have string, number, boolean. Protobuf is more specific — it has int32, int64, float, double, string, bool, etc. — because being specific about size means the computer can pack data tighter.
Protobuf has a fixed set of scalar (primitive) types. Each maps to a specific type in Go:
| Proto Type | Go Type | Default | Notes |
|---|---|---|---|
double | float64 | 0 | 64-bit IEEE 754 |
float | float32 | 0 | 32-bit IEEE 754 |
int32 | int32 | 0 | Variable-length. Inefficient for negative numbers (use sint32). |
int64 | int64 | 0 | Variable-length. Inefficient for negative numbers (use sint64). |
uint32 | uint32 | 0 | Variable-length unsigned |
uint64 | uint64 | 0 | Variable-length unsigned |
sint32 | int32 | 0 | ZigZag encoding — efficient for negative values |
sint64 | int64 | 0 | ZigZag encoding — efficient for negative values |
fixed32 | uint32 | 0 | Always 4 bytes. More efficient than uint32 when values > 2^28. |
fixed64 | uint64 | 0 | Always 8 bytes. More efficient than uint64 when values > 2^56. |
sfixed32 | int32 | 0 | Always 4 bytes, signed |
sfixed64 | int64 | 0 | Always 8 bytes, signed |
bool | bool | false | |
string | string | "" | Must be UTF-8 or 7-bit ASCII |
bytes | []byte | empty | Arbitrary byte sequence |
Regular int32/int64 use varint encoding which treats the number as unsigned internally. Negative numbers always take 10 bytes (the maximum). If your field frequently holds negative values (like temperature, offsets, deltas), use sint32/sint64 which uses ZigZag encoding — it maps -1 to 1, 1 to 2, -2 to 3, etc., keeping small negatives small on the wire.
- IDs, counts, sizes (always positive) →
uint32/uint64 - Ages, quantities (small positives) →
int32 - Temperatures, offsets (can be negative) →
sint32 - Unix timestamps, big IDs →
int64orfixed64 - Hash values, UUIDs as numbers →
fixed64(always 8 bytes, no varint overhead)
04 Messages & Enums
Message = a template for data. Like a form you fill out. A "User" message says: every user has a name, email, and age. In Go, this becomes a struct. In JavaScript, it's like the shape of an object {name: "", email: "", age: 0}.
Enum = a fixed list of choices. Like a dropdown menu. A "UserStatus" enum says: a user can be ACTIVE, INACTIVE, or BANNED — and nothing else. It prevents someone from accidentally setting status to "banana."
Repeated = a list/array. repeated string tags means "a list of strings" — like ["go", "backend", "grpc"].
Messages
A message is the protobuf equivalent of a Go struct. Messages can be nested, reference each other, and contain any combination of fields.
message Address {
string street = 1;
string city = 2;
string state = 3;
string zip_code = 4;
string country = 5;
}
message User {
string id = 1;
string name = 2;
string email = 3;
Address address = 4; // nested message reference
repeated string tags = 5; // repeated = slice/array
// Nested message — only accessible as User.PhoneNumber
message PhoneNumber {
string number = 1;
PhoneType type = 2;
}
repeated PhoneNumber phones = 6;
}
Generated Go code:
// Auto-generated — DO NOT EDIT
type User struct {
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
Email string `protobuf:"bytes,3,opt,name=email,proto3" json:"email,omitempty"`
Address *Address `protobuf:"bytes,4,opt,name=address,proto3" json:"address,omitempty"`
Tags []string `protobuf:"bytes,5,rep,name=tags,proto3" json:"tags,omitempty"`
Phones []*User_PhoneNumber `protobuf:"bytes,6,rep,name=phones,proto3" json:"phones,omitempty"`
}
// Getters are generated — safe to call on nil receivers
func (x *User) GetName() string {
if x != nil {
return x.Name
}
return ""
}
func (x *User) GetAddress() *Address {
if x != nil {
return x.Address
}
return nil
}
Enums
Enums define a fixed set of constants. The first value MUST be 0 and serves as the default.
enum PhoneType {
PHONE_TYPE_UNSPECIFIED = 0; // MUST have 0 value — it's the default
PHONE_TYPE_MOBILE = 1;
PHONE_TYPE_HOME = 2;
PHONE_TYPE_WORK = 3;
}
enum UserStatus {
USER_STATUS_UNSPECIFIED = 0;
USER_STATUS_ACTIVE = 1;
USER_STATUS_INACTIVE = 2;
USER_STATUS_BANNED = 3;
}
message User {
string id = 1;
string name = 2;
UserStatus status = 3;
}
In proto3, the default value for an enum field is 0. If you don't set a status field, it will be USER_STATUS_UNSPECIFIED. This lets you distinguish between "the client explicitly set ACTIVE" vs "the client didn't set anything." Always name your 0 value *_UNSPECIFIED and handle it as "unknown/not set" in your business logic.
Generated Go enum code:
type UserStatus int32
const (
UserStatus_USER_STATUS_UNSPECIFIED UserStatus = 0
UserStatus_USER_STATUS_ACTIVE UserStatus = 1
UserStatus_USER_STATUS_INACTIVE UserStatus = 2
UserStatus_USER_STATUS_BANNED UserStatus = 3
)
// Usage in Go:
user := &pb.User{
Id: "u-123",
Name: "Alice",
Status: pb.UserStatus_USER_STATUS_ACTIVE,
}
if user.Status == pb.UserStatus_USER_STATUS_BANNED {
// handle banned user
}
Enum Aliases
You can give multiple enum values the same number with allow_alias:
enum Priority {
option allow_alias = true;
PRIORITY_UNSPECIFIED = 0;
PRIORITY_LOW = 1;
PRIORITY_P3 = 1; // alias for LOW
PRIORITY_MEDIUM = 2;
PRIORITY_P2 = 2; // alias for MEDIUM
PRIORITY_HIGH = 3;
PRIORITY_P1 = 3; // alias for HIGH
}
05 Oneof, Maps & Any
- Oneof = "pick exactly one." Like a food order: you get a burger OR pizza OR salad — not all three. If you set the pizza field, the burger field gets automatically cleared. In code, this is like a TypeScript union type:
target: string | number | boolean. - Map = a dictionary / key-value store. Like a phone book: name → phone number. In Go it's
map[string]string, in JS it's a plain object{"key": "value"}. - Any = a wildcard. "I don't know what type of data this will be, but I'll figure it out later." Like a box labeled "surprise" — you pack something in, and the receiver unpacks it and checks what's inside. Use sparingly.
Oneof — Mutually Exclusive Fields
oneof means "exactly one of these fields can be set at a time." Setting one automatically clears the others. Think TypeScript union types or Go interfaces.
message NotificationTarget {
string notification_id = 1;
oneof target {
string email = 2; // send via email
string phone = 3; // send via SMS
string slack_id = 4; // send via Slack
string webhook_url = 5; // send via webhook
}
}
message PaymentMethod {
oneof method {
CreditCard credit_card = 1;
BankTransfer bank_transfer = 2;
CryptoWallet crypto_wallet = 3;
}
}
message CreditCard {
string number = 1;
string expiry = 2;
string cvv = 3;
}
Generated Go uses an interface pattern:
type NotificationTarget struct {
NotificationId string
// oneof target
Target isNotificationTarget_Target // interface
}
// Each option implements the interface
type NotificationTarget_Email struct { Email string }
type NotificationTarget_Phone struct { Phone string }
type NotificationTarget_SlackId struct { SlackId string }
// Usage — type switch to determine which is set
switch t := notification.Target.(type) {
case *pb.NotificationTarget_Email:
sendEmail(t.Email)
case *pb.NotificationTarget_Phone:
sendSMS(t.Phone)
case *pb.NotificationTarget_SlackId:
sendSlack(t.SlackId)
case nil:
// not set
}
Maps
map fields are key-value pairs, equivalent to Go's map[K]V:
message Project {
string id = 1;
string name = 2;
// map<key_type, value_type>
map<string, string> labels = 3; // map[string]string
map<string, int32> priorities = 4; // map[string]int32
map<int32, User> members = 5; // map[int32]*User
}
// Key type MUST be integral or string — NOT float, bytes, or message
// Value type can be anything EXCEPT another map
- Map keys can only be:
int32,int64,uint32,uint64,sint32,sint64,fixed32,fixed64,sfixed32,sfixed64,bool,string - Map keys CANNOT be:
float,double,bytes,enum, ormessage - Map fields CANNOT be
repeated - Map order is NOT guaranteed on the wire
google.protobuf.Any — Dynamic Typing
Any holds an arbitrary serialized message along with a URL that describes its type. It's protobuf's escape hatch for dynamic typing.
import "google/protobuf/any.proto";
message Event {
string id = 1;
string type = 2;
google.protobuf.Any payload = 3; // can hold ANY message
google.protobuf.Timestamp time = 4;
}
// Go usage — packing and unpacking Any
import "google.golang.org/protobuf/types/known/anypb"
// Pack a User into Any
user := &pb.User{Id: "u-1", Name: "Alice"}
anyPayload, err := anypb.New(user)
event := &pb.Event{
Id: "evt-1",
Type: "user.created",
Payload: anyPayload,
}
// Unpack Any back to a specific type
var unpacked pb.User
if err := event.Payload.UnmarshalTo(&unpacked); err != nil {
log.Fatal(err)
}
fmt.Println(unpacked.Name) // "Alice"
06 Imports & Packages
As your project grows, you don't want everything in one giant file. Imports let you split your .proto definitions across multiple files and reuse types — just like import in JavaScript or Python.
Packages prevent name collisions. If Team A and Team B both create a User message, packages keep them separate: team_a.v1.User vs team_b.v1.User. The v1 part is the version — so when you need to make breaking changes, you create v2 and old clients keep working.
Imports
Use import to reference types defined in other .proto files:
// common/types.proto
syntax = "proto3";
package common.v1;
option go_package = "github.com/org/project/gen/common/v1;commonv1";
message Pagination {
int32 page = 1;
int32 page_size = 2;
string cursor = 3;
}
message ErrorDetail {
string code = 1;
string message = 2;
map<string, string> metadata = 3;
}
// user/v1/user.proto
syntax = "proto3";
package user.v1;
option go_package = "github.com/org/project/gen/user/v1;userv1";
import "common/types.proto"; // import the common types
message ListUsersRequest {
common.v1.Pagination pagination = 1; // use fully qualified name
string filter = 2;
}
message ListUsersResponse {
repeated User users = 1;
int32 total_count = 2;
}
Public vs Weak Imports
// Regular import — only this file can use the imported types
import "common/types.proto";
// Public import — anyone who imports THIS file also gets common/types.proto
// Useful when re-exporting / moving types between files
import public "common/types.proto";
Package Versioning Strategy
Always include a version in your package name (user.v1, not just user). When you need to make breaking changes, create a v2 package. Old clients keep using v1, new clients use v2. Both can coexist.
07 Well-Known Types
Imagine every team reinventing how to represent "a point in time" or "a time duration." One team uses int64 seconds_since_epoch, another uses string iso_date, a third uses two fields seconds + nanos. Chaos!
Google said: "Here are standard, pre-built types everyone should use." These are the Well-Known Types. Timestamp for dates, Duration for time spans, FieldMask for partial updates, Empty for "no data" responses. They come built into protobuf — just import and use them.
Google provides a set of well-known types (WKTs) — standard protobuf messages for common patterns. Always use these instead of inventing your own.
| Type | Import | Use For | Go Type |
|---|---|---|---|
Timestamp | google/protobuf/timestamp.proto | Points in time (created_at, updated_at) | *timestamppb.Timestamp |
Duration | google/protobuf/duration.proto | Time spans (timeout, ttl) | *durationpb.Duration |
Struct | google/protobuf/struct.proto | Arbitrary JSON-like data | *structpb.Struct |
Value | google/protobuf/struct.proto | A single dynamic value | *structpb.Value |
Any | google/protobuf/any.proto | Arbitrary serialized message | *anypb.Any |
Empty | google/protobuf/empty.proto | RPCs with no request/response body | *emptypb.Empty |
FieldMask | google/protobuf/field_mask.proto | Partial updates (PATCH semantics) | *fieldmaskpb.FieldMask |
StringValue | google/protobuf/wrappers.proto | Nullable string (distinguish "" from unset) | *wrapperspb.StringValue |
Int32Value | google/protobuf/wrappers.proto | Nullable int32 | *wrapperspb.Int32Value |
BoolValue | google/protobuf/wrappers.proto | Nullable bool | *wrapperspb.BoolValue |
Timestamp & Duration
import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";
message Session {
string id = 1;
google.protobuf.Timestamp created_at = 2;
google.protobuf.Timestamp expires_at = 3;
google.protobuf.Duration ttl = 4;
}
// Go usage
import (
"time"
"google.golang.org/protobuf/types/known/timestamppb"
"google.golang.org/protobuf/types/known/durationpb"
)
session := &pb.Session{
Id: "sess-1",
CreatedAt: timestamppb.Now(), // time.Now() → Timestamp
ExpiresAt: timestamppb.New(time.Now().Add(24 * time.Hour)),
Ttl: durationpb.New(24 * time.Hour), // time.Duration → Duration
}
// Convert back to Go types
goTime := session.CreatedAt.AsTime() // → time.Time
goDuration := session.Ttl.AsDuration() // → time.Duration
FieldMask — Partial Updates
import "google/protobuf/field_mask.proto";
message UpdateUserRequest {
User user = 1;
google.protobuf.FieldMask update_mask = 2;
}
// Client sends: update_mask.paths = ["name", "email"]
// → Server only updates name and email, ignores all other fields
// This is how Google APIs implement PATCH semantics
// Go server-side handling
func (s *server) UpdateUser(ctx context.Context, req *pb.UpdateUserRequest) (*pb.User, error) {
existing, err := s.db.GetUser(ctx, req.User.Id)
if err != nil {
return nil, err
}
// Apply only the fields specified in the mask
for _, path := range req.UpdateMask.GetPaths() {
switch path {
case "name":
existing.Name = req.User.Name
case "email":
existing.Email = req.User.Email
case "address.city":
existing.Address.City = req.User.Address.City
}
}
return s.db.SaveUser(ctx, existing)
}
Wrapper Types — Nullable Scalars
import "google/protobuf/wrappers.proto";
message SearchFilter {
string query = 1;
google.protobuf.Int32Value min_age = 2; // nil = no filter, 0 = filter for age 0
google.protobuf.Int32Value max_age = 3;
google.protobuf.BoolValue is_active = 4; // nil = no filter, false = filter for inactive
google.protobuf.StringValue country = 5; // nil = no filter, "" = filter for empty country
}
08 Field Numbers & Wire Format
= 1, = 2, etc. after each field are field numbers (tags), NOT default values. They are the unique identifier for each field in the binary encoding. Once assigned, a field number should never be changed — it's part of the wire format contract."email" (5 bytes + quotes + colon) becomes just a 1-byte tag.This confuses everyone at first! The = 1 after a field is NOT the default value. It's a tag number — like a barcode for that field.
When protobuf sends data over the network, it doesn't send field names like JSON does (no "name":, "email":). Instead, it sends the tag number: "field #1 = John, field #2 = john@example.com". Both sides agreed (via the .proto file) that #1 means name and #2 means email.
This is why you must never change or reuse a field number — old clients still think #4 means "phone", so if you repurpose #4 to mean "address", data gets corrupted.
Wire Types
| Wire Type | ID | Used For |
|---|---|---|
| Varint | 0 | int32, int64, uint32, uint64, sint32, sint64, bool, enum |
| 64-bit | 1 | fixed64, sfixed64, double |
| Length-delimited | 2 | string, bytes, embedded messages, packed repeated fields |
| 32-bit | 5 | fixed32, sfixed32, float |
Field Number Ranges
message Example {
// 1-15: Tag encoded in 1 byte → USE THESE for frequently-set fields
string id = 1; // 1 byte tag
string name = 2; // 1 byte tag
// 16-2047: Tag encoded in 2 bytes
string description = 16; // 2 byte tag
// 2048-262143: Tag encoded in 3 bytes
// ... and so on
// 19000-19999: RESERVED by protobuf implementation — DO NOT USE
// Max field number: 536,870,911 (2^29 - 1)
}
Field numbers 1-15 use only 1 byte for the tag on the wire. Fields 16-2047 use 2 bytes. Put your most frequently used fields in the 1-15 range. This seems tiny, but in high-throughput systems serializing millions of messages per second, it adds up.
Reserved Fields
When you remove a field, reserve its number and name to prevent future developers from accidentally reusing them (which would break backward compatibility):
message User {
reserved 4, 7, 9 to 11; // reserve field numbers
reserved "old_field", "legacy_name"; // reserve field names
string id = 1;
string name = 2;
string email = 3;
// field 4 was 'phone' — removed in v2
int32 age = 5;
}
09 Backward & Forward Compatibility
In a real company, you might have 50 microservices all talking to each other. When you update your User message on one service, you can't update all 50 services at the same time. Some will still be running the old version for hours, days, or even weeks.
Backward compatible = your new service can read messages from old services (old format still works).
Forward compatible = your old services can read messages from your new service (they just ignore unknown fields).
Protobuf makes this easy IF you follow the rules below. Break the rules, and suddenly half your services crash after a deploy.
Safe Changes (DO)
- Add new fields with new field numbers
- Remove fields (but reserve their numbers and names)
- Rename fields (wire format only uses numbers, not names)
- Add new enum values
- Add new RPC methods to a service
- Change
int32toint64(oruint32touint64) — widening is safe - Change
stringtobytes(if the string was valid UTF-8)
Breaking Changes (DON'T)
- Change a field number — old clients will read the wrong data
- Change a field's type to an incompatible type (e.g.,
stringtoint32) - Reuse a field number that was previously used (even if the old field was removed)
- Change the name of a package (affects generated code imports)
- Remove an enum value — old clients may still send it
- Change a
repeatedfield to a scalar or vice versa - Change
oneofto regular fields or vice versa
Never change a field number. Never reuse a deleted field number. Always use reserved to prevent accidental reuse. If you need a fundamentally different structure, create a v2 package.
10 Protoc & Code Generation
protoc is the Protocol Buffer Compiler. It's a command-line tool that reads your .proto files and spits out code in your target language. Think of it like a translator:
- You write the blueprint (
.protofile) in protobuf language protoctranslates it into Go code (.pb.gofiles)- The generated Go code has structs, marshal/unmarshal functions, and gRPC interfaces — all ready to use
You also need plugins — protoc-gen-go for message structs and protoc-gen-go-grpc for gRPC server/client code. Think of plugins as language-specific translators that plug into protoc.
Installing the Toolchain
# 1. Install the protoc compiler
# macOS
brew install protobuf
# Linux (Ubuntu/Debian)
apt install -y protobuf-compiler
# Verify
protoc --version # libprotoc 27.x
# 2. Install Go plugins for protoc
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
# Make sure $GOPATH/bin is in your PATH
export PATH="$PATH:$(go env GOPATH)/bin"
Running protoc
# Generate Go code from a single .proto file
protoc \
--go_out=. \
--go_opt=paths=source_relative \
--go-grpc_out=. \
--go-grpc_opt=paths=source_relative \
proto/user/v1/user.proto
# What each flag means:
# --go_out=. → generate message structs, put output relative to current dir
# --go_opt=paths=source_relative → output file path mirrors the .proto file path
# --go-grpc_out=. → generate gRPC service stubs
# --go-grpc_opt=paths=source_relative → same path strategy for gRPC files
# Generate all .proto files in a directory tree
protoc \
--go_out=. --go_opt=paths=source_relative \
--go-grpc_out=. --go-grpc_opt=paths=source_relative \
--proto_path=proto \
proto/**/*.proto
Output Files
For each .proto file, protoc generates two Go files:
Buf — Modern Protobuf Tooling
buf is the modern alternative to raw protoc. It handles linting, breaking change detection, code generation, and dependency management.
# Install buf
brew install bufbuild/buf/buf
# Initialize a buf project
buf mod init
# buf.yaml — module configuration
version: v2
modules:
- path: proto
name: buf.build/yourorg/project
lint:
use:
- STANDARD
breaking:
use:
- FILE
# buf.gen.yaml — code generation config
version: v2
plugins:
- remote: buf.build/protocolbuffers/go
out: gen
opt: paths=source_relative
- remote: buf.build/grpc/go
out: gen
opt: paths=source_relative
# Generate code
buf generate
# Lint your proto files
buf lint
# Check for breaking changes against the main branch
buf breaking --against '.git#branch=main'
Raw protoc is fine for learning, but buf is what you should use in production. It handles proto dependencies (like go.mod for protos), has built-in linting rules that enforce Google's API design guide, and can detect breaking changes in your CI pipeline.
11 What is gRPC
- You define your service and messages in a
.protofile protocgenerates a server interface (you implement) and a client stub (you call)- The client calls methods on the stub as if they were local functions
- Under the hood: the request is serialized to protobuf binary, sent over HTTP/2, deserialized on the server, your handler runs, and the response goes back the same way
.proto contract), and when you press button 3, you know exactly what happens. Faster, type-safe, and no ambiguity about the "message format."You already know REST APIs — you send a GET request to /users/123 and get back JSON. gRPC is a different way for programs to talk to each other.
Instead of URLs + JSON + HTTP verbs, gRPC uses:
- Function calls — calling
client.GetUser(req)feels like calling a local function, but it runs on another server - Protobuf — binary data instead of JSON (smaller, faster)
- HTTP/2 — the newer, faster version of HTTP that supports multiplexing (many requests on one connection)
- A strict contract — the
.protofile IS the API documentation, and code is auto-generated from it
Think of REST as ordering food at a restaurant (you read the menu, say your order in English, get food back). gRPC is like a vending machine with numbered buttons — press button #3, you get exactly what's labeled. Faster, no ambiguity, but less human-friendly.
12 gRPC vs REST
| Aspect | gRPC | REST (JSON over HTTP) |
|---|---|---|
| Protocol | HTTP/2 (binary framing) | HTTP/1.1 or HTTP/2 (text) |
| Serialization | Protobuf (binary, compact) | JSON (text, verbose) |
| Contract | .proto file (strict) | OpenAPI/Swagger (optional) |
| Code generation | Built-in, first-class | Optional (swagger-codegen) |
| Streaming | Native (4 types) | SSE / WebSocket (bolted on) |
| Browser support | Needs grpc-web proxy | Native |
| Human readability | Binary (not readable) | JSON (readable) |
| Performance | ~7-10x faster serialization | Slower, larger payloads |
| Latency | Lower (HTTP/2 multiplexing) | Higher (HTTP/1.1 head-of-line) |
| Deadlines | Built-in, propagated | Manual (timeout headers) |
| Load balancing | Client-side or L7 proxy | Any standard LB |
| Error handling | Rich status codes + details | HTTP status codes |
| Best for | Microservices, internal APIs | Public APIs, browser clients |
Use gRPC: service-to-service communication within your infrastructure, real-time streaming (live data feeds, chat), high-throughput systems (>10K RPS), polyglot environments (auto-generate clients for Go, Python, Java, etc.).
Use REST: public-facing APIs, browser clients (without proxy), simple CRUD, when human debugging matters, when your team knows REST well and gRPC adds complexity.
Use both: gRPC internally + gRPC-Gateway to auto-generate a REST proxy for external consumers. Best of both worlds.
13 Service Definition
A service definition is like a menu at a restaurant. It lists all the things the server can do:
- "You can ask me to CreateProduct — give me a name and price, I'll give you back the product"
- "You can ask me to GetProduct — give me an ID, I'll give you back the product"
- "You can ask me to WatchProducts — give me some IDs, I'll keep streaming updates to you"
Each menu item (RPC method) says: "You give me THIS (request message), and I give you back THAT (response message)." The stream keyword means "I'll keep sending/receiving data continuously" instead of one-and-done.
A service in a .proto file defines the RPC methods that a server implements and a client can call.
syntax = "proto3";
package product.v1;
option go_package = "github.com/org/shop/gen/product/v1;productv1";
import "google/protobuf/timestamp.proto";
import "google/protobuf/empty.proto";
import "google/protobuf/field_mask.proto";
// ============ Messages ============
message Product {
string id = 1;
string name = 2;
string description = 3;
int64 price_cents = 4; // store money as cents to avoid float issues
string currency = 5;
repeated string categories = 6;
ProductStatus status = 7;
google.protobuf.Timestamp created_at = 8;
google.protobuf.Timestamp updated_at = 9;
}
enum ProductStatus {
PRODUCT_STATUS_UNSPECIFIED = 0;
PRODUCT_STATUS_DRAFT = 1;
PRODUCT_STATUS_ACTIVE = 2;
PRODUCT_STATUS_ARCHIVED = 3;
}
// ============ Request/Response Messages ============
message CreateProductRequest {
string name = 1;
string description = 2;
int64 price_cents = 3;
string currency = 4;
repeated string categories = 5;
}
message GetProductRequest {
string id = 1;
}
message UpdateProductRequest {
Product product = 1;
google.protobuf.FieldMask update_mask = 2;
}
message DeleteProductRequest {
string id = 1;
}
message ListProductsRequest {
int32 page_size = 1;
string page_token = 2; // cursor-based pagination
string filter = 3; // e.g., "status=ACTIVE AND price_cents>1000"
string order_by = 4; // e.g., "created_at desc"
}
message ListProductsResponse {
repeated Product products = 1;
string next_page_token = 2;
int32 total_count = 3;
}
message WatchProductsRequest {
repeated string product_ids = 1;
}
// ============ Service ============
service ProductService {
// Unary — one request, one response
rpc CreateProduct(CreateProductRequest) returns (Product);
rpc GetProduct(GetProductRequest) returns (Product);
rpc UpdateProduct(UpdateProductRequest) returns (Product);
rpc DeleteProduct(DeleteProductRequest) returns (google.protobuf.Empty);
// Server streaming — one request, stream of responses
rpc ListProducts(ListProductsRequest) returns (ListProductsResponse);
// Server streaming — watch for real-time updates
rpc WatchProducts(WatchProductsRequest) returns (stream Product);
// Client streaming — stream of requests, one response
rpc BulkCreateProducts(stream CreateProductRequest) returns (ListProductsResponse);
// Bidirectional streaming — stream both ways
rpc SyncProducts(stream Product) returns (stream Product);
}
14 The 4 RPC Types
rpc GetUser(GetUserRequest) returns (User);rpc WatchOrders(WatchRequest) returns (stream Order);rpc UploadFile(stream FileChunk) returns (UploadStatus);rpc Chat(stream ChatMessage) returns (stream ChatMessage);In Node.js, you'd use @grpc/grpc-js + @grpc/proto-loader or the nice-grpc library (more modern, TypeScript-first). The same 4 RPC types exist. The biggest difference: Node uses callbacks/async generators for streaming, while Go uses explicit Send()/Recv() loops.
// Node.js gRPC server (using nice-grpc)
import { createServer } from 'nice-grpc';
import { ProductServiceDefinition } from './gen/product';
const server = createServer();
server.add(ProductServiceDefinition, {
async getProduct(request) {
return { id: request.id, name: 'Widget', priceCents: 999 };
},
async *watchProducts(request) { // server streaming = async generator
while (true) {
yield { id: '1', name: 'Updated Widget' };
await sleep(1000);
}
}
});
await server.listen('0.0.0.0:50051');
Think of gRPC like a walkie-talkie conversation:
- Unary = "Hey, what's the weather?" ... "It's sunny." (one question, one answer)
- Server streaming = "Give me the news." ... "Breaking news #1... #2... #3... done." (one question, many answers dripping in)
- Client streaming = "Here's photo 1... photo 2... photo 3... all done!" ... "OK, uploaded 3 photos." (many inputs, one summary back)
- Bidirectional = "Hi!" "Hey!" "How are you?" "Good, you?" ... (both sides talk freely, like a real conversation)
15 Project Setup
Directory Structure
Go Module Setup
# Initialize the module
mkdir grpc-demo && cd grpc-demo
go mod init github.com/yourorg/grpc-demo
# Install gRPC Go dependencies
go get google.golang.org/grpc
go get google.golang.org/protobuf
The Proto File
// proto/user/v1/user.proto
syntax = "proto3";
package user.v1;
option go_package = "github.com/yourorg/grpc-demo/gen/user/v1;userv1";
import "google/protobuf/timestamp.proto";
import "google/protobuf/empty.proto";
message User {
string id = 1;
string name = 2;
string email = 3;
int32 age = 4;
google.protobuf.Timestamp created_at = 5;
}
message CreateUserRequest {
string name = 1;
string email = 2;
int32 age = 3;
}
message GetUserRequest {
string id = 1;
}
message ListUsersRequest {
int32 page_size = 1;
}
message ListUsersResponse {
repeated User users = 1;
}
message DeleteUserRequest {
string id = 1;
}
service UserService {
rpc CreateUser(CreateUserRequest) returns (User);
rpc GetUser(GetUserRequest) returns (User);
rpc ListUsers(ListUsersRequest) returns (stream User);
rpc DeleteUser(DeleteUserRequest) returns (google.protobuf.Empty);
}
Makefile for Code Generation
A Makefile is a file (literally named Makefile, no extension) that defines shortcuts for shell commands. You run make <target> in the terminal and it executes the recipe for that target.
The protoc command for generating Go code is long, fiddly, and easy to mistype. Every developer on your team would otherwise have to remember the exact flags, plugin names, paths, and the order of arguments. Instead, we put it in a Makefile once, and from then on everybody just types:
make proto— regenerate all the.pb.goand_grpc.pb.gofiles from your.protosourcesmake clean— wipe the generatedgen/folder so the nextmake protostarts fresh
It's the Go-ecosystem equivalent of an npm run script — a tiny, standardized entry point so the build is reproducible on every laptop and on CI.
# Makefile
.PHONY: proto clean
proto:
protoc \
--go_out=gen --go_opt=paths=source_relative \
--go-grpc_out=gen --go-grpc_opt=paths=source_relative \
--proto_path=proto \
proto/user/v1/user.proto
clean:
rm -rf gen/
make is famously strict — the line under each target (the recipe) must start with a literal TAB character. If your editor inserts spaces instead, you'll get the cryptic error Makefile:4: *** missing separator. Stop.
Most editors (VS Code, GoLand) auto-detect Makefiles and switch to tabs. If you copy-paste from a web page, double-check by running cat -A Makefile — tabs show as ^I.
Line-by-line breakdown
# MakefileA regular shell-style comment. # through end of line is ignored by make. Useful for documenting what the file does or labeling sections. Not a directive — the actual file just needs to be named Makefile (capital M) in your project root..PHONY: proto cleanDeclares proto and clean as phony targets — targets that aren't filenames. Without this, if a file named proto happened to exist in the directory, make would see it, decide "the target is already up to date," and refuse to run the recipe. .PHONY tells make "always run this recipe, never check for a file with this name." This is the single most-overlooked thing about Makefiles — always declare phony targets.proto:Defines a target named proto. The : is required. After this colon you can list prerequisites (other targets/files that must be built first) — we have none here, so it's just bare. The indented lines that follow are the recipe (the shell commands to run).protoc \protoc is the Protocol Buffer Compiler — the tool that reads .proto files and emits language-specific code. The trailing \ is a line continuation: it tells the shell "this command isn't done yet, the next line is part of it." It's purely cosmetic — you could write the whole thing on one giant line, but breaking it across lines makes flags readable.--go_out=genTells protoc to invoke the protoc-gen-go plugin and write its output into the gen/ directory. The plugin generates message structs — the Go types for your User, CreateUserRequest, etc., plus their Marshal/Unmarshal methods. The pattern --<name>_out is how protoc discovers plugins: it looks for an executable called protoc-gen-<name> on your PATH.--go_opt=paths=source_relativeA flag passed to the protoc-gen-go plugin (not to protoc itself). paths=source_relative means the output file path mirrors the input file path. So proto/user/v1/user.proto produces gen/user/v1/user.pb.go — the directory structure under gen/ matches the structure under proto/. The alternative is paths=import, which uses the go_package option from the proto file to derive the path — that nests output deeper under gen/github.com/yourorg/..., which is rarely what you want.--go-grpc_out=genSame idea, different plugin. Invokes protoc-gen-go-grpc, which generates the gRPC service stubs — the UserServiceServer interface you implement on the server, the UserServiceClient stub you call on the client, and the registration glue. Without this flag, you'd only get the message types and would have no way to actually do RPCs.--go-grpc_opt=paths=source_relativeSame path-strategy flag, but for the gRPC plugin. You want it to match --go_opt so that user.pb.go and user_grpc.pb.go end up in the same directory (they need to share a Go package — the gRPC file references the message types from the .pb.go file).--proto_path=protoDefines the import root for proto files — equivalent to a Go module root or a Java classpath. When one .proto file does import "user/v1/user.proto", protoc resolves that path relative to --proto_path. It also affects how the input file's path is interpreted: the file proto/user/v1/user.proto is seen as user/v1/user.proto (with proto/ stripped), which combined with paths=source_relative produces gen/user/v1/user.pb.go. Short alias: -I proto (mirrors GCC's include flag).proto/user/v1/user.protoThe input file — the actual .proto source you want to compile. You can list multiple files separated by spaces, or use a shell glob like proto/**/*.proto if your shell supports it. For larger projects you'd typically loop over a list or switch to buf generate, which discovers files automatically.clean:A second target. make clean is a near-universal convention — almost every Makefile in the world has a clean target that wipes build artifacts so you can rebuild from scratch.rm -rf gen/Recursively delete the entire gen/ directory. -r recurses into subdirectories, -f suppresses errors if the directory is missing (so the target works even on a fresh checkout). After this, make proto will recreate gen/ from scratch — useful when you've renamed files in proto/ and want to drop stale generated files that no longer correspond to any source.gen/ directory?Putting generated code in gen/ (rather than alongside the .proto files) gives you three nice properties:
- One-shot cleanup —
rm -rf genwipes everything generated, no risk of accidentally deleting a hand-written file. - Clear mental model — anything inside
gen/is "do not edit, regenerate." Everything outside is hand-written. - Tooling can ignore it — you can exclude
gen/from linters, code review heuristics, or coverage reports without complex glob rules.
The convention is to commit gen/ to git anyway — so a fresh clone can go build without needing protoc installed. CI runs make proto and fails the build if the diff is non-empty (i.e. someone forgot to regenerate).
A Each line of a Makefile recipe is a separate shell invocation. If you wrote --go_out=gen on its own line without a backslash, make would try to run --go_out=gen as a standalone command and fail with "command not found." The trailing \ joins lines into one logical command before the shell sees it. Alternative: put everything on one long line — works identically, just harder to read.
A Two reasons:
- Resolving
importstatements inside proto files. Ifuser.protodoesimport "common/types.proto",protoclooks for that file relative to--proto_path. Without setting it, imports break. - Computing output paths under
paths=source_relative. The output mirror only strips the--proto_pathprefix — not the current working directory. Without--proto_path=proto, the input file's full pathproto/user/v1/user.protowould land atgen/proto/user/v1/user.pb.go(with an extraproto/nested insidegen/), which is almost certainly not what you want.
A --go_out says where to write output (and implicitly says "use the protoc-gen-go plugin"). --go_opt passes options to that same plugin — configuration like paths=source_relative, module=..., etc. Same naming pattern for every plugin: --<name>_out for output dir, --<name>_opt for plugin options. So protoc-gen-go-grpc uses --go-grpc_out and --go-grpc_opt.
A The Go community convention is the opposite — commit the generated code. Reasons:
- A fresh
git clone+go buildworks without anyone needing to installprotocor its plugins. - Code review can see exactly what the protobuf change produced — if a field rename accidentally breaks the wire format, the diff makes it visible.
- Pinning the generated output makes builds reproducible — different
protocversions can produce slightly different code.
CI then runs make proto and verifies git diff --exit-code gen/ is empty — if not, the PR forgot to regenerate after editing a .proto file.
A From the project root (the directory containing the Makefile):
make proto— runs theprotorecipe (compiles all.protofiles into Go).make clean— runs thecleanrecipe (deletesgen/).make clean proto— runs both in order. Useful after deleting or renaming a.protofile so stale outputs don't linger.makewith no argument — runs the first target in the file (here,proto). Many projects put ahelporalltarget first as a safer default.
16 Unary Server & Client
A gRPC server has 3 parts:
- The proto-generated interface — protoc generates a Go interface called
UserServiceServerwith all your RPC methods. You MUST implement this interface. - Your implementation struct — a Go struct that holds your business logic (database connections, etc.) and implements every method from the interface.
- The server boilerplate — create a TCP listener, create a gRPC server, register your implementation, and start serving. ~15 lines of code you'll copy-paste for every project.
The client is even simpler: connect to the server, get a "stub" (auto-generated client), and call methods on it like normal functions. The stub handles all the serialization and HTTP/2 magic.
Server Implementation
// internal/service/user.go
package service
import (
"context"
"fmt"
"sync"
"github.com/google/uuid"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/types/known/emptypb"
"google.golang.org/protobuf/types/known/timestamppb"
pb "github.com/yourorg/grpc-demo/gen/user/v1"
)
// UserServer implements the generated UserServiceServer interface
type UserServer struct {
pb.UnimplementedUserServiceServer // embed for forward compatibility
mu sync.RWMutex
users map[string]*pb.User
}
func NewUserServer() *UserServer {
return &UserServer{
users: make(map[string]*pb.User),
}
}
// CreateUser — Unary RPC
func (s *UserServer) CreateUser(ctx context.Context, req *pb.CreateUserRequest) (*pb.User, error) {
// Validate input
if req.Name == "" {
return nil, status.Error(codes.InvalidArgument, "name is required")
}
if req.Email == "" {
return nil, status.Error(codes.InvalidArgument, "email is required")
}
user := &pb.User{
Id: uuid.New().String(),
Name: req.Name,
Email: req.Email,
Age: req.Age,
CreatedAt: timestamppb.Now(),
}
s.mu.Lock()
s.users[user.Id] = user
s.mu.Unlock()
return user, nil
}
// GetUser — Unary RPC
func (s *UserServer) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.User, error) {
if req.Id == "" {
return nil, status.Error(codes.InvalidArgument, "id is required")
}
s.mu.RLock()
user, ok := s.users[req.Id]
s.mu.RUnlock()
if !ok {
return nil, status.Errorf(codes.NotFound, "user %s not found", req.Id)
}
return user, nil
}
// DeleteUser — Unary RPC
func (s *UserServer) DeleteUser(ctx context.Context, req *pb.DeleteUserRequest) (*emptypb.Empty, error) {
s.mu.Lock()
defer s.mu.Unlock()
if _, ok := s.users[req.Id]; !ok {
return nil, status.Errorf(codes.NotFound, "user %s not found", req.Id)
}
delete(s.users, req.Id)
return &emptypb.Empty{}, nil
}
Server main.go
// cmd/server/main.go
package main
import (
"log"
"net"
"google.golang.org/grpc"
"google.golang.org/grpc/reflection"
pb "github.com/yourorg/grpc-demo/gen/user/v1"
"github.com/yourorg/grpc-demo/internal/service"
)
func main() {
// 1. Create a TCP listener
lis, err := net.Listen("tcp", ":50051")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
// 2. Create a gRPC server
grpcServer := grpc.NewServer()
// 3. Register our service implementation
userService := service.NewUserServer()
pb.RegisterUserServiceServer(grpcServer, userService)
// 4. Enable server reflection (for grpcurl / grpcui debugging)
reflection.Register(grpcServer)
// 5. Start serving
log.Printf("gRPC server listening on %s", lis.Addr())
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
Client Implementation
// cmd/client/main.go
package main
import (
"context"
"fmt"
"log"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
pb "github.com/yourorg/grpc-demo/gen/user/v1"
)
func main() {
// 1. Create a connection to the gRPC server
conn, err := grpc.NewClient(
"localhost:50051",
grpc.WithTransportCredentials(insecure.NewCredentials()), // no TLS for dev
)
if err != nil {
log.Fatalf("failed to connect: %v", err)
}
defer conn.Close()
// 2. Create a client stub
client := pb.NewUserServiceClient(conn)
// 3. Set a deadline/timeout
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
// 4. Call the RPC — looks like a local function call!
user, err := client.CreateUser(ctx, &pb.CreateUserRequest{
Name: "Alice",
Email: "alice@example.com",
Age: 28,
})
if err != nil {
log.Fatalf("CreateUser failed: %v", err)
}
fmt.Printf("Created user: %s (id: %s)\n", user.Name, user.Id)
// 5. Get the user back
fetched, err := client.GetUser(ctx, &pb.GetUserRequest{Id: user.Id})
if err != nil {
log.Fatalf("GetUser failed: %v", err)
}
fmt.Printf("Fetched: %s, email: %s, created: %s\n",
fetched.Name, fetched.Email, fetched.CreatedAt.AsTime())
}
A When protoc generates the UserServiceServer interface, it also generates UnimplementedUserServiceServer — a struct that provides default implementations returning "unimplemented" errors for every method.
By embedding it in your server struct:
- Forward compatibility: If someone adds a new RPC method to the proto file and regenerates, your code still compiles. The new method returns "unimplemented" by default instead of a compile error.
- Gradual implementation: You can implement methods one at a time.
If you want compile-time safety (error when a method is missing), embed UnsafeUserServiceServer instead. But this is not recommended for production.
17 Server Streaming RPC
In a normal (unary) RPC, you send one request and get one response — like ordering a single item online. Streaming means data flows continuously, like watching a YouTube video — the server keeps sending frames, and you keep receiving them in real-time.
In code, the server calls stream.Send(msg) in a loop (sending one message at a time), and the client calls stream.Recv() in a loop (receiving one message at a time). When the server is done, it returns nil, and the client gets an io.EOF ("end of file" — meaning "nothing more to read").
The client sends one request, and the server sends back a stream of responses. The server calls Send() for each message, and the client calls Recv() in a loop.
// Server-side: ListUsers streams all users one by one
func (s *UserServer) ListUsers(req *pb.ListUsersRequest, stream pb.UserService_ListUsersServer) error {
s.mu.RLock()
defer s.mu.RUnlock()
count := 0
for _, user := range s.users {
// Check if client cancelled
if err := stream.Context().Err(); err != nil {
return status.FromContextError(err).Err()
}
// Send each user as a separate message in the stream
if err := stream.Send(user); err != nil {
return err
}
count++
if req.PageSize > 0 && int32(count) >= req.PageSize {
break
}
}
return nil // returning nil closes the stream successfully
}
// Client-side: receiving the stream
func listUsers(client pb.UserServiceClient, ctx context.Context) {
stream, err := client.ListUsers(ctx, &pb.ListUsersRequest{PageSize: 10})
if err != nil {
log.Fatalf("ListUsers failed: %v", err)
}
// Read messages until the stream ends
for {
user, err := stream.Recv()
if err == io.EOF {
break // stream finished
}
if err != nil {
log.Fatalf("error receiving: %v", err)
}
fmt.Printf("User: %s (%s)\n", user.Name, user.Email)
}
}
Notice the server streaming method signature is different from unary:
Unary: func (s *Server) Method(ctx, *Request) (*Response, error)
Server stream: func (s *Server) Method(*Request, ServiceName_MethodServer) error
The ctx is accessed via stream.Context() instead of being a direct parameter.
18 Client Streaming RPC
Let's add a BulkCreateUsers method where the client streams user creation requests and the server responds with a summary.
// In the .proto file:
// rpc BulkCreateUsers(stream CreateUserRequest) returns (BulkCreateResponse);
message BulkCreateResponse {
int32 created_count = 1;
repeated string user_ids = 2;
}
// Server-side: receive a stream of create requests
func (s *UserServer) BulkCreateUsers(stream pb.UserService_BulkCreateUsersServer) error {
var ids []string
for {
req, err := stream.Recv()
if err == io.EOF {
// Client finished sending — send back the response
return stream.SendAndClose(&pb.BulkCreateResponse{
CreatedCount: int32(len(ids)),
UserIds: ids,
})
}
if err != nil {
return err
}
// Create each user
user := &pb.User{
Id: uuid.New().String(),
Name: req.Name,
Email: req.Email,
Age: req.Age,
CreatedAt: timestamppb.Now(),
}
s.mu.Lock()
s.users[user.Id] = user
s.mu.Unlock()
ids = append(ids, user.Id)
}
}
// Client-side: send a stream of create requests
func bulkCreate(client pb.UserServiceClient, ctx context.Context) {
stream, err := client.BulkCreateUsers(ctx)
if err != nil {
log.Fatal(err)
}
// Send multiple users
users := []struct{ name, email string }{
{"Alice", "alice@example.com"},
{"Bob", "bob@example.com"},
{"Charlie", "charlie@example.com"},
}
for _, u := range users {
if err := stream.Send(&pb.CreateUserRequest{
Name: u.name,
Email: u.email,
}); err != nil {
log.Fatal(err)
}
}
// Close the stream and get the response
resp, err := stream.CloseAndRecv()
if err != nil {
log.Fatal(err)
}
fmt.Printf("Created %d users: %v\n", resp.CreatedCount, resp.UserIds)
}
19 Bidirectional Streaming RPC
Both the client and server independently send streams of messages. A real-world example: a chat service.
// chat.proto
syntax = "proto3";
package chat.v1;
message ChatMessage {
string user = 1;
string content = 2;
int64 sent_at = 3;
}
service ChatService {
rpc Chat(stream ChatMessage) returns (stream ChatMessage);
}
// Server-side: bidirectional chat
func (s *ChatServer) Chat(stream pb.ChatService_ChatServer) error {
for {
// Receive a message from the client
msg, err := stream.Recv()
if err == io.EOF {
return nil
}
if err != nil {
return err
}
log.Printf("[%s]: %s", msg.User, msg.Content)
// Echo back (in real app, you'd broadcast to all connected clients)
resp := &pb.ChatMessage{
User: "server",
Content: fmt.Sprintf("Echo: %s", msg.Content),
SentAt: time.Now().Unix(),
}
if err := stream.Send(resp); err != nil {
return err
}
}
}
// Client-side: bidirectional chat
func chat(client pb.ChatServiceClient, ctx context.Context) {
stream, err := client.Chat(ctx)
if err != nil {
log.Fatal(err)
}
// Goroutine to receive messages
go func() {
for {
msg, err := stream.Recv()
if err == io.EOF {
return
}
if err != nil {
log.Printf("receive error: %v", err)
return
}
fmt.Printf("[%s]: %s\n", msg.User, msg.Content)
}
}()
// Send messages
messages := []string{"Hello!", "How are you?", "Goodbye!"}
for _, text := range messages {
if err := stream.Send(&pb.ChatMessage{
User: "alice",
Content: text,
SentAt: time.Now().Unix(),
}); err != nil {
log.Fatal(err)
}
time.Sleep(500 * time.Millisecond)
}
stream.CloseSend() // signal we're done sending
time.Sleep(time.Second) // wait for remaining responses
}
20 Error Handling & Status Codes
In REST, you return HTTP status codes: 404 Not Found, 500 Internal Server Error, 401 Unauthorized. gRPC has its own set of status codes — they're similar but not identical.
Every gRPC error has two parts:
- Code — a number like
NotFound(5),InvalidArgument(3),Internal(13) - Message — a human-readable string like "user abc-123 not found"
You create errors with status.Error(codes.NotFound, "user not found") and check them on the client with status.FromError(err). Clients use the code to decide what to do — Unavailable means "retry later", InvalidArgument means "fix your input."
google.golang.org/grpc/codes. Every gRPC error is a *status.Status containing a code, a human-readable message, and optional detail messages (structured error information).gRPC Status Codes
| Code | Number | HTTP Equiv | When to Use |
|---|---|---|---|
OK | 0 | 200 | Success |
Cancelled | 1 | 499 | Client cancelled the request |
Unknown | 2 | 500 | Unknown error (catch-all) |
InvalidArgument | 3 | 400 | Client sent invalid data |
DeadlineExceeded | 4 | 504 | Operation timed out |
NotFound | 5 | 404 | Resource doesn't exist |
AlreadyExists | 6 | 409 | Resource already exists |
PermissionDenied | 7 | 403 | Caller doesn't have permission |
ResourceExhausted | 8 | 429 | Rate limit, quota exceeded |
FailedPrecondition | 9 | 400 | System not in required state |
Aborted | 10 | 409 | Transaction conflict |
OutOfRange | 11 | 400 | Operation past valid range |
Unimplemented | 12 | 501 | Method not implemented |
Internal | 13 | 500 | Internal server error |
Unavailable | 14 | 503 | Service temporarily unavailable (retry) |
DataLoss | 15 | 500 | Unrecoverable data loss |
Unauthenticated | 16 | 401 | Missing/invalid authentication |
Returning Errors
import (
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
// Simple error with code + message
return nil, status.Error(codes.NotFound, "user not found")
// Formatted error message
return nil, status.Errorf(codes.InvalidArgument, "name must be at least %d characters", 3)
// Wrapping an existing error
if err := db.Query(ctx); err != nil {
return nil, status.Errorf(codes.Internal, "database error: %v", err)
}
Rich Error Details
import (
"google.golang.org/genproto/googleapis/rpc/errdetails"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
func validateCreateUser(req *pb.CreateUserRequest) error {
var violations []*errdetails.BadRequest_FieldViolation
if req.Name == "" {
violations = append(violations, &errdetails.BadRequest_FieldViolation{
Field: "name",
Description: "name is required",
})
}
if len(req.Name) > 100 {
violations = append(violations, &errdetails.BadRequest_FieldViolation{
Field: "name",
Description: "name must be under 100 characters",
})
}
if req.Email == "" {
violations = append(violations, &errdetails.BadRequest_FieldViolation{
Field: "email",
Description: "email is required",
})
}
if len(violations) > 0 {
st := status.New(codes.InvalidArgument, "validation failed")
st, _ = st.WithDetails(&errdetails.BadRequest{
FieldViolations: violations,
})
return st.Err()
}
return nil
}
Handling Errors on the Client
// Client-side error handling
user, err := client.GetUser(ctx, &pb.GetUserRequest{Id: "bad-id"})
if err != nil {
// Extract the gRPC status from the error
st, ok := status.FromError(err)
if !ok {
log.Fatalf("not a gRPC error: %v", err)
}
// Check the status code
switch st.Code() {
case codes.NotFound:
fmt.Println("User not found")
case codes.InvalidArgument:
fmt.Printf("Bad request: %s\n", st.Message())
// Extract rich error details
for _, detail := range st.Details() {
if badReq, ok := detail.(*errdetails.BadRequest); ok {
for _, v := range badReq.FieldViolations {
fmt.Printf(" - %s: %s\n", v.Field, v.Description)
}
}
}
case codes.Unavailable:
fmt.Println("Service unavailable — retrying...")
default:
fmt.Printf("RPC error: [%s] %s\n", st.Code(), st.Message())
}
}
21 Metadata (Headers)
Metadata = HTTP headers for gRPC. Just like when you send an HTTP request you can include headers like Authorization: Bearer token123 or X-Request-ID: abc, gRPC lets you attach key-value pairs to every RPC call.
Common uses: sending auth tokens, passing request IDs for tracing, sending client version info. The server reads them from the context, and can also send metadata back in the response (like response headers).
gRPC metadata is equivalent to HTTP headers. It carries key-value pairs alongside RPC calls — authentication tokens, request IDs, tracing info, etc.
import "google.golang.org/grpc/metadata"
// ============ CLIENT SIDE: Sending metadata ============
// Create metadata
md := metadata.New(map[string]string{
"authorization": "Bearer eyJhbGciOi...",
"x-request-id": "req-abc-123",
})
// Attach to context
ctx := metadata.NewOutgoingContext(context.Background(), md)
// Or append to existing metadata
ctx = metadata.AppendToOutgoingContext(ctx,
"x-custom-header", "value1",
"x-another", "value2",
)
// Make the call — metadata is sent automatically
user, err := client.GetUser(ctx, req)
// ============ CLIENT SIDE: Reading response metadata ============
var header, trailer metadata.MD
user, err := client.GetUser(ctx, req,
grpc.Header(&header), // captures response headers
grpc.Trailer(&trailer), // captures response trailers
)
fmt.Println("Server version:", header.Get("x-server-version"))
// ============ SERVER SIDE: Reading incoming metadata ============
func (s *UserServer) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.User, error) {
// Extract metadata from the incoming context
md, ok := metadata.FromIncomingContext(ctx)
if !ok {
return nil, status.Error(codes.Internal, "no metadata")
}
// Read a specific header
authHeader := md.Get("authorization")
if len(authHeader) == 0 {
return nil, status.Error(codes.Unauthenticated, "missing auth token")
}
requestID := md.Get("x-request-id")
log.Printf("Request ID: %s", requestID)
// ============ SERVER SIDE: Sending response metadata ============
// Send headers (before the response)
header := metadata.Pairs(
"x-server-version", "1.2.0",
"x-request-id", requestID[0],
)
grpc.SendHeader(ctx, header)
// Send trailers (after the response)
trailer := metadata.Pairs("x-processing-time-ms", "42")
grpc.SetTrailer(ctx, trailer)
// ... return the response
return user, nil
}
gRPC metadata keys are always lowercase. "X-Request-ID" becomes "x-request-id". Binary values must use keys ending in -bin (they're automatically base64 encoded).
22 Interceptors (Middleware)
If you've used Express.js, you know middleware — functions that run before/after every request (logging, auth checks, error handling). Interceptors are gRPC's middleware.
An interceptor wraps every RPC call. The flow is: request comes in → interceptor #1 runs → interceptor #2 runs → your actual handler runs → interceptors see the response on the way back out. You chain multiple interceptors together (logging + auth + rate limiting).
There are 4 types because gRPC has both unary and streaming RPCs, and interceptors exist on both server and client sides: unary server, stream server, unary client, stream client.
Unary Server Interceptor
// Logging interceptor — logs every RPC call
func loggingInterceptor(
ctx context.Context,
req interface{},
info *grpc.UnaryServerInfo,
handler grpc.UnaryHandler,
) (interface{}, error) {
start := time.Now()
// Call the actual handler
resp, err := handler(ctx, req)
// Log after the call
duration := time.Since(start)
st, _ := status.FromError(err)
log.Printf("method=%s duration=%s code=%s error=%v",
info.FullMethod, duration, st.Code(), err)
return resp, err
}
Auth Interceptor
// Authentication interceptor
func authInterceptor(
ctx context.Context,
req interface{},
info *grpc.UnaryServerInfo,
handler grpc.UnaryHandler,
) (interface{}, error) {
// Skip auth for health check
if info.FullMethod == "/grpc.health.v1.Health/Check" {
return handler(ctx, req)
}
// Extract token from metadata
md, ok := metadata.FromIncomingContext(ctx)
if !ok {
return nil, status.Error(codes.Unauthenticated, "no metadata")
}
tokens := md.Get("authorization")
if len(tokens) == 0 {
return nil, status.Error(codes.Unauthenticated, "missing token")
}
// Validate the token
claims, err := validateJWT(tokens[0])
if err != nil {
return nil, status.Error(codes.Unauthenticated, "invalid token")
}
// Add user info to context for downstream handlers
ctx = context.WithValue(ctx, "user_id", claims.UserID)
ctx = context.WithValue(ctx, "user_role", claims.Role)
return handler(ctx, req)
}
Recovery Interceptor (Panic Handler)
// Recover from panics — prevents a single bad request from crashing the server
func recoveryInterceptor(
ctx context.Context,
req interface{},
info *grpc.UnaryServerInfo,
handler grpc.UnaryHandler,
) (resp interface{}, err error) {
defer func() {
if r := recover(); r != nil {
log.Printf("PANIC in %s: %v\n%s", info.FullMethod, r, debug.Stack())
err = status.Errorf(codes.Internal, "internal error")
}
}()
return handler(ctx, req)
}
Chaining Interceptors
// Register multiple interceptors — they execute in order
grpcServer := grpc.NewServer(
grpc.ChainUnaryInterceptor(
recoveryInterceptor, // 1st: catch panics
loggingInterceptor, // 2nd: log the call
authInterceptor, // 3rd: authenticate
),
grpc.ChainStreamInterceptor(
streamLoggingInterceptor,
streamAuthInterceptor,
),
)
Stream Server Interceptor
// Stream interceptor has a slightly different signature
func streamLoggingInterceptor(
srv interface{},
ss grpc.ServerStream,
info *grpc.StreamServerInfo,
handler grpc.StreamHandler,
) error {
start := time.Now()
log.Printf("Stream started: %s", info.FullMethod)
err := handler(srv, ss)
log.Printf("Stream ended: %s duration=%s error=%v",
info.FullMethod, time.Since(start), err)
return err
}
Client-Side Interceptor
// Client interceptor — add auth token to every outgoing call
func clientAuthInterceptor(
ctx context.Context,
method string,
req, reply interface{},
cc *grpc.ClientConn,
invoker grpc.UnaryInvoker,
opts ...grpc.CallOption,
) error {
// Inject auth token into metadata
ctx = metadata.AppendToOutgoingContext(ctx,
"authorization", "Bearer "+getToken(),
)
return invoker(ctx, method, req, reply, cc, opts...)
}
// Use it when creating the client connection
conn, err := grpc.NewClient(
"localhost:50051",
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithChainUnaryInterceptor(clientAuthInterceptor),
)
23 Deadlines & Timeouts
Imagine you order food and the restaurant says "it'll be ready soon." You wait 5 minutes. 10 minutes. 30 minutes. An hour. At some point, you should give up and go somewhere else, right?
That's what a deadline does — it says "if this request isn't done in 5 seconds, cancel it." Without deadlines, a slow or broken server will make your client wait forever. And in microservices, if Service A waits forever for Service B, then clients waiting for Service A also wait forever — the slowness cascades and takes down your whole system.
The magic of gRPC deadlines: they propagate. If A gives B a 5-second deadline, and 2 seconds pass before B calls C, then C automatically gets a 3-second deadline. No manual calculation needed.
// CLIENT: Set a timeout (most common)
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
user, err := client.GetUser(ctx, req)
// CLIENT: Set an absolute deadline
ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(5*time.Second))
defer cancel()
user, err := client.GetUser(ctx, req)
// Check error type
if err != nil {
st, _ := status.FromError(err)
if st.Code() == codes.DeadlineExceeded {
fmt.Println("Request timed out!")
}
}
// SERVER: Check remaining deadline
func (s *UserServer) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.User, error) {
// Check if there's a deadline
deadline, ok := ctx.Deadline()
if ok {
remaining := time.Until(deadline)
log.Printf("Time remaining: %v", remaining)
if remaining < 100*time.Millisecond {
return nil, status.Error(codes.DeadlineExceeded, "not enough time")
}
}
// Long operation — check context periodically
select {
case <-ctx.Done():
return nil, status.FromContextError(ctx.Err()).Err()
case result := <-doExpensiveWork(ctx):
return result, nil
}
}
If a client doesn't set a deadline, the RPC can hang forever. Every gRPC call should have a deadline. Google's internal guidelines require all RPCs to have a deadline. A good default is 5-30 seconds for normal operations, longer for streaming or batch jobs.
24 TLS & mTLS
- TLS (Transport Layer Security) = encryption. Like sending a letter in a locked box instead of a postcard. Only the intended recipient can read it. The server proves its identity with a certificate (like showing an ID card). This is what HTTPS does.
- mTLS (mutual TLS) = BOTH sides prove their identity. Not only does the server show its ID, but the client also shows its ID. Like both people showing passports at a meeting. Used in microservices where you want to ensure only your own services can talk to each other — no random outsider can connect.
Rule: Never run gRPC without TLS in production. Plaintext (insecure.NewCredentials()) is only for local development.
Server-Side TLS
// Server with TLS
import "google.golang.org/grpc/credentials"
func main() {
// Load server certificate and key
creds, err := credentials.NewServerTLSFromFile("certs/server.crt", "certs/server.key")
if err != nil {
log.Fatalf("failed to load certs: %v", err)
}
grpcServer := grpc.NewServer(
grpc.Creds(creds),
)
// ... register services and serve
}
// Client connecting with TLS
creds, err := credentials.NewClientTLSFromFile("certs/ca.crt", "server.example.com")
if err != nil {
log.Fatal(err)
}
conn, err := grpc.NewClient(
"server.example.com:50051",
grpc.WithTransportCredentials(creds),
)
Mutual TLS (mTLS)
// mTLS — both server AND client present certificates
import (
"crypto/tls"
"crypto/x509"
"os"
"google.golang.org/grpc/credentials"
)
// Server with mTLS
func newMTLSServer() *grpc.Server {
// Load server certificate
serverCert, err := tls.LoadX509KeyPair("certs/server.crt", "certs/server.key")
if err != nil {
log.Fatal(err)
}
// Load CA certificate to verify clients
caCert, err := os.ReadFile("certs/ca.crt")
if err != nil {
log.Fatal(err)
}
caPool := x509.NewCertPool()
caPool.AppendCertsFromPEM(caCert)
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{serverCert},
ClientAuth: tls.RequireAndVerifyClientCert, // ← require client cert
ClientCAs: caPool,
MinVersion: tls.VersionTLS13,
}
creds := credentials.NewTLS(tlsConfig)
return grpc.NewServer(grpc.Creds(creds))
}
25 Health Checking & Reflection
Health checking: Kubernetes and load balancers constantly ask your server "are you alive? are you ready to handle requests?" If your server says "no", traffic gets routed to other healthy instances. gRPC has a built-in standard protocol for this — you just register a health server and set your status.
Reflection: Normally, you need the .proto file to know what RPCs a server supports. Reflection lets tools like grpcurl (the curl of gRPC) discover services at runtime — like an API that describes itself. Super useful for debugging, but disable in production for security.
grpcurl is like Postman/curl for gRPC. grpcui is a web UI version of it — you get a browser form where you can pick an RPC method, fill in fields, and see the response.
Health Checking
gRPC has a standard health checking protocol. Kubernetes, load balancers, and service meshes use it to determine if your service is ready.
import (
"google.golang.org/grpc/health"
healthpb "google.golang.org/grpc/health/grpc_health_v1"
)
func main() {
grpcServer := grpc.NewServer()
// Create and register health server
healthServer := health.NewServer()
healthpb.RegisterHealthServer(grpcServer, healthServer)
// Set service status
healthServer.SetServingStatus(
"user.v1.UserService", // service name
healthpb.HealthCheckResponse_SERVING, // status
)
// Set overall server status
healthServer.SetServingStatus(
"", // empty string = overall server
healthpb.HealthCheckResponse_SERVING,
)
// When shutting down or during maintenance:
// healthServer.SetServingStatus("", healthpb.HealthCheckResponse_NOT_SERVING)
}
# Test health from command line
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check
# Check specific service
grpcurl -plaintext -d '{"service": "user.v1.UserService"}' \
localhost:50051 grpc.health.v1.Health/Check
Server Reflection
Reflection lets clients (like grpcurl) discover your services and methods at runtime, without having the .proto files.
import "google.golang.org/grpc/reflection"
func main() {
grpcServer := grpc.NewServer()
// Register your services...
pb.RegisterUserServiceServer(grpcServer, userService)
// Enable reflection — only for dev/staging, disable in prod
reflection.Register(grpcServer)
// ...
}
grpcurl & grpcui — Debugging Tools
# Install
brew install grpcurl grpcui
# List all services
grpcurl -plaintext localhost:50051 list
# Describe a service
grpcurl -plaintext localhost:50051 describe user.v1.UserService
# Describe a message
grpcurl -plaintext localhost:50051 describe user.v1.User
# Call an RPC
grpcurl -plaintext -d '{"name": "Alice", "email": "alice@example.com", "age": 28}' \
localhost:50051 user.v1.UserService/CreateUser
# grpcui — web-based UI for gRPC (like Postman for gRPC)
grpcui -plaintext localhost:50051
# Opens a browser with a form-based UI for making RPC calls
26 gRPC Gateway (REST Proxy)
- Internal services communicate via gRPC (fast, type-safe)
- External/browser clients use REST/JSON (familiar, no proxy needed)
- One source of truth (
.protofile) for both APIs - No need to maintain two separate API implementations
The problem: you built a fast gRPC service for your microservices, but your React frontend can't call gRPC directly from the browser (browsers only speak HTTP/JSON natively).
The solution: gRPC-Gateway automatically generates a REST API proxy. You add annotations to your .proto file saying "this RPC should also be available at GET /v1/users/{id}", and the gateway translates:
- Browser sends:
GET /v1/users/123(REST/JSON) - Gateway converts to:
client.GetUser({id: "123"})(gRPC/protobuf) - gRPC server responds with protobuf
- Gateway converts back to JSON and returns to browser
You write your service once and get both gRPC and REST for free.
Annotating your Proto for REST
syntax = "proto3";
package user.v1;
import "google/api/annotations.proto";
service UserService {
rpc CreateUser(CreateUserRequest) returns (User) {
option (google.api.http) = {
post: "/v1/users"
body: "*"
};
}
rpc GetUser(GetUserRequest) returns (User) {
option (google.api.http) = {
get: "/v1/users/{id}" // {id} maps to GetUserRequest.id
};
}
rpc UpdateUser(UpdateUserRequest) returns (User) {
option (google.api.http) = {
patch: "/v1/users/{user.id}"
body: "user"
};
}
rpc DeleteUser(DeleteUserRequest) returns (google.protobuf.Empty) {
option (google.api.http) = {
delete: "/v1/users/{id}"
};
}
rpc ListUsers(ListUsersRequest) returns (ListUsersResponse) {
option (google.api.http) = {
get: "/v1/users" // query params: ?page_size=10&page_token=abc
};
}
}
Gateway Server
// cmd/gateway/main.go
package main
import (
"context"
"log"
"net/http"
"github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
gw "github.com/yourorg/grpc-demo/gen/user/v1"
)
func main() {
ctx := context.Background()
// Create a gRPC-Gateway mux
mux := runtime.NewServeMux()
// Register the gRPC service endpoint
opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())}
err := gw.RegisterUserServiceHandlerFromEndpoint(ctx, mux, "localhost:50051", opts)
if err != nil {
log.Fatal(err)
}
// Start HTTP server
log.Println("REST gateway listening on :8080")
if err := http.ListenAndServe(":8080", mux); err != nil {
log.Fatal(err)
}
}
// Now you can:
// curl -X POST http://localhost:8080/v1/users -d '{"name":"Alice","email":"alice@example.com"}'
// curl http://localhost:8080/v1/users/u-123
// curl http://localhost:8080/v1/users?page_size=10
27 Testing gRPC Services
Normally, testing a gRPC service requires starting a real server on a port, connecting a client, and cleaning up afterward. This is slow and can have port conflicts when running tests in parallel.
bufconn creates a fake, in-memory network connection. The server and client talk through memory instead of TCP — no ports, no network, instant. Your tests run in milliseconds and never conflict with each other.
The pattern: create a bufconn listener → start server on it → create client that connects through it → test your RPCs → cleanup. Same behavior as real network, but way faster.
In-Process Testing (No Network)
// internal/service/user_test.go
package service
import (
"context"
"net"
"testing"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/status"
"google.golang.org/grpc/test/bufconn"
pb "github.com/yourorg/grpc-demo/gen/user/v1"
)
const bufSize = 1024 * 1024
// Setup: create an in-memory gRPC server (no real TCP)
func setupTest(t *testing.T) (pb.UserServiceClient, func()) {
t.Helper()
// bufconn creates an in-memory listener — no network needed
lis := bufconn.Listen(bufSize)
srv := grpc.NewServer()
pb.RegisterUserServiceServer(srv, NewUserServer())
go srv.Serve(lis)
// Create client that connects through bufconn
conn, err := grpc.NewClient(
"passthrough://bufnet",
grpc.WithContextDialer(func(ctx context.Context, addr string) (net.Conn, error) {
return lis.DialContext(ctx)
}),
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil {
t.Fatalf("failed to dial: %v", err)
}
client := pb.NewUserServiceClient(conn)
cleanup := func() {
conn.Close()
srv.Stop()
}
return client, cleanup
}
// Test: CreateUser success
func TestCreateUser(t *testing.T) {
client, cleanup := setupTest(t)
defer cleanup()
ctx := context.Background()
user, err := client.CreateUser(ctx, &pb.CreateUserRequest{
Name: "Alice",
Email: "alice@example.com",
Age: 28,
})
if err != nil {
t.Fatalf("CreateUser failed: %v", err)
}
if user.Name != "Alice" {
t.Errorf("expected name Alice, got %s", user.Name)
}
if user.Id == "" {
t.Error("expected non-empty ID")
}
}
// Test: GetUser not found
func TestGetUser_NotFound(t *testing.T) {
client, cleanup := setupTest(t)
defer cleanup()
_, err := client.GetUser(context.Background(), &pb.GetUserRequest{
Id: "nonexistent",
})
if err == nil {
t.Fatal("expected error, got nil")
}
st, ok := status.FromError(err)
if !ok {
t.Fatalf("expected gRPC status error, got: %v", err)
}
if st.Code() != codes.NotFound {
t.Errorf("expected NotFound, got %s", st.Code())
}
}
// Test: CreateUser validation
func TestCreateUser_Validation(t *testing.T) {
client, cleanup := setupTest(t)
defer cleanup()
tests := []struct {
name string
req *pb.CreateUserRequest
wantCode codes.Code
}{
{
name: "missing name",
req: &pb.CreateUserRequest{Email: "a@b.com"},
wantCode: codes.InvalidArgument,
},
{
name: "missing email",
req: &pb.CreateUserRequest{Name: "Alice"},
wantCode: codes.InvalidArgument,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := client.CreateUser(context.Background(), tt.req)
if err == nil {
t.Fatal("expected error")
}
st, _ := status.FromError(err)
if st.Code() != tt.wantCode {
t.Errorf("expected %s, got %s", tt.wantCode, st.Code())
}
})
}
}
google.golang.org/grpc/test/bufconn creates an in-memory listener. Tests run without network I/O — no port conflicts, no cleanup, blazing fast. Always use bufconn for unit tests. Use real TCP listeners for integration/E2E tests.
28 Best Practices
Proto Design
Don't reuse the same message for multiple RPCs. Each RPC should have its own request and response message, even if they look identical today.
// BAD — reusing User as both input and output
rpc CreateUser(User) returns (User);
rpc UpdateUser(User) returns (User);
// GOOD — dedicated messages per RPC
rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse);
Why: Requirements evolve differently. CreateUser might later need an idempotency_key, UpdateUser might need a field_mask. Shared messages couple evolution.
Always use versioned packages: package user.v1;, not package user;. When you need breaking changes, create user.v2. Both versions coexist — old clients use v1, new clients use v2.
Don't rely on "zero value means don't update." Use google.protobuf.FieldMask to explicitly specify which fields to update. This is the pattern all Google APIs use.
message UpdateUserRequest {
User user = 1;
google.protobuf.FieldMask update_mask = 2;
}
Enum values are scoped to the package, not the enum. This means two enums in the same package can't have values with the same name. Prefix all values with the enum name.
// BAD — ACTIVE could conflict with another enum
enum UserStatus { UNSPECIFIED = 0; ACTIVE = 1; }
enum OrderStatus { UNSPECIFIED = 0; ACTIVE = 1; } // CONFLICT!
// GOOD — prefixed, no conflicts
enum UserStatus { USER_STATUS_UNSPECIFIED = 0; USER_STATUS_ACTIVE = 1; }
enum OrderStatus { ORDER_STATUS_UNSPECIFIED = 0; ORDER_STATUS_ACTIVE = 1; }
Server Implementation
Every RPC call should have a deadline. Without one, a hung server means a hung client — forever.
// Always do this
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
resp, err := client.GetUser(ctx, req)
Don't use codes.Internal for everything. Use the right code:
- User sent bad data →
InvalidArgument - Resource doesn't exist →
NotFound - Missing auth →
Unauthenticated - No permission →
PermissionDenied - Rate limited →
ResourceExhausted - Bug in your code →
Internal - Dependency down →
Unavailable
Clients rely on codes for retry logic — Unavailable is retryable, InvalidArgument is not.
If your handler does expensive work (DB queries, external calls), check ctx.Err() periodically. This ensures you stop work immediately when the client cancels or the deadline expires.
func (s *Server) HeavyOperation(ctx context.Context, req *pb.Request) (*pb.Response, error) {
// Step 1
result1, err := s.db.Query(ctx, ...)
if ctx.Err() != nil {
return nil, status.FromContextError(ctx.Err()).Err()
}
// Step 2
result2, err := s.externalAPI.Call(ctx, ...)
if ctx.Err() != nil {
return nil, status.FromContextError(ctx.Err()).Err()
}
// ...
}
Always handle SIGTERM/SIGINT for graceful shutdown. GracefulStop() finishes in-flight RPCs before stopping.
func main() {
grpcServer := grpc.NewServer()
// ... register services
// Graceful shutdown on signal
go func() {
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
<-sigCh
log.Println("Shutting down gracefully...")
grpcServer.GracefulStop()
}()
grpcServer.Serve(lis)
}
Production Checklist
| Item | Status | Notes |
|---|---|---|
| TLS enabled | Required | Never run plaintext in prod |
| Health check registered | Required | K8s liveness/readiness probes |
| Deadlines on all client calls | Required | Prevents cascading hangs |
| Logging interceptor | Required | Log method, duration, status code |
| Recovery interceptor | Required | Catch panics, return Internal |
| Metrics (Prometheus) | Recommended | Use go-grpc-prometheus |
| Tracing (OpenTelemetry) | Recommended | Distributed tracing across services |
| Proto linting (buf lint) | Recommended | Enforce naming conventions |
| Breaking change detection | Recommended | buf breaking in CI |
| Reflection disabled | Nice to have | Disable in prod for security |
| Graceful shutdown | Required | Handle SIGTERM properly |
| Rate limiting | Recommended | Interceptor or service mesh |
| Max message size configured | Recommended | Default is 4MB, adjust if needed |
This guide covered the core of protobuf and gRPC. For advanced patterns, look into: gRPC load balancing (client-side with grpc.WithDefaultServiceConfig), OpenTelemetry integration (otelgrpc), protovalidate (schema-level validation), and Connect-Go (a modern alternative that supports gRPC, gRPC-Web, and Connect protocols with a single handler).