Envoy as an API Gateway: Part I

Oct 03, 2021
Bazel Golang Envoy API Proxy Microservices
9 min read
Source code

Part N: Overview
Part I: gRPC microservice
Part II: Envoy proxy container
Part III: Kubernetes and Tilt
Part IV: Authentication and authorization
Part V: CI/CD pipeline with Github Actions

gRPC

Outcome
Service One
Protobuf definition.
Implementation
- It’s time to run go build, right?
BUILD files
Rules and targets

Outcome

By finishing this part, we will have a Go binary and a Docker image containing this binary.

Service One

It’s an elementary gRPC server. It returns a message with a msg field and value Hello, %name%, where %name% is an incoming parameter.

Here is a service structure.

% tree

service-one
├── BUILD
├── main.go
├── pb
│   ├── BUILD
│   └── service-one.proto
└── server
    ├── BUILD
    └── server.go

Let’s take a closer look at what it contains.

Protobuf definition.

// service-one.proto

syntax = "proto3";

package svc;

option go_package = "github.com/ekhabarov/bazel-k8s-envoy/service-one/pb";

import "google/api/annotations.proto";

service ServiceOne {
  rpc Hello(HelloRequest) returns (HelloResponse){
    option (google.api.http) = { get: "/v1/hello" };
  }
}

message HelloRequest {
  string name = 1;
 }

message HelloResponse {
  string msg = 1;
}

The core of our service is a service-one.proto file, which defines an interface of our service: method(s), messages and types. One of the important thing here is a method option:

  rpc Hello(HelloRequest) returns (HelloResponse){
    option (google.api.http) = { get: "/v1/hello" };
  }

It says which HTTP endpoint to use for the gRPC method. Envoy will later use this information.

Implementation

Implementation has an empty Server struct with the attached method Hello, which returns a HelloResponse message.

// server.go

package server

import (
	"context"

	// go_package value form service-one.proto
	"github.com/ekhabarov/bazel-k8s-envoy/service-one/pb"
)

type Server struct{}

func (s *Server) Hello(ctx context.Context, req *pb.HelloRequest) (*pb.HelloResponse, error) {
	return &pb.HelloResponse{
		Body: "Hello, " + req.Name,
	}, nil
}

And the main package. It combines all together, creates a new gRPC server, attaches our implementation to it, and binds the gRPC server to listener port 5000.

// main.go

package main

import (
	"net"

	"github.com/ekhabarov/bazel-k8s-envoy/service-one/pb"
	"github.com/ekhabarov/bazel-k8s-envoy/service-one/server"
	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection" // One
)

func main() {
	s := grpc.NewServer()
	pb.RegisterServiceOneServer(s, &server.Server{})
	reflection.Register(s) // Two

	lis, err := net.Listen("tcp", ":5000")
	if err != nil {
		panic(err)
	}

	s.Serve(lis)
}

What are the comments about? These two lines add support for the gRPC reflection API to our server. Without these lines, grpcurl cannot request our server; namely, you’ll get an error like this:

Error invoking method "svc.ServiceOne.Hello": failed to query for service
descriptor "svc.ServiceOne": server does not support the reflection API

It’s time to run `go build`, right?

Well, technically, you can run it, but it doesn’t work, and you will get an error because:

there is no go.sum file in the repo (go.mod is located on the upper level),
there is no generated service-one.pb.go file, which should contain gRPC stubs. What is the *.pb.go file? See here.

Instead of go build we will be using bazel build, and BUILD files in each project directory contain instructions for Bazel on how to build each package and what it makes as an output. Let’s look at them closer.

BUILD files

The first file we look at is a BUILD file for the pb package.

# service-one/pb/BUILD

load("@rules_proto//proto:defs.bzl", "proto_descriptor_set", "proto_library")
load("@io_bazel_rules_go//go:def.bzl", "go_library")
load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")

package(default_visibility = ["//visibility:public"])

proto_library(
    name = "pb_proto",
    srcs = ["service-one.proto"],
    deps = ["@go_googleapis//google/api:annotations_proto"],
)

go_proto_library(
    name = "pb_go_proto",
    compilers = ["@io_bazel_rules_go//proto:go_grpc"],
    importpath = "github.com/ekhabarov/bazel-k8s-envoy/service-one/pb",
    proto = ":pb_proto",
    deps = ["@go_googleapis//google/api:annotations_go_proto"],
)

go_library(
    name = "pb",
    embed = [":pb_go_proto"],
    importpath = "github.com/ekhabarov/bazel-k8s-envoy/service-one/pb",
)

proto_descriptor_set(
    name = "service_one_descriptor_set",
    deps = [
        ":pb_proto",
        "@go_googleapis//google/api:annotations_proto",
    ],
)

it contains:

load directives to load external rules, like proto_library, go_library, etc.
Rule calls, each of them does a piece of work towards building a complete package.

Rules and targets

One rule is one piece of work. Each rule (or target¹ in terms of Makefiles) has an input (srcs param) and an output (will reside in Bazel cache). srcs param usually accepts a list of source files, while production heavily depends on what rule does: it could be one file or set of files, and so on. Also, a rule has a name by which we can execute it.

Due to the nature of Bazel, each rule must explicitly define input and output. Bazel will not process other files.

In our project, we use the following rules:

proto_library (rules_proto)

Docs

proto_library(
    name = "pb_proto",
    srcs = ["service-one.proto"],
    deps = ["@go_googleapis//google/api:annotations_proto"],
)

This rule produces bazel-bin/service-one/pb/pb_proto-descriptor-set.proto.bin, which is the Protobuf descriptor set binary file. Due to our proto file containing an import, we have to provide this dependency for the rule. In this case, Bazel takes only two files: service-one.proto, and file produced by @go_googleapis//google/api:annotations_proto target and runs the rule.

NOTE: we also can run bazel build @go_googleapis//google/api:annotations_proto and see what we will receive as an output.

% bazel build @go_googleapis//google/api:annotations_proto
...
Target @go_googleapis//google/api:annotations_proto up-to-date:
  bazel-bin/external/go_googleapis/google/api/annotations_proto-descriptor-set.proto.bin
...

go_proto_library (rules_go)

Docs

go_proto_library(
    name = "pb_go_proto",
    compilers = ["@io_bazel_rules_go//proto:go_grpc"],
    importpath = "github.com/ekhabarov/bazel-k8s-envoy/service-one/pb",
    proto = ":pb_proto",
    deps = ["@go_googleapis//google/api:annotations_go_proto"],
)

This rule takes a proto descriptor set, generated by :pb_proto rule (proto = ":pb_proto", parameter), and produces a pb_go_proto.a and service-one.pb.go files. *.a is a compiled package, created by go tool pack, and *.pg.go contains server and client stubs. Yes, Bazel calls Go and tools under the hood.

compilers parameter says, to add server and client interfaces, ServiceOneServer and ServiceOneClient respectively, to generated code. Technically, it equals to protoc argument: --go_out=plugins=grpc:..
importpath defines how we can import this Go library, plus it is used by Gazelle when it generates BUILD files.
deps = ["@go_googleapis//google/api:annotations_go_proto"] generates annotations.pb.go file which is imported by service-one.pb.go.

go_library (rules_go)

Docs

go_library(
    name = "pb",
    embed = [":pb_go_proto"],
    importpath = "github.com/ekhabarov/bazel-k8s-envoy/service-one/pb",
)

In our case, with just one *.proto file in a package, this rule is just a wrapper for :pb_go_proto. The package can contain more *.go files, and then they will be listed in srcs parameter of the rule.

proto_descriptor_set (rules_proto)

Docs

proto_descriptor_set(
    name = "service_one_descriptor_set",
    deps = [
        ":pb_proto",
        "@go_googleapis//google/api:annotations_proto",
    ],

proto_descriptor_set generates descriptor sets, i.e. binary representation of proto file(s).

How about :pb_proto target? Doesn’t it do the same thing? Almost. The difference between the two is :pb_proto generates descriptor sets for files listed in srcs only, while Envoy, which uses these files, needs to have descriptor sets for all proto files and their dependencies. proto_descriptor_set does it for all labels listed in deps.

The following BUILD file is in the server package, but nothing new there, just a go_library which includes //service-one/pb target from pb package as a dependency. We skip it.

More interesting fine is an one in service-one package.

load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
load("@io_bazel_rules_docker//container:container.bzl", "container_image", "container_push")

go_library(
    name = "service-one_lib",
    srcs = ["main.go"],
    importpath = "github.com/ekhabarov/bazel-k8s-envoy/service-one",
    visibility = ["//visibility:private"],
    deps = [
        "//service-one/pb",
        "//service-one/server",
        "@org_golang_google_grpc//:go_default_library",
        "@org_golang_google_grpc//reflection",
    ],
)

go_binary(
    name = "service-one",
    embed = [":service-one_lib"],
    goos = "linux",
    goarch = "amd64",
    visibility = ["//visibility:public"],
)

container_image(
    name = "image",
    base = "@go_base//image",
    entrypoint = ["/service-one"],
    files = [
        ":service-one",
    ],
    ports = [
        "5000/tcp",
    ],
)

genrule(
    name = "yaml",
    srcs = [
        "//k8s:defaults.yaml",
        "//k8s/base:service.yaml",
        ":values.yaml",
    ],
    outs = ["service-one.yaml"],
    cmd = """
      echo $(location @com_github_vmware_tanzu_carvel_ytt//cmd/ytt) \
          --dangerous-allow-all-symlink-destinations \
         `ls $(SRCS) | sed 's/^/-f /g'` \
          > $@
    """,
    executable = True,
    tools = [
        "@com_github_vmware_tanzu_carvel_ytt//cmd/ytt",
    ],
)

container_push(
    name = "push_image",
    format = "Docker",
    image = ":image",
    registry = "ghcr.io",
    repository = "ekhabarov/service-one",
    tag = "$(IMAGE_TAG)",
)

Besides already familiar to us target go_library, it contains several new.

go_binary (rules_go)

Docs

go_binary(
    name = "service-one",
    embed = [":service-one_lib"],
    goos = "linux",
    goarch = "amd64",
    visibility = ["//visibility:public"],
)

As you can guess by the name, it produces a binary, and this target is executable, which means we can run² a binary with the next command:

% bazel run //service-one

INFO: Analyzed target //service-one:service-one (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //service-one:service-one up-to-date:
  bazel-bin/service-one/service-one_/service-one
INFO: Elapsed time: 0.143s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

and query our service on port 5000 with

% grpcurl -plaintext -d '{"name": "Bazel"}' \
  127.0.0.1:5000 svc.ServiceOne.Hello
{
  "msg": "Hello, Bazel"
}

We can find binary itself in Bazel’s cache:

% bazel build //service-one
INFO: Analyzed target //service-one:service-one (1 packages loaded, 3 targets configured).
INFO: Found 1 target...
Target //service-one:service-one up-to-date:
  bazel-bin/service-one/service-one_/service-one      <<< HERE IS A BINARY
INFO: Elapsed time: 0.646s, Critical Path: 0.51s
INFO: 3 processes: 1 internal, 2 darwin-sandbox.
INFO: Build completed successfully, 3 total actions

% file -b bazel-bin/service-one/service-one_/service-one

ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2,
Go BuildID=redacted, not stripped

go_image (rules_docker)

I’ve deleted this target from my example, but it still worth to mention.

Docs

go_image(
    name = "image",
    embed = ["service-one_lib"],
    importpath = "github.com/ekhabarov/bazel-k8s-envoy/service-one",
)

This rule creates a Docker image. At the same time, it doesn’t require an installed Docker. Though, if it’s installed, with the following command, you can load a new image into local Docker:

% bazel run //service-one:image -- --norun

INFO: Analyzed target //service-one:image (0 packages loaded, 2 targets configured).
INFO: Found 1 target...
Target //service-one:image up-to-date:
  bazel-bin/service-one/image-layer.tar
INFO: Elapsed time: 0.138s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
7f01b59692db: Loading layer   11.7MB/11.7MB
Loaded image ID: sha256:b156681a45619ce4e922c6f8aa2a56353ba0d0bf7637acc6fa0d2c1fd405d01a
Tagging b156681a45619ce4e922c6f8aa2a56353ba0d0bf7637acc6fa0d2c1fd405d01a as bazel/service-one:image

Run, not run. --norun parameter here means building and loading the image but not running a binary itself.

Look at the embed parameter. It contains a link to go_library rule instead of go_binary. That’s done for cross-compilation support, the same way we did before, for go_binary target. See embedding for details.

container_image (rules_docker)

Docs

container_image(
    name = "image",
    base = "@go_base//image",
    entrypoint = ["/service-one"],
    env = {
        "ENV_VAR1": "value1",
        "ENV_VAR2": "value2",
    },
    files = [
        ":service-one",
    ],
    labels = {
        "version": "1.0.0",
    },
    ports = [
        "5000/tcp",
    ],
)

The idea of container_image is the same as the previous target - create an image but customize the created one. For instance, add config file into the image, if binary requires it, set environment variables, etc.

k8s_object

This rule have been replaced with Ytt tool

~~Docs~~

We will discuss later on when we will talk about the Kubernetes cluster setup.

See then next part.

Bazel Concepts and Terminology ↩︎
if you run this command on Linux machine. If you need to run a binary of a different OS, just add a target with an appropriate arch + os combination. Without goarch and goos defined, :image target complains about cgo toolchain. See issue #10134 for details. ↩︎

Outcome

Service One

Protobuf definition.

Implementation

It’s time to run go build, right?

BUILD files

Rules and targets

proto_library (rules_proto)

go_proto_library (rules_go)

go_library (rules_go)

proto_descriptor_set (rules_proto)

go_binary (rules_go)

go_image (rules_docker)

container_image (rules_docker)

k8s_object

See Also

It’s time to run `go build`, right?