Problem
Understanding the plugin protocol is essential for building providers in languages other than Go. The Terraform Plugin SDK abstracts away critical details that you need to know when implementing providers from scratch.
Most Terraform provider developers use the terraform-plugin-framework or terraform-plugin-sdk Go packages, which handle protocol implementation automatically. But if you want to build providers in other languages, you need to understand what’s happening under the hood.
Context
Terraform’s architecture separates the core engine from providers using a plugin system:
1
| terraform (core) <--> go-plugin <--> gRPC <--> provider binary
|
The provider binary must implement the Provider service defined in tfplugin6.proto:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| service Provider {
rpc GetProviderSchema(GetProviderSchema.Request) returns (GetProviderSchema.Response);
rpc ValidateProviderConfig(ValidateProviderConfig.Request) returns (ValidateProviderConfig.Response);
rpc ConfigureProvider(ConfigureProvider.Request) returns (ConfigureProvider.Response);
rpc ValidateResourceConfig(ValidateResourceConfig.Request) returns (ValidateResourceConfig.Response);
rpc ReadResource(ReadResource.Request) returns (ReadResource.Response);
rpc PlanResourceChange(PlanResourceChange.Request) returns (PlanResourceChange.Response);
rpc ApplyResourceChange(ApplyResourceChange.Request) returns (ApplyResourceChange.Response);
rpc ValidateDataResourceConfig(ValidateDataResourceConfig.Request) returns (ValidateDataResourceConfig.Response);
rpc ReadDataSource(ReadDataSource.Request) returns (ReadDataSource.Response);
rpc UpgradeResourceState(UpgradeResourceState.Request) returns (UpgradeResourceState.Response);
}
|
Key concepts the SDK handles for you:
- Schema marshaling: Converting Go structs to/from
tfprotov6.Schema with nested blocks and attributes - State encoding: Serializing resource state as MessagePack for storage in
.tfstate - Plan diffing: Computing diffs between prior state and proposed changes
- Unknown value propagation: Handling computed values that aren’t known until apply
- Diagnostics collection: Converting errors to structured diagnostic messages
Fix
When building a provider without the SDK, you must implement these concerns manually.
Schema Marshaling
Providers must return their schema in GetProviderSchema:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
| def GetProviderSchema(self, request, context):
return tfplugin6_pb2.GetProviderSchema.Response(
provider=tfplugin6_pb2.Schema(
version=1,
block=tfplugin6_pb2.Schema.Block(
version=1,
attributes=[
tfplugin6_pb2.Schema.Attribute(
name="api_key",
type=json.dumps({"type": "string"}), # Type must be JSON-encoded
required=True,
sensitive=True,
),
tfplugin6_pb2.Schema.Attribute(
name="region",
type=json.dumps({"type": "string"}),
optional=True,
),
],
),
),
resource_schemas={
"example_server": tfplugin6_pb2.Schema(
version=1,
block=tfplugin6_pb2.Schema.Block(
attributes=[
# ... resource attributes
],
),
),
},
)
|
State Encoding
Resource state is stored as MessagePack-encoded bytes:
1
2
3
4
5
6
7
8
9
10
11
| import msgpack
def encode_state(state: dict) -> bytes:
"""Encode resource state as MessagePack."""
return msgpack.packb(state, use_bin_type=True)
def decode_state(state_bytes: bytes) -> dict:
"""Decode resource state from MessagePack."""
if not state_bytes:
return {}
return msgpack.unpackb(state_bytes, raw=False)
|
Plan Diffing
Providers must compute the diff between prior state and proposed configuration:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| def PlanResourceChange(self, request, context):
prior_state = decode_state(request.prior_state)
proposed_new_state = decode_state(request.proposed_new_state)
# Compute which attributes changed
requires_replace = []
for attr in ["name", "size"]: # Immutable attributes
if prior_state.get(attr) != proposed_new_state.get(attr):
requires_replace.append(AttributePath(steps=[attr]))
# Return planned state
return tfplugin6_pb2.PlanResourceChange.Response(
planned_state=encode_state(proposed_new_state),
requires_replace=requires_replace,
)
|
Unknown Values
During planning, some values are not yet known (computed). These are represented as a special MessagePack value:
1
2
3
4
5
| UNKNOWN = msgpack.ExtType(code=0, data=b'')
def mark_computed(state: dict, attr: str):
"""Mark an attribute as computed (unknown until apply)."""
state[attr] = UNKNOWN
|
This tells Terraform that the value will be populated during ApplyResourceChange.
Apply Implementation
The apply step creates/updates/deletes the actual resource:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| def ApplyResourceChange(self, request, context):
prior_state = decode_state(request.prior_state)
planned_state = decode_state(request.planned_state)
diagnostics = []
if not planned_state: # Destroy
delete_resource(prior_state["id"])
return tfplugin6_pb2.ApplyResourceChange.Response(
new_state=b'', # Empty state = destroyed
)
elif not prior_state: # Create
resource = create_resource(planned_state)
else: # Update
resource = update_resource(prior_state["id"], planned_state)
# Populate computed values
new_state = {**planned_state}
new_state["id"] = resource.id
new_state["created_at"] = resource.created_at.isoformat()
return tfplugin6_pb2.ApplyResourceChange.Response(
new_state=encode_state(new_state),
diagnostics=diagnostics,
)
|
Understanding these internals allows you to build Terraform providers in any language that supports gRPC and MessagePack. The SDK is just a convenience layer on top of the protocol.
Result
You can now implement Terraform providers in Python, Rust, TypeScript, or any language with gRPC support. 😀
Comments