What are the causes of various Envoy RESPONSE_FLAGS

I'm trying to debug a mesh of services, with Envoy sat in the middle. The access logs are showing a lot of 503s, which vary in their %RESPONSE_FLAGS%. Other than the access logging docs, I can't find any explaination of what the flags actually mean.

Eg, pithily, NR means "I sent a 404 downstream, but not because upstream sent me one, but because I have no matching route, so I, envoy generated the 404". I'd love a one-liner like that for some of the other. Specially I'm struggling with (these all appear in our logs...)

• UR vs UC - I think these are "I sent a 503 downstream because I connected upstream but then there was a subsequent problem". What problem? Specifically, what do remote reset vs connection termination mean?
• LR connection local reset - what does this mean? Envoy sent a 503 downstream becuase it decided to stop talking to upstream, mid-way through? Why would it do that?
• <no flag> - am I right in assuming that if there's an error code like 404 or 503 and no RESPONSE_FLAGS, that that code was sent by upstream and is just being forwarded?

1 Answer

From the source code below,

• UR: UPSTREAM_REMOTE_RESET If a remote codec level reset was received on the stream.
• UC: UPSTREAM_CONNECTION_TERMINATION If the stream was locally reset due to connection termination.
• LR: If a local codec level reset was sent on the stream

/** * Stream reset reasons. */ enum class StreamResetReason { // If a local codec level reset was sent on the stream. LocalReset, // If a local codec level refused stream reset was sent on the stream (allowing for retry). LocalRefusedStreamReset, // If a remote codec level reset was received on the stream. RemoteReset, // If a remote codec level refused stream reset was received on the stream (allowing for retry). RemoteRefusedStreamReset, // If the stream was locally reset by a connection pool due to an initial connection failure. ConnectionFailure, // If the stream was locally reset due to connection termination. ConnectionTermination, // The stream was reset because of a resource overflow. Overflow, // Either there was an early TCP error for a CONNECT request or the peer reset with CONNECT_ERROR ConnectError, // Received payload did not conform to HTTP protocol. ProtocolError }; StreamInfo::ResponseFlag Filter::streamResetReasonToResponseFlag(Http::StreamResetReason reset_reason) { switch (reset_reason) { case Http::StreamResetReason::ConnectionFailure: return StreamInfo::ResponseFlag::UpstreamConnectionFailure; case Http::StreamResetReason::ConnectionTermination: return StreamInfo::ResponseFlag::UpstreamConnectionTermination; case Http::StreamResetReason::LocalReset: case Http::StreamResetReason::LocalRefusedStreamReset: return StreamInfo::ResponseFlag::LocalReset; case Http::StreamResetReason::Overflow: return StreamInfo::ResponseFlag::UpstreamOverflow; case Http::StreamResetReason::RemoteReset: case Http::StreamResetReason::RemoteRefusedStreamReset: case Http::StreamResetReason::ConnectError: return StreamInfo::ResponseFlag::UpstreamRemoteReset; case Http::StreamResetReason::ProtocolError: return StreamInfo::ResponseFlag::UpstreamProtocolError; } NOT_REACHED_GCOVR_EXCL_LINE; } constexpr static absl::string_view NONE = "-"; constexpr static absl::string_view DOWNSTREAM_CONNECTION_TERMINATION = "DC"; constexpr static absl::string_view FAILED_LOCAL_HEALTH_CHECK = "LH"; constexpr static absl::string_view NO_HEALTHY_UPSTREAM = "UH"; constexpr static absl::string_view UPSTREAM_REQUEST_TIMEOUT = "UT"; constexpr static absl::string_view LOCAL_RESET = "LR"; constexpr static absl::string_view UPSTREAM_REMOTE_RESET = "UR"; constexpr static absl::string_view UPSTREAM_CONNECTION_FAILURE = "UF"; constexpr static absl::string_view UPSTREAM_CONNECTION_TERMINATION = "UC"; constexpr static absl::string_view UPSTREAM_OVERFLOW = "UO"; constexpr static absl::string_view UPSTREAM_RETRY_LIMIT_EXCEEDED = "URX"; constexpr static absl::string_view NO_ROUTE_FOUND = "NR"; constexpr static absl::string_view DELAY_INJECTED = "DI"; constexpr static absl::string_view FAULT_INJECTED = "FI"; constexpr static absl::string_view RATE_LIMITED = "RL"; constexpr static absl::string_view UNAUTHORIZED_EXTERNAL_SERVICE = "UAEX"; constexpr static absl::string_view RATELIMIT_SERVICE_ERROR = "RLSE"; constexpr static absl::string_view STREAM_IDLE_TIMEOUT = "SI"; constexpr static absl::string_view INVALID_ENVOY_REQUEST_HEADERS = "IH"; constexpr static absl::string_view DOWNSTREAM_PROTOCOL_ERROR = "DPE"; constexpr static absl::string_view UPSTREAM_MAX_STREAM_DURATION_REACHED = "UMSDR"; constexpr static absl::string_view RESPONSE_FROM_CACHE_FILTER = "RFCF"; constexpr static absl::string_view NO_FILTER_CONFIG_FOUND = "NFCF"; constexpr static absl::string_view DURATION_TIMEOUT = "DT"; constexpr static absl::string_view UPSTREAM_PROTOCOL_ERROR = "UPE"; constexpr static absl::string_view NO_CLUSTER_FOUND = "NC"; static constexpr std::array ALL_RESPONSE_STRING_FLAGS{ FlagStringAndEnum{FAILED_LOCAL_HEALTH_CHECK, ResponseFlag::FailedLocalHealthCheck}, FlagStringAndEnum{NO_HEALTHY_UPSTREAM, ResponseFlag::NoHealthyUpstream}, FlagStringAndEnum{UPSTREAM_REQUEST_TIMEOUT, ResponseFlag::UpstreamRequestTimeout}, FlagStringAndEnum{LOCAL_RESET, ResponseFlag::LocalReset}, FlagStringAndEnum{UPSTREAM_REMOTE_RESET, ResponseFlag::UpstreamRemoteReset}, FlagStringAndEnum{UPSTREAM_CONNECTION_FAILURE, ResponseFlag::UpstreamConnectionFailure}, FlagStringAndEnum{UPSTREAM_CONNECTION_TERMINATION, ResponseFlag::UpstreamConnectionTermination}, FlagStringAndEnum{UPSTREAM_OVERFLOW, ResponseFlag::UpstreamOverflow}, FlagStringAndEnum{NO_ROUTE_FOUND, ResponseFlag::NoRouteFound}, FlagStringAndEnum{DELAY_INJECTED, ResponseFlag::DelayInjected}, FlagStringAndEnum{FAULT_INJECTED, ResponseFlag::FaultInjected}, FlagStringAndEnum{RATE_LIMITED, ResponseFlag::RateLimited}, FlagStringAndEnum{UNAUTHORIZED_EXTERNAL_SERVICE, ResponseFlag::UnauthorizedExternalService}, FlagStringAndEnum{RATELIMIT_SERVICE_ERROR, ResponseFlag::RateLimitServiceError}, FlagStringAndEnum{DOWNSTREAM_CONNECTION_TERMINATION, ResponseFlag::DownstreamConnectionTermination}, FlagStringAndEnum{UPSTREAM_RETRY_LIMIT_EXCEEDED, ResponseFlag::UpstreamRetryLimitExceeded}, FlagStringAndEnum{STREAM_IDLE_TIMEOUT, ResponseFlag::StreamIdleTimeout}, FlagStringAndEnum{INVALID_ENVOY_REQUEST_HEADERS, ResponseFlag::InvalidEnvoyRequestHeaders}, FlagStringAndEnum{DOWNSTREAM_PROTOCOL_ERROR, ResponseFlag::DownstreamProtocolError}, FlagStringAndEnum{UPSTREAM_MAX_STREAM_DURATION_REACHED, ResponseFlag::UpstreamMaxStreamDurationReached}, FlagStringAndEnum{RESPONSE_FROM_CACHE_FILTER, ResponseFlag::ResponseFromCacheFilter}, FlagStringAndEnum{NO_FILTER_CONFIG_FOUND, ResponseFlag::NoFilterConfigFound}, FlagStringAndEnum{DURATION_TIMEOUT, ResponseFlag::DurationTimeout}, FlagStringAndEnum{UPSTREAM_PROTOCOL_ERROR, ResponseFlag::UpstreamProtocolError}, FlagStringAndEnum{NO_CLUSTER_FOUND, ResponseFlag::NoClusterFound}, };