Access Dapr Application with Ingress Controller

In the previous article, we shared the use of the distributed runtime Dapr. In the example, the state storage capability was separated into the Dapr runtime, and the application used this capability through the Dapr API. This article will introduce how to access Dapr applications through an Ingress Controller.

Solution

There are two solutions to expose access to Dapr applications:

1. As tranditional usage, configure the Service of the application as the backend, and the ingress controller directly forwards traffic to the application container. In other words, this acts as an automatically configured L7 load balancer.
2. Instead of accessing the application container directly, access it through the Daprd runtime. In this case, we need to declare the ingress controller as a Dapr application and inject the Daprd runtime container into it. The backend pointed to by the ingress rule is the Dapr Service of the Ingress.

Both methods have their pros and cons. The first method has a simple architecture, while the second method, although it introduces the Daprd container and has a more complex architecture that consumes more resources, can use Dapr’s service governance capabilities, such as timeouts, retries, access control, and so on.

Next, we will use examples to verify both methods separately.

Demonstration

Prerequistes

• Helm
• Dapr CLI

Setup Kubernetes cluster

We will use the Kubernetes distribution k3s.

export INSTALL_K3S_VERSION=v1.23.8+k3s2
curl -sfL https://get.k3s.io | sh -s - --disable traefik --disable servicelb --write-kubeconfig-mode 644 --write-kubeconfig ~/.kube/config

Install Dapr

Execute command bellow to install Dapr in cluster.

dapr init --kubernetes --wait

Install Ingress Controller

Here, let’s use the Flomesh Ingress Controller(referred as FSM below). We add Dapr-related annotations to the Ingress Controller through values.yaml. It is important to note that by default, the Daprd runtime listens on a loopback address, while Ingress uses Pod IP to forward requests to the backend. Therefore, we need to annotate dapr.io/sidecar-listen-addresses: "[::],0.0.0.0" to allow the Daprd runtime to receive requests from the Pod IP.

FSM(Flomesh Service Mesh ) is Kubernetes North-South traffic manager, provides Ingress controllers, Gateway API, Load Balancer, and cross-cluster service registration and service discovery. FSM uses Pipy as data plane and suitable for cloud, edge and IoT.

description from FSM GitHub repo.

# values.yaml
fsm:
  ingress:
    podAnnotations:
      dapr.io/sidecar-listen-addresses: "[::],0.0.0.0"
      dapr.io/enabled: "true"
      dapr.io/app-id: "ingress"
      dapr.io/app-port: "8000"
      dapr.io/enable-api-logging: "true"
      dapr.io/config: "ingressconfig"

Execute command to install FSM via helm (here I choose the latest alpha version). Don’t forget to specify the values.yaml. The FSM will be deployed in namespace Flomesh.

helm repo add fsm https://charts.flomesh.io
helm repo update

helm install \
  --namespace flomesh \
  --create-namespace \
  --version=0.2.1-alpha.3 \
  -f values.yaml \
  fsm fsm/fsm

Deploy sample application

In order to demonstrate the access with Ingress Controller, we need a sample application. Here we will take the kennethreitz/httpbin as our sample app. Don't forget to decare it as a Dapr application by annotating it with Dapr annotations .

kubectl create ns httpbin

kubectl apply -n httpbin -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: httpbin
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  strategy: {}
  template:
    metadata:
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "httpbin"
        dapr.io/app-port: "80"
        dapr.io/enable-api-logging: "true"
        dapr.io/config: "httpbinconfig"
      labels:
        app: httpbin
    spec:
      containers:
      - image: kennethreitz/httpbin
        name: httpbin
        resources: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: httpbin
  name: httpbin
  namespace: httpbin
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: httpbin
EOF

Solution 1

Refer to the FSM Ingress doc to create Ingress resource using Service httpbin as backend.

kubectl apply -n httpbin -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  annotations:
    pipy.ingress.kubernetes.io/rewrite-target-from: ^/httpbin/?
    pipy.ingress.kubernetes.io/rewrite-target-to: /
spec:
  ingressClassName: pipy
  rules:
  - http:
      paths:
      - backend:
          service:
            name: httpbin
            port:
              number: 80
        path: /httpbin
        pathType: Prefix
EOF

Access path /httpbin/get with the host IP address and the 80 port of the ingress controller. Due to the path rewriting configured above, the final request /get will be sent to the target Pod.

Yes, it works!

curl HOST_IP:80/httpbin/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Content-Length": "0",
    "Host": "10.0.0.13:80",
    "User-Agent": "curl/7.86.0"
  },
  "origin": "10.42.0.18",
  "url": "http://10.0.0.13:80/get"
}

Solution 2

Solution 2 also requires to configure the ingress rules, but this time the backend service configuration is the Dapr Service of ingress controller , through kubectl get svc -n flomesh, you can find Service with the suffix of -dapr.

kubectl apply -n flomesh -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dapr
  annotations:
    pipy.ingress.kubernetes.io/rewrite-target-from: ^/dapr/?
    pipy.ingress.kubernetes.io/rewrite-target-to: /
spec:
  ingressClassName: pipy
  rules:
  - http:
      paths:
      - backend:
          service:
            name: ingress-dapr
            port:
              number: 80
        path: /dapr
        pathType: Prefix
EOF

It is still the path /get to access httpbin, but the access method should follow Dpar Service Invocation API: Specify dapr-app-id as httpbin.httpbin through the request header. Since the ingress controller and the target application are not in the same namespace, the application id needs to include its namespace.

It also returns successfully, but you can see that the final application will receive some more information in the request header, which is from the Daprd runtime, such as Dapr-Caller-App-Id, "Dapr-Callee-App- Id identifies the caller and callee of the request; Forwarded identifies the host name that initiated the request. If tracing is enabled, there will also be link-related information (not enabled in this example).

curl HOST_IP:80/dapr/get -H 'dapr-app-id:httpbin.httpbin'
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip",
    "Connection": "keep-alive",
    "Dapr-App-Id": "httpbin.httpbin",
    "Dapr-Callee-App-Id": "httpbin",
    "Dapr-Caller-App-Id": "ingress",
    "Forwarded": "for=10.42.0.18;by=10.42.0.18;host=fsm-ingress-pipy-864d8d9c76-4kb7r",
    "Host": "127.0.0.1:80",
    "Proto": "HTTP/1.1",
    "Protomajor": "1",
    "Protominor": "1",
    "Referer": "",
    "Traceparent": "00-00000000000000000000000000000000-0000000000000000-00",
    "User-Agent": "curl/7.86.0",
    "X-Forwarded-Host": "fsm-ingress-pipy-864d8d9c76-4kb7r"
  },
  "origin": "10.42.0.18",
  "url": "http://fsm-ingress-pipy-864d8d9c76-4kb7r/get"
}

Access control with solution 2

At the beginning of the article, it is mentioned that using Solution 2 brings delay and increases complexity, but service governance capabilities are provided by Dapr. Here we take Dapr’s access control function as an example.

I don’t know if you have noticed, when deploying the entry controller and the sample application, there is an annotation dapr.io/config: xxx. This annotation is to specify the configuration for the Daprd runtime of the application. When the runtime starts, it will obtain the configuration from the Configuration resource named xxx.

Execute the following command to set access control rules for the httpbin application: deny all requests by default, and only allow the application with the id ingress under the flomesh namespace to access the path /get through GET. For more access control configurations, please refer to Dapr Access Control Documentation.

kubectl apply -n httpbin -f - <<EOF
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: httpbinconfig
spec:
  accessControl:
    defaultAction: deny
    trustDomain: "public"
    policies:
    - appId: ingress
      defaultAction: deny
      trustDomain: 'public'
      namespace: "flomesh"
      operations:
      - name: /get
        httpVerb: ['GET']
        action: allow
EOF

Once the configuration applied, we need to restart application to reload configuration.

kubectl rollout restart deploy httpbin -n httpbin

After application restarted, access the path ‘/get’ and it respond successfully.

curl HOST_IP:80/dapr/get -H 'dapr-app-id:httpbin.httpbin'
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip",
    "Connection": "keep-alive",
    "Dapr-App-Id": "httpbin.httpbin",
    "Dapr-Callee-App-Id": "httpbin",
    "Dapr-Caller-App-Id": "ingress",
    "Forwarded": "for=10.42.0.40;by=10.42.0.40;host=fsm-ingress-pipy-864d8d9c76-jctcx",
    "Host": "127.0.0.1:80",
    "Proto": "HTTP/1.1",
    "Protomajor": "1",
    "Protominor": "1",
    "Referer": "",
    "Traceparent": "00-00000000000000000000000000000000-0000000000000000-00",
    "User-Agent": "curl/7.86.0",
    "X-Forwarded-Host": "fsm-ingress-pipy-864d8d9c76-jctcx"
  },
  "origin": "10.42.0.40",
  "url": "http://fsm-ingress-pipy-864d8d9c76-jctcx/get"
}

But when trying to access path /headers, you will get the response like below. This means the acess is denied.

curl HOST_IP:80/dapr/headers -H 'dapr-app-id:httpbin.httpbin'
{
  "errorCode": "ERR_DIRECT_INVOKE",
  "message": "fail to invoke, id: httpbin.httpbin, err: rpc error: code = PermissionDenied desc = access control policy has denied access to appid: ingress operation: headers verb: GET"
}

Summarize

The two ingress solutions introduced in this article have their own advantages and disadvantages. Solution 1 has a simple structure and is also our commonly used approach; solution 2 is equivalent to declaring the entrance controller as a Dapr application, which actually exposes the Dapr API, which is more convenient in implementation. In fact, it acts as a global application runtime for all external access.