Releasing Spring Boot Applications on Kubernetes (4)

11 minute read

The previous post in this series demonstrates how to deploy and run Spring Boot applications on K8s, using the mutual TLS authentication. According to the proposed scenario, the TLS hanshake is accepted or refused, depending wether the X509 certificate provided by the client is signed by a trusted authority or not. While this scenario is a great improvment of our use case, from the security point of view, , it still presented a drawback. The application itself has no idea who the caller is and every certificate signed by the trusted CA is equally, anonymously accepted.

In this 4th part we’re pushing authentication a step further. Here, Spring Security reads the client certificate that already satisfied the handshake, extracts its subject Common Name and maps it to a named principal with roles. The certificate stops being a mere door key and becomes the user’s identity. Endpoints are then authorized on that identity.

You can find the project in the same GitHub repository, on the mtls-security branch.

Identity-based authorization with Spring Security X.509

The mTLS handshake is unchanged. The property server.ssl.client-auth = need defined in the application.properties file still requires a CA-signed client certificate. What is new is a Spring Security filter chain declared in the class SecurityConfig, configured for X.509 authentication, as shown below:

@Configuration
@EnableWebSecurity
@EnableMethodSecurity(jsr250Enabled = true)
public class SecurityConfig
{
  @Bean
  public SecurityFilterChain filterChain(HttpSecurity http) throws Exception
  {
    return http
      .authorizeHttpRequests(auth -> auth.anyRequest().authenticated())
      .x509(x509 -> x509
      .subjectPrincipalRegex("CN=(.*?)(?:,|$)")
      .userDetailsService(userDetailsService()))
      .csrf(AbstractHttpConfigurer::disable)
      .build();
  }

  @Bean
  public UserDetailsService userDetailsService()
  {
    UserDetails admin = User.withUsername("sb-k8s-admin")
      .password("{noop}unused")
      .roles("ADMIN", "USER")
      .build();
    UserDetails user = User.withUsername("sb-k8s-user")
      .password("{noop}unused")
      .roles("USER")
      .build();
    return new InMemoryUserDetailsManager(admin, user);
  }
}

In the listing above the expression x509().subjectPrincipalRegex("CN=(.*?)(?:,|$)") pulls the CN out of the certificate subject as the principal name. The filter chain carries a single baseline rule, anyRequest().authenticated(), so every call is at least a certificate-resolved identity. Then a UserDetailsService resolves that name to roles. In our case, the principal CN=sb-k8s-admin is assigned the ROLE_ADMIN and the ROLE_USER roles while the principal CN=sb-k8s-user is assigned the ROLE_USER role.

The annotation @EnableMethodSecurity(jsr250Enabled = true) activates the JSR-250 annotations on the controller class K8sSbController:

@RestController
public class K8sSbController
{
  @RolesAllowed("USER")
  @GetMapping("/hello/{who}")
  public String sayHello(@PathVariable String who, Authentication authentication)
  {
    return "Hello %s, greeted by %s".formatted(who, authentication.getName());
  }

  @RolesAllowed("USER")
  @GetMapping("/whoami")
  public Map<String, Object> whoami(Authentication authentication)
  {
    return Map.of(
      "identity", authentication.getName(),
      "authorities", authentication.getAuthorities().stream()
      .map(GrantedAuthority::getAuthority)
      .toList());
  }

  @RolesAllowed("ADMIN")
  @GetMapping("/admin")
  public String admin(Authentication authentication)
  {
    return "Hello %s, you have elevated (admin) access".formatted(authentication.getName());
  }
}

Here the endpoint /admin is allowed for prinicpals having the ROLE_ADMIN role while the /whoami and /hello/{who} ones are allowed for prinicpals having either the ROLE_ADMIN or ROLE_USER roles.

The decisive consequence: a certificate that is CA-signed, such that so it clears the handshake, but whose CN is not in the UserDetailsService, is rejected at the authentication layer. Hence, CA trust alone is no longer enough.

What changed compared to the `mtls` branch

Artifact	Change
`pom.xml`	Adds `spring-boot-starter-security`; test-only `spring-security-test`/`webmvc-test` (unit tier) and `rest-assured` (e2e tier, via failsafe; Groovy pinned to 4.0.x, the line REST Assured expects).
`SecurityConfig.java`	New — the X.509 filter chain (baseline `anyRequest().authenticated()`), CN→principal regex, the in-memory identity registry and `@EnableMethodSecurity(jsr250Enabled = true)`.
`K8sSbController.java`	`/hello/{who}` now names the authenticated caller; adds `/whoami` (echoes the resolved identity + authorities) and an admin-only `/admin`. Roles are enforced with JSR-250 `@RolesAllowed`.
`k8s/client-certificate.yaml`	Now issues three client certificates with different CNs — `sb-k8s-admin`, `sb-k8s-user` and `sb-k8s-intruder` (CA-signed but not a known identity, for the e2e rejection case) — replacing the single `sb-k8s-client`.
`skaffold.yaml` / `redeploy.sh` / `start-all.sh`	The `reset` hook deletes the new secrets; the scripts extract the three client certs and build the PKCS12 keystores (`admin.p12`/`user.p12`/`intruder.p12` + `truststore.p12`) that `SecurityE2eIT` loads.

Testing identity-based authorization

Deploy and extract both identities (redeploy.sh does this automatically on this branch):

$ ./redeploy.sh mtls-security
$ kubectl port-forward svc/sb-k8s 8443:8443      # or: skaffold dev --port-forward

Confirm the two client certificates are issued:

$ kubectl get certificate
NAME           READY   SECRET              AGE
sb-k8s         True    sb-k8s-cert         1m
sb-k8s-ca      True    sb-k8s-ca           1m
sb-k8s-admin   True    sb-k8s-admin-cert   1m
sb-k8s-user    True    sb-k8s-user-cert    1m

Either identity can greet, and the response names the caller — the CN travelled all the way from the certificate to the controller:

$ curl --cacert ca.crt --cert user.crt --key user.key https://localhost:8443/hello/nicolas
Hello nicolas, greeted by sb-k8s-user

$ curl --cacert ca.crt --cert admin.crt --key admin.key https://localhost:8443/whoami
{"identity":"sb-k8s-admin","authorities":["ROLE_ADMIN","ROLE_USER"]}

Only the admin identity reaches the admin-only endpoint. The user certificate is a perfectly valid, CA-signed, authenticated identity — it simply lacks the role, so it is rejected with 403 Forbidden inside the application, not at the handshake:

$ curl --cacert ca.crt --cert admin.crt --key admin.key https://localhost:8443/admin
Hello sb-k8s-admin, you have elevated (admin) access

$ curl --cacert ca.crt --cert user.crt --key user.key https://localhost:8443/admin
{"status":403,"error":"Forbidden", ...}

That 403 is the whole point of this branch: two callers that are indistinguishable to plain mTLS (both hold a CA-signed certificate) are told apart by identity. Dropping the client certificate entirely still fails at the handshake exactly as on the mtls branch, since client-auth = need is unchanged.

Testing

The tests come in two tiers:

SecurityAuthorizationTest — unit (mvn test). A fast @WebMvcTest MockMvc slice with no cluster. @WithUserDetails("sb-k8s-admin"/"sb-k8s-user") loads the principal from the real UserDetailsService, so the CN→roles mapping and the @RolesAllowed rules are exercised; only the TLS/certificate handshake itself is out of scope.
SecurityE2eIT — end to end (mvn verify). A REST Assured test that hits the deployed app on https://localhost:8443 over a real mTLS handshake, presenting the actual cluster-issued certificates. It reuses the PKCS12 keystores that start-all.sh / redeploy.sh build from the extracted certs, so it covers the two cases the unit tier structurally cannot: a missing certificate (handshake refused) and a CA-signed certificate with an unknown CN — the sb-k8s-intruder cert — rejected by the app. It self-skips when no cluster is up, so mvn verify stays green on a bare checkout; to run it for real:
```
$ ./start-all.sh                              # or ./redeploy.sh mtls-security
$ kubectl port-forward svc/sb-k8s 8443:8443   # if not already forwarding
$ mvn verify
```

Certificate renewal and rotation

cert-manager owns the whole lifecycle of the server certificate, not just its first issuance. renewBefore on k8s/certificate.yaml says how long before expiry a replacement is issued. When that moment arrives cert-manager signs a new server certificate from ca-issuer, rebuilds the JKS keystore and truststore, and overwrites the sb-k8s-cert secret in place. The secret name never changes, only its contents.

The property that makes this branch interesting is its stable CA. The server leaf is signed by ca-issuer, which is backed by the long-lived sb-k8s-ca certificate (see Why a CA hierarchy is required). So while the server leaf rotates, ca.crt, as the CA the client pins with --cacert, does not change. That is exactly what lets a client keep trusting the connection across a rotation, and it is the difference from the self-signed cert-manager branch, where ca.crt rotates together with the leaf and this whole walkthrough is impossible.

Trying it end to end

The steps below are the runnable version of everything in this section, framed as two experiments:

without in-place reload the pod eventually breaks;
with it, the pod keeps working.

Here are the steps:

With the cluster running, as documented on the master branch, shorten the server certificate so it expires within the hour by adding duration: 1h to k8s/certificate.yaml (it already carries renewBefore: 5m):
```
spec:
  duration: 1h
  renewBefore: 5m
```
If you manually have activated the port forwarding then kill it. Then, redeploy by running redeploy.sh:
```
$ ./redeploy.sh mtls-security
```
Forward the port either manually, as shown below, or use skaffold dev --port-forward, which survives pod restarts:
```
$ kubectl port-forward svc/sb-k8s 8443:8443
```

Confirm that the curl request below works.

$ curl --cacert ca.crt --cert admin.crt --key admin.key https://localhost:8443/hello/nicolas
Hello nicolas, greeted by sb-k8s-admin

Experiment A — no reload-on-update: the pod breaks after an hour

Go to do something else and come back after an hour. cert-manager rotated the secret ~5 minutes before expiry, but the running app never reloaded it and is still serving the certificate that it have read at startup, which has now expired, so the same call fails:
```
$ curl --cacert ca.crt --cert admin.crt --key admin.key \
    https://localhost:8443/hello/nicolas
curl: (60) SSL certificate problem: certificate has expired
```

Experiment B — with reload-on-update: the pod keeps working

Enable in-place reload by adding to the bundle in k8s/configmap.yaml:
```
spring.ssl.bundle.jks.server.reload-on-update = true
```

Redeploy and forward again:

$ ./redeploy.sh mtls-security
$ kubectl port-forward svc/sb-k8s 8443:8443

Come back again after an hour and repeat the exact same call. This time the server reloaded the rotated certificate in place, and because ca.crt (the CA) never changed, the unchanged client command still succeeds:
```
$ curl --cacert ca.crt --cert admin.crt --key admin.key https://localhost:8443/hello/nicolas
Hello nicolas, greeted by sb-k8s-admin
```

The subsections below explain each moving part.

Watching it happen

By default, cert-manager issues a 90-day certificate, so nothing visibly rotates during a test session. To watch a full cycle on a human timescale, after having shortened the server certificate’s lifetime, as explained above, look at the serial inside the issued certificate:

$ kubectl get secret sb-k8s-cert -o jsonpath='{.data.tls\.crt}' \
    | base64 -d | openssl x509 -noout -serial -dates

You’ll se two timestamps:

notAfter is the expiry;
renewalTime is when cert-manager plans to rotate.

This serial is the cleanest rotation fingerprint. It changes on every renewal, while ca.crt stays the same.

Does the running application pick up the new certificate?

Rotating the secret is only half the story. The pod mounts keystore.jks and truststore.jks from that secret (see CERT_PATH in k8s/deployment.yaml), and two things stand between a rotated secret and a server that actually serves the new certificate.

First, kubelet refreshes the mounted files a short while after the secret changes (up to ~60–90 s), not instantly. Second, and more importantly, Spring Boot reads the keystore when it builds its SSL context. According to the property server.ssl.bundle = server in k8s/configmap.yaml, this project configures TLS through an SSL bundle, rather than the classic server.ssl.key-store, and that matters: SSL bundles can be reloaded without a restart.

Add the following property to the configmap.yaml:

spring.ssl.bundle.jks.server.reload-on-update = true

and Spring Boot watches the keystore and truststore files and rebuilds the SSL context in place when cert-manager rotates them, so the new certificate is served with no downtime. Without that property, the bundle is read once at startup and a rotated secret is ignored until the pod restarts:

$ kubectl rollout restart deployment/sb-k8s

To verify end-to-end whether the server presents the new certificate, compare the currently served against the old one. In order to do that, before rotating, with the port-forward still running, record the current serial and keep a copy of the leaf:

$ echo | openssl s_client -connect localhost:8443 2>/dev/null \
    | openssl x509 -noout -serial -enddate         # note the serial + notAfter
$ echo | openssl s_client -connect localhost:8443 2>/dev/null \
    | openssl x509 > old-cert.pem                  # keep the old leaf

Wait for renewBefore to fire and repeat. Read the wire again:

$ echo | openssl s_client -connect localhost:8443 2>/dev/null \
    | openssl x509 -noout -serial -enddate

The renewed certificate works when this handshake still succeeds and the serial has changed to a later notAfter. Confirm a real mTLS request still goes through with the same ca.crt and client certificate as before:

$ curl --cacert ca.crt --cert admin.crt --key admin.key \
    https://localhost:8443/hello/nicolas
Hello nicolas, greeted by sb-k8s-admin

To confirm that the old certificate is retired:

$ openssl x509 -in old-cert.pem -noout -checkend 0    # "is it expired right now?"
Certificate expired

$ openssl verify -CAfile ca.crt old-cert.pem          # validate old leaf against the CA
old-cert.pem: ... certificate has expired

As opposed to before, when the old one-hour certificate reaches its notAfter, both commands still report it valid. Now, once it expires they flip as shown above.

Watching an expired certificate get rejected

The checks above validate a saved file; the more convincing proof is a live handshake refused because the certificate on the wire has expired. That happens naturally when reload-on-update is not enabled: the server keeps serving the certificate it read at startup and ignores the rotated secret, so once that in-memory certificate passes its one-hour notAfter the running server is presenting an expired certificate. From then on the client rejects it:

$ curl --cacert ca.crt --cert admin.crt --key admin.key \
    https://localhost:8443/hello/nicolas
curl: (60) SSL certificate problem: certificate has expired

$ echo | openssl s_client -connect localhost:8443 2>/dev/null | grep -i verify
Verify return code: 10 (certificate has expired)

(The exact wording varies a little between curl/OpenSSL versions.) This is the concrete reason to enable reload-on-update or restart the pod on rotation: left alone, a long-running pod will eventually serve a stale, expired certificate and break TLS on its own — even though cert-manager rotated the secret an hour earlier.

The client certificates rotate too. sb-k8s-admin and sb-k8s-user renew on their own lifecycle, and because client-auth = need the server validates them against its truststore on every handshake. An expired client certificate is refused exactly like the missing one in The mTLS check — the mutual counterpart of the rejection above.

Switching branches and redeploying

This repository has more than one branch: cert-manager, mtls, mtls-security, …. Switching between them is a cluster operation. minikube and cert-manager stay up the whole time, only the application’s own resources are recreated. In particular there is no need to stop/restart minikube or to build by hand since skaffold run does it in one step.

The redeploy.sh helper automates the full sequence:

tear down:
optionally switch branch;
rebuild and redeploy;

re-extract ca.crt and, on mtls, the client certificate, or on mtls-security, the admin and user client certificates.

$ ./redeploy.sh               # redeploy the current branch
$ ./redeploy.sh mtls-security # switch to mtls-security, then redeploy
$ ./redeploy.sh mtls          # switch to mtls, then redeploy
$ ./redeploy.sh cert-manager  # switch to cert-manager, then redeploy

Under the hood it runs skaffold delete, git checkout <branch>, then skaffold run -p reset. The reset profile, defined in skaffold.yaml, deletes the cert-manager-generated secrets before deploying, so the new branch gets freshly issued certificates. This matters when switching branches because those secrets are not garbage-collected on their own and their contents differ between branches.

Two things the script takes care of that are easy to forget by hand:

the ca.crt changes between branches (different issuer), so it must be re-extracted;
a manual kubectl port-forward dies with the old pod and has to be restarted. Using skaffold dev --port-forward avoids that entirely.

Enjoy !

Twitter Facebook LinkedIn

Nicolas DUMINIL