How to Use SIGTERM Signals in Unix-based and Cloud-based Systems | by Pavel Durov | Nov, 2022

Graceful Termination in K8S

photo by Grotica Developer Marketing Agency Feather unsplash

In this article, we will cover UNIX Process signals and in particular the `SIGTERM` signal. We’ll cover how to handle them with practical examples using Node. typescript, postal workerAnd kind local cluster.

UNIX Signals and SIGTERM

A Unix-based operating system (OS) consists of multiple processes. uses os software comes in between (aka signals) As a way of communicating with running processes, these signals are signals indicating that some sort of event has occurred and they can vary in their intent and purpose.

Here is the list of available signals on my machine:

$ kill -l
1) SIGHUP  2) SIGINT  3) SIGQUIT  4) SIGILL
5) SIGTRAP  6) SIGABRT  7) SIGEMT  8) SIGFPE
9) SIGKILL 10) SIGBUS 11) SIGSEGV 12) SIGSYS
13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGURG
17) SIGSTOP 18) SIGTSTP 19) SIGCONT 20) SIGCHLD
21) SIGTTIN 22) SIGTTOU 23) SIGIO 24) SIGXCPU
25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH
29) SIGINFO 30) SIGUSR1 31) SIGUSR2

there are some, but we’re going to focus on those SIGTERM,

SIGTERM Signals are a way for the operating system to gracefully terminate a program. By graceful, we mean that the program is given time to perform a final cleanup before shutdown. Depending on the application, cleaning tasks may vary. Interestingly enough, Unix processes can block and ignore SIGTERM, But if we want quality process/service then we have to handle these signals with respect, otherwise our process will be force-closed.

Unix process as HTTP server

For demonstration, we are going to create a sample HTTP server using typescriptAnd Hapi,

let’s make npm Project and follow the hint:

$ npm init
…

Install dependencies:

$ npm install @hapi/hapi typesript @types/node @types/node @types/hapi__hapi

We’re not going to bother with the separation of development and production dependencies here.

create a local file and call it index.ts,

import  Server  from "@hapi/hapi";
function sleep(ms: number, value: string) 
return new Promise((resolve) => setTimeout(() => resolve(value), ms));

export async function startServer(host: string, port: number): Promise 
const server = new Server( port, host );
server.route(
method: "GET",
path: "/work",
handler: async () => sleep(10 * 1000, `done something for 10 seconds\n`),
);
await server.start();
console.log(`Server running at: $server.info.uri, PID: $process.pid`);
return server;

startServer("0.0.0.0", 3000);

Here, we start the local server on host 0.0.0.0 and port 3000. We have also configured a single endpoint, GET /work This simulates something that takes time to calculate – 10 seconds in our case. In real situation there may be time required to perform any kind of database query.

I love node, it’s so cool that in 27 lines of code you can define a server, you actually need less, because I added the sleep function etc., but yeah, it’s great.

Let’s run our server:


$ ./node_modules/.bin/ts-node ./index.ts
Server running at: http://0.0.0.0:3000, PID: 16544

All good We have a server running!

send now GET /work HTTP request to our endpoint. Pick any networking tool you like, I’m going to use curl,


$ curl http://0.0.0.0:3000/work
done something for 10 seconds

So far, so good. But what happens to that HTTP request if the server has suddenly timed out, expired, or in other words, isn’t there anymore? What feedback will we get from the client? Will we get anything? What will happen if the request is served after the server is dead? Many questions! Let’s try it:

Send another request to the server:

$ curl http://0.0.0.0:3000/work
done something for 10 seconds

But this time, while the server is doing simulated work for 10 seconds — let’s KILL This

$ kill -15 19346

Note: I am using Process ID (PID) which I got from index.ts output! Your local PID we’ll do something else for sure! If it’s not, it’s destiny and of course send me an email.

One may wonder, why not simply terminate the server’s shell process by pressing CTRL+C keys? ok he will send one SIGINT hint, but we want SIGTERM,

OK, so what happened to our client connections when the server was down? This is what happens:

$ curl http://0.0.0.0:3000/work
curl: (52) Empty reply from server

It got an empty reply, meaning it didn’t get any information, nada. let’s add -V (stands for verbose) flag our CURL command to see more information.

$ curl -v http://0.0.0.0:3000/work
* Trying 0.0.0.0:3000…
* Connected to 0.0.0.0 (127.0.0.1) port 3000 (#0)
> GET /work HTTP/1.1
> Host: 0.0.0.0:3000
> User-Agent: curl/7.84.0
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

It looks like the server closed the connection abruptly 👀. This is not good. What if this happens to the server client in production? If you’re running your application in one of the cloud orchestration tools, then shutting down containers might not be too out of the ordinary.

i am using Kubernetes (K8S), so I’m going to talk about that. K8S is like an OS for the cloud. K8S can end running Pod (aka containers) Voluntarily, after all, the whole purpose of K8S is to orchestrate distributed systems. If one of the pods is requesting too many resources or if the application is being minimized, the container may receive a signal from the K8S that it is time to go.

So we have the notion of pointers to gracefully terminate our containers.

to beautiful ending

In the previous section, we talked about how our processes or containers can receive different signals.

When K8S needs to terminate a pod, it will send SIGTERM, This way our service is not only getting cut off from resources, but it will have some time to do finalization tasks.

Let’s implement this logic in our node server:

import  Server  from "@hapi/hapi";
function sleep(ms: number, value: string) 
return new Promise((resolve) => setTimeout(() => resolve(value), ms));

export async function startServer(host: string, port: number): Promise 
const server = new Server( port, host );
server.route(
method: "GET",
path: "/work",
handler: async () => sleep(10 * 1000, `done something for 10 seconds\n`),
);
process.on("SIGTERM", async function () 
console.log(`Received SIGTERM`);
await server.stop( timeout: 10 * 1000 );
console.log(`Server stopped.`);
process.exit(0);
);
await server.start();
console.log(`Server running at: $server.info.uri, PID: $process.pid`);
return server;

startServer("0.0.0.0", 3000);

we’ve added a listener to SIGTERM Competition. When these events happen, we are stopping the server, but we are not just killing it. between that time SIGTERM arrives and the specified `timeout` parameter, our server will refuse to accept any new requests and will finalize ongoing requests.

We can test that statement! Restart server:

$ ts-node ./index.ts
Server running at: http://0.0.0.0:3000, PID: 67116

run CURL Make request and terminate server:

$ curl -v http://0.0.0.0:3000/work

We should see that the request is finished and a response is sent back to the client – no more “(52) Empty reply from server” errors.

If we try to access the server between the time limits SIGTERM And the actual shutdown we’ll get:

$ curl -v http://0.0.0.0:3000/work
* Trying 0.0.0.0:3000…
* connect to 0.0.0.0 port 3000 failed: Connection refused
* Failed to connect to 0.0.0.0 port 3000 after 3 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 0.0.0.0 port 3000 after 3 ms: Connection refused

If there is no server we will get the same error!

But that’s all we need to do for graceful termination of node processes!

Sending SIGTERM to K8S pods

As we mentioned, whenever a K8S pod is terminated, it is sent SIGTERM Signal.

the way we used to kill command for local processes, we can end up using pod delete command:

$ kubectl delete pod my-pod-qgldf

When K8S decides to terminate the pod for any reason, SIGTERM The signal will be sent to it, then to the Docker container, and finally to the running process.

You don’t have to believe me, just give it a go.

K8S Sample Application

We’re going to reuse the server code, but containerize it and deploy it to a local K8S cluster.

Our Dockerfile:

FROM node:19-bullseye
WORKDIR /app
COPY index.ts package.json package-lock.json /app/
RUN npm install
EXPOSE 3000
ENTRYPOINT [“/app/node_modules/.bin/ts-node”, “index.ts”]

Create Docker container:

$ docker build -t poc -f ./Dockerfile .

First, let’s run it and see if we can get SIGTERM Related log:

$ docker run -t poc:latest

Get Docker ID:

$ docker ps
CONTAINER ID IMAGE 
86b0a46730ba poc:latest

and stop it:

$ docker stop 86b0a46730ba

We should see the docker run command ending as well as our docker container with the same output as we had with the process.

$ docker run -t poc:latest
Server running at: http://0.0.0.0:3000, PID: 1
Received SIGTERM
Server stopped.

Now let’s do some K8S!

First, we need to load our container image into the cluster:

$ kind load docker-image poc:latest — name my-cluster

K8S deployment manifest:

apiVersion: v1
kind: Namespace
metadata:
name: poc-namespace
— -
apiVersion: apps/v1
kind: Deployment
metadata:
name: poc-deployment
namespace: poc-namespace
spec:
selector:
matchLabels:
app: poc-app
template:
metadata:
labels:
app: poc-app
spec:
containers:
— name: poc
image: poc:latest
ports:
— containerPort: 3000

Deploy to our cluster:

$ kubectl apply -f ./deployment.yaml 
namespace/poc-namespace created
deployment.apps/poc-deployment created

If you stream pod logs:

$ k logs -f poc-deployment-bf749f576-wfmv9

and then delete pod:

$ kubectl delete pod poc-deployment-bf749f576-wfmv9

Should get the same result:

$ k logs -f poc-deployment-bf749f576-wfmv9
Server running at: http://0.0.0.0:3000, PID: 1
Received SIGTERM
Server stopped.

and that’s a wrap!

In this article, we covered SIGTERM Shows the practical applications of signals, how they are used in Unix-based and cloud-based systems as well as handling these signals and their potential impact.

SIGTERM need to handle horizontally scalable cloud application. This allows the application to scale up and down on demand without affecting the stability of the client.

Post Views: 20

Leave a Reply