Lecture 15 – Implementing processes and messages (Digital Systems)

Copyright © 2024 J. M. Spivey
Jump to navigation Jump to search

Context switching as implemented in the previous lecture gives us a platform on which we can implement a process scheduler with no further magic: the scheduler is simply a program that receives calls to send, receive and other, similar, functions and decides the order in which processes should run. In this lecture, we look at the data structures and the algorithms that are used to make these decisions.

Process table

micro:bian keeps information about each process in a structure (source).

typedef struct _proc *proc;

struct _proc {
    int pid;                 /* Process id */
    char name[16];           /* Name for debugging */
    unsigned state;          /* SENDING, RECEIVING, etc. */
    unsigned *sp;            /* Saved stack pointer */
    void *stack;             /* Stack area */
    unsigned stksize;        /* Stack size (bytes) */
    int priority;            /* Priority: 0 is highest */
    proc waiting;            /* Processes waiting to send */
    int pending;             /* Whether INTERRUPT message pending */
    int msgtype;             /* Message type to send or receive */
    message *message;        /* Pointer to message buffer */

    proc next;               /* Next process in ready or send queue */

/* Possible state values */
#define DEAD 0
#define ACTIVE 1
#define SENDING 2
#define RECEIVING 3
#define SENDREC 4
#define IDLING 5

The gist of this is that each process will be represented by a structure containing certain fixed fields. Space for the structure will be allocated when a process is created, not here. Two of the fields, waiting and next, are pointers to other processes, and by means of them we can construct linked lists of process records, without having to use malloc or its equivalent to allocate storage dynamically: that's good, because dynamic storage allocation is difficult to do well in limited memory, and potentially too slow to incorporate in the heart of our operating system.

Each process has

  • an integer process id.
  • a string name, copied from the string given when the process was created. This is used in a function that dumps the state of all processes (a counterpart to the ps command of unix). Also the name of the current process could be printed in case of a panic, just before the Seven Stars of Death appear.
  • an integer state that has one of the values listed later.
    • DEAD indicates a process that has exited. It is also the value present in slots of the table that have never been used.
    • ACTIVE indicates a process that is ready to run. The process is either the current process, or the ready queue waiting to become current.
    • SENDING indicates that the process is waiting to send to another; the process will appear in the sending queue of the destination.
    • RECEIVING indicates that the process is waiting to receive a message; the accept field will specify who may send to it.
    • SENDREC indicates a call to sendrec, combining the functions of send and receive.
    • IDLING indicates the idle process, which runs when no other process is ready.
  • three pieces of information about the stack of the process, which contains all its state: the current stack pointer sp, the base of the stack area stack, and its size size. The base and size of the stack are used in the process dump, which attempts to find out how much of the allocated space has actually been used. The stack pointer is of more vital importance, because it is needed in order to resume the process after a system call.
  • a small integer priority for the process. Possible values are from 0 to 3, with 0 (the most urgent) reserved for processes that are connected to interrupts, 1 and 2 (with symbolic names P_HIGH and P_LOW allowed for normal processes, and 3 reserved for the idle process, which runs only when no other process is ready.
  • some fields that are used to save information about processes that are waiting to send or receive.
    • if waiting is not null, it points to a list of processes waiting to send to this one.
    • pending is non-zero if an interrupt message is waiting to be delivered to this process.
    • if the process is in state RECEIVING, then msgtype determines what types of message the process is willing to receive: either a specific message type, or the special value ANY.
    • if the process is in state SENDING or RECEIVING (or SENDREC), then message is a pointer to the message buffer passed in the call to send or receive (or sendrec). This buffer may be in the stack space of the process, and may be NULL@ if there is no additional data to pass as part of the message.
  • a pointer next that can be used to link the process into a queue, contining either processes waiting to run, or processes waiting to send a message.

Ready queue

The ready queue

There are three queue structures, corresponding to priorities 0, 1, and 2. There's no need for a queue structure for priority level 3, because that level always contains exactly one process, the idle process, which is always ready to run (source).

Each queue structure has two pointers: one to the head of the queue and another to the tail. If the queue is empty, then both of these pointers are null, and it the queue contains one process, then both head and tail point to it. In each case, it's quick to add a process to the end of the queue, or to remove a process from the front.

In order to find the ready process with the highest priority, we need to look at each queue structure in turn, but that doesn't take long if there are only a few different priority levels. With a more elaborate system of priorities, a more carefully designed data structure might be needed; but the number of processes might have to grow quite large before clever data structures gained any advantage, even if the running time of operations on them was asymptotically better.

All the processes that are on a ready queue are in the ACTIVE state.

Processes queueing to send

There are other queues for processes that are waiting to send a message and are in the SENDING state. In fact, each process has two pointers in its record: one called next for linking it into a queue, and another called waiting that points to the head of a queue of processes waiting to send to it. Because these queues are usually short, we don't bother with a pointer to the tail of the queue, but just find it when we want by chaining along the list.

In the picture, there is an ACTIVE process that is busy doing something, and three processes are waiting to send it a message when it has finished. One of these processes itself has a third process waiting to send to it: for that to happen, the first process must accept the message from the second process, allowing the second process to proceed, and the second process must then call receive to allow the third process to deliver its message.

In the diagram, there is also a process that is in RECEIVING state, waiting to receive a message. It is not linked to any sender, but potential senders know its process id, and can get in touch when they are ready to send. In addition, there is a process that is in RECEIVING state but nevertheless has another process waiting to send to it. That process must have asked to receive a message with a specific type, and the process that is waiting wants to send a different type of message. The receiving process will have to receive first the message it is waiting for; then perhaps it will call receive again and accept the message from the waiting process.

As the diagram shows, any process that is not DEAD (or the idle process), and is not currently waiting to receive from ANY, can have a queue of other processes waiting to send. Each waiting process is in SENDING state (so is not on the ready queue), and can be on the queue of only one process – the one it is hoping to send to. Therefore, we can manage with only one next link for each process, and we don't have to allocate storage dynamically to cope with any eventuality that might arise.

Not shown here

There are a couple of possibilities not shown in these diagrams: they add only details to the story, so do not need to be included in a first telling.

As well as send and receive, there is an operation sendrec(dst, type, msg) that combines both, equivalent to send(dst, type, msg) followed by receive(REPLY, msg). This single operation neatly captures the idiom of sending a request to a server and waiting for a reply. Besides the brevity that it offers, it is also a bit more efficient, and avoids a potential problem known as priority inversion. Supporting sendrec means adding another process state, called SENDREC. A process in this state is waiting to send to some destination, and will be on the destination's queue. When it has succeeded in sending, its state becomes RECEIVING, just as a process in state SENDING will enter state ACTIVE at the same moment.

As well as receiving a message from a process that is waiting in its queue, a driver process that calls receive can get an INTERRUPT message from the fictitious process HARDWARE. Such messages are not reflected in the queue of senders but by setting the pending flag for the process. If an interrupt is pending, then it jumps the queue and is delivered before any message from a waiting process.

Send and receive

When a process calls send, there are two possibilities.

  • If the destination process is in state RECEIVING, and it is prepared to accept a message of this type, then the message can be copied immediately, and both the sender and the receiver are now ready to run.
  • If the destination process is not waiting for a message of this type, then the sender must be suspended and join the destination's queue (at the end, it being a queue). Since the sender cannot continue, there must be a call to choose_proc to find another process to run.

When a process calls receive, three possibilities must be considered.

  • If the process is connected to an interrupt, and an interrupt is pending, then an INTERRUPT message should be delivered, and the process continues in order to service it.
  • If not, and some other process is waiting to send to this one, then the message can perhaps be delivered immediately. If the process is hoping to receive a specific type of message, we must search the queue to see if a process is waiting to send a message of the right type; otherwise, the first process on the queue is taken. After the message is copied, both sender and receiver are ready to run. The need to search the queue here means that we must accept that some operations on sender queues will take time proportional to the length of the queue, and partially justifies traversing the queue also to add a process to the end.
  • If there is not acceptable process waiting, then the caller must be suspended in state RECEVIING.

Both send (implemented by mini_send) and receive (implemented by mini_receive) are system calls that are handled by the operating system after it as been entered by the system call mechanism. System calls and all interrupts have the same priority in micro:bian, so we can be sure that the operations on queues needed to implement the system calls will not be interrupted: this is much simpler than generally allowing interrupts to system calls and disabling interrupts when necessary, but it means that interrupts will occasionally be disabled for a relatively long time.

System calls arrive in the operating system via the interrupt mechanism at the function system_call (source), which decodes the arguments of the call by delving into the exception frame. Going into more detail, we can look at a function mini_send (source) that implements the send operation and is invoked from the system call interface.

Note: Update needed for changed micro:bian API.

/* mini_send -- send a message */
static void mini_send(int dest, int type, message *msg)
    int src = os_current->pid;
    proc pdest = os_ptable[dest];

    if (dest < 0 || dest >= os_nprocs || pdest->state == DEAD)
        panic("Sending to a non-existent process %d", dest);

The function gets the destination in the form of a process number: the process table is entirely private to the scheduler, and pointers to process records are never passed to the client program. The implementation of send begins by finding the process id src of the current process, which is sending the message, and also for brevity a pointer pdest to the process record for the destination.

What happens next depends on whether the destination process is waiting to receive a message with this type.

    if (accept(pdest, type)) {
        /* Receiver is waiting: deliver the message and run receiver */
        deliver(pdest->message, src, type, msg);

If so, then we can immediately copy across the message and fill in src as the sender field and type as the type field of the message: this is done by deliver(), which takes into account the possibility that pdest->message or msg or both may be NULL. Filling in the sender automatically is a matter of convenience for us, but in Minix it is part of the security system, ensuring against forgery. Finally we call make_ready() on both the destination and the current (source) processes to add them to a queue, each according to its priority. This is the simplest way of ensuring that the next process is chosen in a way that satisfies the scheduling policy – but note that the destination goes on the queue first, since it is the process most likely to have real work to do.

    else {
        /* Sender must wait by joining the receiver's queue */
        set_state(os_current, SENDING, type, msg);

If the destination is not waiting for the sender's message, then the sender must block until the destination can accept the message. We record the message type and the address of the message buffer as part of the sender's state, set its status to SENDING, and add the sender to the queue of processes associated with the receiver.


Finally, whichever case applied, we need to choose a new process to run. If the message has been delivered, then it's more than likely the destination process will be chosen; otherwise the source is blocked, and another process must run, perhaps the destination, perhaps some other process.

Here's a simple version of mini_receive, which implements the receive system call. It first checks whether an INTERRUPT message is pending, then searches the queue of waiting senders to see if any of them have a message of the right type. If neither of these works, then the receiving process must block and wait for a sender or interrupt to appear.

/* mini_receive -- receive a message */
static void mini_receive(int type, message *msg)
    /* First see if an interrupt is pending */
    if (os_current->pending && (type == ANY || type == INTERRUPT)) {
        os_current->pending = 0;
        deliver(msg, HARDWARE, INTERRUPT, NULL);

    /* Now see if a sender is waiting */
    if (type != INTERRUPT) {
        proc psrc = find_sender(os_current, type);

        if (psrc != NULL) {
            deliver(msg, psrc->pid, psrc->msgtype, psrc->message);

    /* No luck: we must wait. */
    set_state(os_current, RECEIVING, type, msg);

(This version of receive doesn't take into account that a message might be sent with sendrec.)

In describing the process of sending and receiving messages, we've used various simple subroutines like accept (which tests if a message is acceptable), make_ready (which puts a process on the right ready queue), and enqueue (which puts the current process on the sneder queue of another process). All these subroutines are easy to implement using the representation of processes, including the linked lists formed by the waiting and next pointers – see the source code for details. There's no need to worry about the time taken by subroutine calls, because GCC is able to substitute the subroutine bodies inline.

As an example, here's an implementation of enqueue(pdest), which hangs the current process on the end of the queue pdest->waiting of processes waiting to send to process pdest.

/* enqueue -- add current process to a receiver's queue */
static inline void enqueue(proc pdest)
    os_current->next = NULL;
    if (pdest->waiting == NULL)
        pdest->waiting = os_current;
    else {
        proc r = pdest->waiting;
        while (r->next != NULL)
            r = r->next;
        r->next = os_current;

We expect very few processes to be simulataneously trying to send to the same receiver, so it's appropriate to use the simplest possible implementation – a singly linked list. If the queue is empty, then the function adds the current process as its only element. If the queue isn't empty, then it searches for the last process in the queue and adds the current process after it.

Lecture 16