IPC Design 

Write up 

Overriding Goals
- Fast 
- Easy to maintain / Flexible 
- Reliable 
- Secure
- Multi platform

Key design points
Strongly Typed Messages inherited from base class
Point to Point queues ( eg no routing /addressing) 
Priority in queues . 
Message may activate sleeping destination ( if destination desires) 
All messaging is Asynchronous. 
User apps , Kernel and Services are treated identically. 
64 bit first class citizen support
First class NUMA multi processor support. 
Heavy degree of isolation ( eg internal data types and very moduler)
A Heavy setup cost low run cost approach 
Solid OO/modular design
App to App is a 1 st class citizen

More specifically how does this achieve our goals. 

- Fast 

We want to achieve an order of magnitude performance over Linux in IPC call.  
At this level the performance of the messenging system becomes less relevant ( accept for malloc see below) 
and we trade it for a more secure and flexible system. 

We do however allow a service waiting for a message to be immediately scheduled.

Asynchronous nature allows fast low latency traversal if services are scheduled. 
( As Synch callers are never scheduled , in this cases they can be services and get immeditaely scheduled when a reply arives) 

Alternatively it can provide high latency high troughput , by not scheduling immediately and then processing a large number of messages. 
As the clients are Asynch we dont need to reply immediately. 

Many OS have fast kernel calls but slow IPC , this is a killer for Micro kernels.

- Easy to maintain / Flexible 

Having a single uniform IPC ( eg everything uses it app to app , app to kernel etc)  is far easier to maintain than seperate ABI and IPC.
Less code is always good.

Again having no special rules for services and kernel makes IPC more simple and uniform.  

Messages are strongly typed to reduce potential errors( If performance does matter we can use UintPtr instead but a strongly typed system will result in less errors ) 

A single entry point into a service allows all messages to be validated and common code applied without having to call these in every method. 

By using messages we never need to change the code for new implementations as the base class will always be SystemMessage. 

Asynch systems by nature are more flexible , messages may be queued , or services may be moved to a different machine etc. Its even possible for 1 app to send and another app to receive.

Naturally integrates with networks . eg a P2P message app would send the messages via UDP . 

- Reliable 

Reliability is increased by using strong typing and ensuring only one writer has access to shared data. 
If the writer fails the data can be corrupt or in an undefined state  , if a reader fails there is no issue with the shared data. 

Also system reliability can be improved by using queues for a service and ordering requests this prevents task  freezing out high priority tasks due to sending large amounts of work  ( eg bring up task manage when your machine is paging to death) , 
with Messages the high priority processes can be serviced first this also enables better real time support. 

- Secure

The queue system with capability guarantees who the sender is and provides for a lot of security , this is assisted by treating everything as a user process preventing the exploitation of services commonly seen in most hacks 
. 

Typed messeges also allow for more security and less tampering. 

Treating everything as a user app encourages Principal of Least Privelage. 

If there is no ABI how do applications start ?

There is at least a send Message call. When launching an applications a loader passes the information ( with a user key) to the OS which creates the process allong with queues and a key. 

Why is the GC not part of the IPC system ( ie a seperate service) ?

The GC needs to be in process , ANY overhead would be extermely costly ( eg creating lots of strings for a web site) . 
Its worth noting the GC which is in process communicates to the MM via IPC but these are chunky calls requesting many or large pages. 
In fact we hope the allocation of the GC is simply 

Mov R1 , DWORD [ObjectSize]
LOCK XADD [GCNurseryPtr]   , R1    ; is the LOCK needed ?
;Note R1 contains the old value of the pointer but this instruction is thread safe
ADD  R1, DWORD [ObjectSize]

; there may be a better way  but this is thread safe even for NUMA architectures
; where other CPUs may update the memory.  So an add and then read is not thread safe,

Are the Messages routed ? 

To avoid complex and weighty dispatching all messages are sent point to point ( direct to the destination) . 
To get a new destination you must request it and have the appropriate Capability. By default an app has a queue to the MM system and the Security Capability system. These queues are located on the capability.
While the startup cost is higher it means 
 - No routing needed
 - Messages do not need to carry the sender and receiver this quickly becomes a major benefit. 
 - Its more secure as the connection is point to point we can identify the sender and it cant be forged.  
 - Applications dont need to talk to the kernel .

Why not route all messages ?
Basically while with routing you could prioritize better , you create a single bottleneck with locks and threading this would be an issue .  
In addition its easier to create deadlock loops. There may also need to be a kernel thread processing this queue ( or you have even more locking issues) which brings a whole range of issues.

With regard to prioritization busy services will be able to do plenty in their own queues and since they understand the nature of their callers 
are better able to serve/ prioritize them . 

Anyway the queue system allows a client to send to any point they just have to ask first . This is more secure and just as flexible. 

How do we ensure that messages passed around do not violate address space of apps ?

The only thing passed around is a pointer to the message. The Message is located in the shared address space when a request is made to create a message it creates it on a shared Heap.

What other options were considered for hangling messages?

Option1 . Have a single ABI call (everything else messages) that creates the messge on  the shared mem Heap , objects referenced in the message will need to be copied. Note this does not need to be a system call.
Option2 . Copy the message and objects on send ( and the address on the stack) ( Minix, Mach , L3/L4  apporach)

Option 3 is really the same as Option 1 and 2 just the ABI is in the GC . I dont like the copy messages on send . 
Even though it CAN be very efficient for small messages it doesnt support shared memory. The main reason its used by those OS is due to the high IPC costs.
By using a shared heap we can send quite large objects via reference this means we DONT need to copy data from the network driver to the higher layers. 
The NIC can place the data immediately in the shared memory Heap.

Why 64 bit and NUMA support ? 

All PCs built in the future will be 64 bit and multi core. It is trivial to add this now but a real pain later. 

How do the Queues work.

When an application is created it has 3 sets of queues  . Each set consists of a SendMessage method which maps to a receiving queuue at the receiver as well as a queue to receive messages from the receiver. 
By default an application has a set for the following
        - Security 
        - Memory Manager
        - Dispatcher
        
 Additional queue pairs maybe created to ANY process however a strong security check is done at the time of creation. Messages are created in shared memory and hence can map directly to large amounts of data eg a 1G file .       

How does the IPC actually work ?

The hard part is the last question , basically the message pointer is passed in a mem location, and its a pointer to the shared memory heap. 
We cant use the stack here unless we copy it which involves an intermediatery copying it. 

In more detail
Client Thread
    UserCode 
        Create Message via CreateMessageAPI [Cosmos.GC]
            Message Created on Shared Memory Heap [Cosmos.SharedMemory]

	queue.SendMessage(message, bool yield) (Ben.IPC)
	    		ProcessMessage(message)
		if ( yield) 
            	Jump to dispatcher.yieldcurrentThread.       

   
       ProcessMessage
                {
                    If ( receiver is blocked wait ing for message) 
                    {
                        Set Receiver current message to message
                        Set Receiver current message sender to send capability. 
                        Schedule Receiver                
                    }
                    else
                    {
                       block caller
				Set GC of thread to destination
                       if ( high priority) 
                            Add message to front of destination queue) 
                       else
                            Add message to back of destination queue
                        
				Restore GC. 
                    }
                    return; // end Syscall
                }

Receiver
Option 1) Good for VERY busy services

    foreach (var queue in input queues) // by priority
        if (queue.Count != 0)
            ProcessMessage ( queue.SenderKey , queue.Dequeue())

    
Option 2)  Override Add to queue , good for most apps 
OnAddToQueue(Message)
    AddMessageToSingleQueue ( queue.SenderKey , Message) 

How is scheduling done ?
After a message is sent an application is automatically stopped . However if their is quanta left it is placed at the head of the queue. It should be noted  the only case for normally sending multiple messages is multiple calls or a broadcast , we support this by allowing a 2nd Syscall which sends multiple messages ( or the same) 

Why does the system use pointers for this ?
Not performance. The main reason is a lot of items require stack which is a not allowed in an int handler.

Where is the receiver queue  in terms of address space ?

The receiver queue is in the applications memory space.  ( and yes IPC assembly does have the right to write there) 

How to handle assembly references to new Messages when building strong typed clients ? 

Apps ( especially services) should provide a seperate assembly with all the messages they require.

Why is thread safety required for sending when its point to point ?

The receiver doesnt really need it , and if a sender is a single thread app or uses one thread for messaging than its not needed either. 
However there can be multiple senders and hence thread safety is an issue.

Do receivers have a single queue or 1 queue per sender ?

This is under consideration. A single queue would be faster to parse for messages. Multiple queues allow for better checking of priviledged instructions and prevent some issues (DOS) .

Note you can really have a single queue at the IPC level ( since it would remove the sender information. ) The question is on what to expose to the higher API.
I suspect some sort of single priority orders Queue with messages containing the sender information after validation and checking 
but this is dependent on the app stack not IPC.

Stack per thread ?

Obviously each thread has its own stack . Calling another app will get a new stack

Why do we need a syscall and cant do a  simple call? 

You will always need to do  a Syscall as the destination will have a different stack and thread and a dispatch in between would be very  bad.

Do we need  a Monitor or Mutex on the shared memory ?

Neither , there can only be one writer and the compiler will prevent writes to shared objects without getting write rights first.  Partial updates are possible  ( but very unlikely for messages) , if this is an issue we can use some sort of update mechanism.

Do we need to make a syscall to create the message and hence 2 syscalls for every message. 

No , I dont think so. Apps can directly access shared memory. If not we would have to do something like SendMessage ( Type MessageType , params paramaters);

Web Services Sample .

TODO 

In the example you show TCP and IP as seperate Why ? 

In most OS the high IPC/Cintext switch cost  means its  necesity to merge things together however
from a software management and security point of view its better to have small services. eg you could allow people 
to use Sockets or TCP but not make your own/read IP packets. While this  may not be the best case it does show that 
traditional boundaries  are not appropriate to a managed OS.

Does the kernel actually exist ? Where is it .

This is almost a kerneless OS , the only thing in the kernel is the Shared Memory Heap , IPC (though its more glue between apps) and the HAL .

Why use C# structures like queues ?

Basically a GC is always available ( Kernel or client) and for readability and extention purposes these structures work best.  Also implementing most things as pointer lists / queues is really premature optomization. I think for the majority of these things the easy abbility to just plug in a new class and hence algorithm will outweigh the benefits of an optomized pointer algortihm . Did i mention less bugs with mature code...

What happens if the queue needs to grow durring a syscall ?

This may be an issue. As the GC is the target process. We can change the GC used or it may be better to use a structure that wont do an allocate.

Will a a Micro Kernel deliver reasonable performance the paper "The Cost of IPC: an Architectural Analysis" seems to say no. 

The paper which contradicts  Liedtke's earlier work is based on the following 
- Heavy message copy costs 
	We dont copy messages.
- Heavy context swith cost especially by mapping memory and the TLB flush cost.
	We dont remap memory and dont need to flush the TLB , more importantly the cost of a context switch is low. 
- LOts of short messages for a few pieces of data.
Our messages are not fixed size. 
- Memory is getting slower
We should be able to keep memory use down to a minimum.

The paper does state Micro kernels deliver a more secure and reliable OS which does apply :-)

Why not a pure Capability OS?
Because i dont understand it well enough. We do use capabilies though just not do the OS call on a capability.  eg ProcessorCapability.Yield()

What is a capability?
http://www.eros-os.org/essays/capintro.html
http://en.wikipedia.org/wiki/Capability-based_security

Why strongly typed messages ?

In no order . 

    /// 
    /// 1. Better checking of security and policy.  This can be placed at the entry point of a system rather than every method. It can also be fine grained eg service xyz cant talk to service abc.  Many exploits have come from lesser services  (eg Blaster worm via  SQL service )  , Media , and Web service then comprimising more fundamental system services like task scheduling. 
    /// 2. ! APIs never need to change and service changes are backward competible. 
    /// 3. ! Can invoke the destination thread waiting on a message without this code being in every method !!!
    /// 4. Can log and debug easier with a single point of entry.  Exceptions can be thrown but before leaving the scope of the service ( or app) can be logged and converted to a more flexible error message
    /// 5. ! Can be prioritized , allowing high thread priority tasks to jump the queue instead of blocking on a lock etc , 
    ///         this really helps provide better realtime support. High priority tasks ALWAYS win not like NT and Linux. (if Subsystes use priority ) 
    /// 6. Allow cross machine kernel to kernel messages 
    /// 7  Encourages fewer but chunkier calls in API design between services , instead of lots of small calls.
    /// 8. Allows easier Asynch calls which provide significant performance benefits
    /// 9. While a small perf hit in simple OS loop test in more complicated scenarios performance is better and more efficient due to Asynch nature.
    /// 10. Encourages non dependent call structures
    /// 11. Internal details are hidden and can be all internal ( or private)
    /// 12. Facilitates lego block design.
    /// 13. Prevent forging of information as 1) caller doenst create security (OS does)  , 2) we can clone on send. ( prevents other issues with direct call ) 
    /// 14. ErrorMessages can  be adjusted to suit the language and culture of the caller , while leaving the messages withing the subsystem english
    /// 15. Subsystems can be changed and restarted very easily. 

    /// 
    /// The negatives 
    /// 1. Requires an API wrapper for Synch operations 
    /// 2. It can be slower especially in tiny not real world benchmarks
	///	3.  When looking at the big picture apears more complex.
	
	API's
	
	ABI Required 
	
	Send Message

	Other API needed ( non kernel calls) , all within user app address
	
	Shared Memory
	    AllocateObject
	    CreateSharedRegion
	    ShareRegion
	    Change Owner
	  
	 GC
	    Alloc 
	    DeAlloc

	Mutex  
	    We can use a spin lock around shared memory and hence no Kernel call. Or use cpu lock around a shared memory structure.

	
	
	Internal kernel calls ( eg called from user thread but running priveledge code)
	    
   	Schedule
	        Save current thread 
	        Call Scheduler  
	        Schedule process  
	        Block current process  

	        
	 Kernel Message API  
	        MM 
	        
	        Scheduler 
	            Yield 

Comparison with other OS

I believe the architectire above can deliver superior performance and be simpler , secure and more reliable than Windows and Linux. 

The key is in other OS calls to another Thread must be Asynch. The only thing we are doing is making all calls Asynch so MM and Sheduler suffer. That being said 

- Our MM commands are infrequent ( 1M page size and GC does the hard work)
- Scheduling calls are rare. 

eg 
Windows Services Have their own thread and comms must be IPC  and blocking.  

Other "services" such as MM and scheduler run in user threads but not always eg Part of the file system runs in user thread but whenin needs to call a device it blocks the user thread and create kernel worker threads which talk to the device driver.

Windows Devices Drivers im not sure  but prob have their own thread as the caller must block .  

http://msdn.microsoft.com/en-us/library/ms795837.aspx

Note it is no conincidence that high performance system Web servers , SQL Servers and the File system all adopt the same strategy . That is block the caller thread and manage seperate IO threads - This design will do this by default.