Cross Domain Communication  (CDC)

Address Space
No such concept ..

An application Domain consists of a number of threads 
an App domain has an Allocator and a Collector and a limit to the memory it can allocate.

Why not a global collector

I dont believe a global GC is viable . However  multiple apps could share a Collector if desired and hence have less CDC issues. eg They are tightly coupled then and have no issues interacting. 

However System services and device drivers will be more loosely coupled and have their own Collector. Though they may share a single system Real time Collector. Which will probably be the first tri colour collector we use.  

The main issues with a sharing a Collector are  
 - Different behaviour required for server , real time , user apps   and device drivers. A larger nursery could help some apps at the cost of more memory.
 - Stop the world generational implementations ( which are the most common) would stop for extended periods eg for 10M+ objects it could be in the range of seconds. Note the stop time is about equal to all the small collectors added together. 
- One app could generate significant collections and flood the nursery affecting other apps. ( Note this app would run faster
- Idle apps cant be collected / compacted.

Sharing data .

A thread may be part of more than 1 app domain ( this affects the Collector scanning the Threads Stack and suspending the thread for collections) .  But only the prime allocator thread is used for allocations.  The prime allocator can be changed in a using block. For an external thread to create objects in the destination this must be done.

What is require to share data 
- a pointer is submitted to the App Domain with a name which represents the root of the tree which is externally accessible.

- The types must be in a non root assembly and caller should have access to it 

- It is recommended that applications create an external interface and an Assembly with the Interfaces to allow other applications to compile against it. 

- It is important to remember the allocator.

Using ( externalAppDOmain.OpenShareForWrite("App1Shared")    // will change allocator and register threads for collection
{
	Create message data objects
	Create message 

	Send message( message)  ; // ie lock queue and add to queue. Note the queue may expand but the Allocator is teh receiver so no issue.
       
}

Note the policy here is Sender creates the object in the receiver.
Developers must be VERY carefull not set references in the destination to the source unless they are sharing a Collector. 

How does the reply work 

Just as it was sent.  Note events can be used to check if the message has been processed especially for void returns this is very efficient.  

Thread pool optomization.

A lot of threadpools work on a small scope of objects it is advisable for high performance scenarios to create a new domain for this. As collections are likely to be small and the work wont be stopped when the main application collects.  

An SOA style message passing style is recommended .

eg Messages are strongly typed  , Types available should be in interfaces or abstract base classes.  ( Note this is more extreme than WCF but WCF has XML as its underlying type. 

The main reason for this is 
•	The API can be changed without all other applications referencing the assembly failing and you dont get API bloat ( which you get with lots of API interfaces) . 
•	Single point of entry. This helps security and checking
•	Messages can be queued for bulk processing.
•	Breaks the chain and hence denies external access to large parts of the application tree.
•	Messages can be prioritized
•	Devs needs to read the API instead of just using an API call and get weird memory issues.
•	Need to hand of to a new thread. Executing the code of the receiver is possible but should in most cases be limited for security and performance reasons ( locking contention , worse cache coherency compared to a thread processing a queue) . The RPC style and block is pointless.  Though we may write a wrapper which does this to make it easier.  

Events are also highly recommended but remember information in EventArgs will belong to the sender GC .

Static Main 

The entry point assembly will be entirely private eg no methods and types are exposed for  other assemblies to link to.

Public Static

Public statics are globally visible ie the whole OS. They should not be exposed in the shared assembly ( data is exposed by registering them) 

Object storage changes  [TODO move following to Allocator/collector]

We need to store a bit with each type to determine if it has no submembers and hence does not need to be scanned in the collector. 

We can add this to the 2 bits we need for the GC coloured sweep. 

Note the type is probably not a good field for this even though it has room ( even 24-26 bits would be fine ) it is probably used so frequently that the extra mask is not worth  the storing saving.

Probably best to use a Synchblock field bits. 

Tricolour Mark Collector

By using a tri colour scheme we can use a multi threaded mark. We still need to stop all Mutator threads since the  mark may occured in between a pointer changing ( note for a ref counting GC the program can continue to run except for the stack scan) .  This will significantly reduce the impact of the GC  on application performance in multi threaded systems eg if the cost is 10 % in a normal Mark  in a Tri colour Mark it will be 2.5% in a quad core. Sweeps can be done while  the heap mutators are running compaction also needs to stop the world.  With this in mind the GC will not be a major issue as compaction only needs to be done when under heavy memory pressure.

Write barrier / Read only pointers. 

At the moment we have VERY fast allocation doing any work on allocation or on putting write barriers on changing reference will significantly impact performance especially since many small string allocations occur in C# apps. Though these techniques could be use to enhance security and allow threads to run while doing a mark.

Note the compiler CAN do reference counting for setting / changing references . This would be very usefull for Device Drivers and real time code since then the program would not need to be stopped for the Mark . I suggest we do this on a case by case basis eg if The assembly is marked include ReferenceCounting include the code though the GC will also need to be set to a ref counting one. 

Where is the security  ?

- Types cant be exposed unless A shared assembly is available to compile against ( note this could be forged as an assembly could be created from hand or from meta data) 
- Internals and private members , methods , fields and properties are not available to the calling party. Note it is quite valid to have a type with public get properties and methods but a private or internal constructor. This mechanism can be used to protect data but references to the type cant be forged . Note they can be exchanged for other objects of the same type the caller has access to.  

How to enforce non forgeable references  

Make the constructor of these type or the type internal hence references can only be forged within the assembly. Be carefull of base type and interface cast pointers  as these may not be secure unless the base types are also internal.

This seems hard how to help developers write secure code ?

The message passing abstraction helps as types are not available and the data created by the sender.  This needs more thought. Keeping key things private / internal and providing public copies to users will also help.  eg Thread  will be an internal object to the scheduler.  ThreadView ( or UserThread ) will be handed out when requesting Thread information ( eg System.Threading.Thread will be a ThreadView) . ThreadView can interact with Thread . 

Note it would be possible to expose Thread and public properties directly and is a legitimate way however the view class is more mistake proof. This is similar to how collections return a array new collection when getting the values rather than exposing the internal representation. 

How is the GC called

The GC is called via Thread.Current.Allocator. This is set when a thread enlists in an app domain. Allocations to the nursery should be just a few cycles and inlined.  eg the following for a NUMA design 

Mov R1 , DWORD [ObjectSize]
LOCK XADD [GCNurseryPtr]   , R1    ; is the LOCK needed ?
;Note R1 contains the old value of the pointer but this instruction is thread safe
ADD  R1, DWORD [ObjectSize]

The GC is called directly by the appropriate thread without CDC.

How does the GC work on private classes and capabilities ? 

It uses pointers and walks the heap.

Service types

A Service is any App Domain that receives communication from another App Domain.  Device Drivers are services as they at least communicate with the HAL.  By Implementing ISystemService they can be managed by the system ( eg , stopped , started , monitored , restarted if not responding etc)

3 Types 

1) Asynch Messaging with Task Switch. The system has in build messaging for more secure communication. This involves exposing a public interface which the caller uses.  ( Since messages require exposing a queue and access requires a granted reference running them on the caller thread is not viable ( since the read from queue is not made public) .
2) RPC with no Task Switch. It is possible to communicate by directly modifying objects accessed through either  public static or which are exposed via the App Domain. Locking must be used and great care in terms of gardbage collecting . Recommended for simple apps that dont require security or servers that dont require a lot of threads. eg a Cache Service which is called by the file system.
3) RPC  with Task Switch . When the caller calls ( as per 2) , a thread of the service is called with the paramaters ( which may be pointers) on the stack . Shared data must be locked. 

The OS mainly uses 1) . 

System Assemblies.  

This is a special case as its not an app domain but a Re-Entrant Assembly can be called from any thread with no risk.  The GC functions as one ( though its nor re-entrant) 

Thread and Allocator 

What code does thread have access to .. 

, Launch entry point.

What data does a  thread have access to 

Anything it created. Via an AppDomain. 

AppDOmain
Code Entry Point.
Allocator
[Threads] 

Why tie treads to appDomain ? 

Just have 

Thread
 domain ->
 capability->  ( in most cases a pointer to the key ring in the app domain) 

can be null etc ..If a thread does no heap allocations it does not need to be in an app domain! This can apply to some worker threads which can then have Capabilities set to the minimum set for execution. 

How to kill / manage a process 
You kill a thread. 
A kill requires all threads sharing the same app domain to die. 
A domainless thread would have no issue if it was killed. 

Pause a service will cause all threads in a service loop to not process messages. This avoids locking issues by  pausing a thread.

Security 

Cap for referenced Assemblies is needed. Hence very few apps can directly access a kernel dll.  

Starting an app and visability .

How do you run an app ?

The loader loads the file ( file system checks rights)  ( if not already loaded) 

It is checked if the user has a execute capability to the Assembly and all referenced assemblies ( dif from file) 

If the file is not loaded the system will patch the actual memory addresses of referenced assemblies and call addresses. ( Elf loader does a lot) 

A thread and App domain are created with appropriate capabilies 
The thread jmp to the entry point. 

Are Code Assemblies global ?

Yes and managed by the loader. But the data for each thread is different. Static data for an assembly is only relevant within the same AppDomain . Hence a new app domain is effectively a reload.

How do you install a new Assembly ?

That is part of installing a program which compiles it. The system does NOT support 2 DLLs that are different but have the same version if so the new install will fail.  If the versions are different the new Assembly will be installed and if applications reference 

Where are compiled assemblies , Capabilities and application meta data stored?

In a special directory to which few app have rights. This is similar to the registry and if we implement it through  SQL there would be some benefit ( eg roll back configuration etc) . However this is probably too optomistic and we are betetr of relying on .Net config files at first. 

Isnt there a security risk with shared data?

Yes , sharing an app domain provides some efficiencies but allows static data to be shared.  It is best for applications that intend to share data create a seperate Share App Domain , and expose an end point. The Shared data should not have any statics .

The Assembly loader itself is a services so is not as easy to corrupt. 

Single thread vs App domain

Why not just single threads?

Because they become bloated . Threads need to be light weight.  
Why not

Process
	AllocatorPool
	Collector
	LWthread
 ( note 1!) 
	Capability Keyring

I think this is best . 
No app domains. ( An Appdomain is a number of threads sharing an Allocator Pool) 

Static variables are created on the heap and are relevant to that AllocatorPool

Threads/Processes  are capability type objects and managed /created by the appropriate service.  As they are not allocated by the allocatorpool we can swap an allocator pool to disk and restore it when a thread needs to go active . ( eg its waiting on something) .

Send Message will be done via queues!

These queues will be fixed length ( in receiver app Meta data) and senders will block when it is full ( allong with a warning) 

The implementation of the queue is a ring buffer.
The send mechanism works on 2 pointers in the queue the start and the end.
The sender controls the end of the queue.
The receiver controls the start of the queue.

An interlocked compare on an owner bit ( sener/receiver) ensures thread safety.  However some optomizations can be made.

[Ensure that receiver ALWAYS releases this bit after removing message this should be very short duration so sender rarely has a lock on the queue

http://f-cpu.seul.org/new/michael98nonblocking.pdf

c#

http://code.google.com/p/zeroflag/source/browse/trunk/Threading/LocklessQueue.cs

eg
for(;;)
{
var listToProcess = new List<Message>(); 
queue.Get()
listToProcess.Add(queue.RemoveAll())
queue.Release();
process(msg)

}

the follwoing is real bad since if the destination is asleep processing sender cannot send.
for(;;)
{
Message msg
queue.Get()
msg = queue.Remove()
process(msg)
queue.Release();
}

If lock not available sender will Idle until set which should schedule the receiver. . 
while (queue.GetLock == false)
	Sleep(100); 

http://code.google.com/p/zeroflag/source/browse/trunk/Threading/LocklessQueue.cs