Wandering through the Wilderness - Real World

rCAT 1.0 beta is online

Mark — Mon, 06 Jul 2009 18:24:44 GMT

A project I've been working on for the last two months is finally online in beta form - check out

http://rcat.codeplex.com/

It's essentially a biological alignment viewer for RNA sequences. Here's a couple of screen shots:

Alignment Viewer

2D Structure Viewer

2D Circle relationship Viewer

You can download the source code from the above link to play with it - it's a MVVM implementation and has quite a bit of interesting optimizations in it for performance purposes, including a read-only data virtualization for large sequence manipulation. The next release will include a full read/write implementation so watch the project if you are interested!

Creating Extensible Applications with MAF (System.AddIn)

Mark — Tue, 06 May 2008 21:46:26 GMT

Last month, my colleague, Pinku Surana, wrote an article about .NET AppDomains and how they can be used to provide component isolation and make your applications more reliable. If you missed it, you can read the Developments archives on Developmentor’s website. This month, I’d like to continue exploring reliability and extensibility by introducing you to a new framework included with .NET 3.5: The Managed Add-in Framework (MAF), sometimes referred to as System.AddIn.

Developers (and managers) have long desired a way to easily create extensible applications that allow new features to be added without jeopardizing the stability of the existing code base. The .NET framework has provided the underlying support to accomplish this from the very beginning through the reflection API and AppDomain support Pinku examined last month. MAF builds on that fundamental support to provide a higher-level service that allow you to dynamically discover, load, secure and interact with external assemblies used to provide features for your application. Several common architectural requirements are made trivial with MAF:

1) Isolate aspects of your code for security reasons or partial-trust scenarios.

2) Allow business partners to extend your application safely without access to the source code; Adobe Illustrator is a great example of this style of application.

3) Separate volatile sections of your application out – where depending on the customer the application needs to execute different sets of code.

4) Add or change code without unloading the application – for example pay-to-play scenarios, or where you need to update an assembly but the application must continue to run.

5) Develop and evolve different sections of the application in parallel without any fear of destabilizing one based on the other.

If any of these scenarios sound like something you could utilize in your application then you should know about MAF!

At the center of the MAF-based application is the pipeline.

The Pipeline

Communication between add-ins and the host application is strictly regulated by the pipeline. The pipeline is dynamically created by MAF through a set of loosely-coupled components which are used to version, marshal and transform the data as it passes back and forth between the host and each loaded add-in.

Each section of the pipeline is contained in a separate assembly, loaded as necessary to manage the specific add-in. MAF discovers each component and then loads them on request using reflection.

For reliability, MAF allows optional isolation boundaries to be created between the add-in and host – everything to the left of the contract is loaded into the main (host) AppDomain and everything to the right loaded into a newly created AppDomain which has its own security permission set. The isolation can also be done through cross-process calls if true process-level isolation is necessary. Under the covers, the system uses traditional remoting calls to do the work of marshaling calls back and forth.

Breaking down the pipeline, there are three main parts starting in the center with the contract.

Contract

As you might expect, MAF is based on interface contracts. Interfaces allow classes in an application to be loosely coupled, reducing the risk when changes are made between dependent sections of the code. The interface contract is shared between the host and add-in and once it has been established, it should never be changed.

Consider the simple example of a translator program. The host will expect to load one or more translator add-ins that will do some work on an inbound string and return the results. To accomplish this, I might create an ITranslator interface that looks like:

[AddInContract]

public interface ITranslator : IContract

{

string Translate(string input);
}

Notice that the interface derives from IContract. This is a requirement for add-in contracts and is what will be used to provide marshalling support when the pipeline is established. We also need to decorate the interface with the [AddInContract] attribute – this is the marker used by MAF to identify the contract when it is dynamically constructing the pipeline. Both of these types come from the System.AddIn.Pipeline namespace in System.AddIn.Contract.dll.

Views

Moving to the edges, we find the views. This is the code that the add-in and host directly interact with. It represents a host or add-in specific “view” of the contract and, like the contract, is contained in a separate assembly.

Both of the view classes will echo the structure of the contract, but not actually be dependent upon the contract. For example, our host side view might look like this:

public abstract class TranslatorHostView

{

public abstract string Translate(string input);
}

The view is commonly exposed as an abstract class to make it look more natural when used by the host, but an interface will work just as well. The host uses the view directly when communicating with any add-in based on the contract.

On the add-in side, we have an almost identical class – except we decorate this type with the [AddInBase] attribute so MAF knows which side of the pipeline this view is for (the add-in). This is located in the System.AddIn.Pipeline namespace in the System.AddIn assembly.

When we create each add-in, the implementation will use this type as the base class.

[AddInBase]

public abstract class TranslatorAddInView

{

public abstract string Translate(stringinp);
}

Adapters

The last piece of the pipeline is the adapters. The adapters play a very special role in the pipeline – they are the glue that binds the contract to the view: implementing lifetime management and any necessary data conversion between the two ends.

It may seem redundant to have this class, but by separating the view from the contract we introduce version tolerance into our architecture – the host can version independently of the add-in and vice-versa. Depending on the situation, the adapters can be as simple as a pass-through class, or can provide higher-level services to allow non-serializable types to be marshaled across the isolation boundaries.

On the host side, the adapter will implement the view (remember it is either an interface or abstract class). It will be passed a reference to the contract in the constructor and it is responsible for connecting the two together.

[HostAdapter]

public class TranslatorHostViewToContract :

TranslatorHostView

{

ITranslator _contract;

ContractHandle _lifetime;

public TranslatorHostViewToContract(

ITranslator contract)

{

_contract = contract;

_lifetime = new ContractHandle(contract);

}

public override string Translate(string inp)

{

return _contract.Translate(inp);
}
}

In this simple example, the code caches off the contract interface. This is a remoting proxy to the actual loaded add-in. The adapter then implements the Translate method and passes it forward to the contract for implementation. If we had non-serializable types, then the adapter would be responsible for converting them into something that was serializable.

It also provides some lifetime management for the contract. Because the contract is a remoting proxy and is likely running in a separate AppDomain (or even process) we have to be concerned with how long it lives. MAF provides all the support for this through the ContractHandle class. Most of the time, all you will need to do is store a ContractHandle in your host adapter and then pass it the inbound contract to wrap in your constructor.

Finally, in order for MAF to identify this class, it must be decorated with the [HostAdapter] attribute from the System.AddIn.Pipeline namespace out of the System.AddIn.dll.

The add-in side looks very similar, but does the opposite: it makes the add-in view look like the contract.

[AddInAdapter]

public class TranslatorAddInViewToContract :

ContractBase, ITranslator

{

TranslatorAddInView _view;

public TranslatorAddInViewToContract(

TranslatorView view)

{

_view = view;

}

public string Translate(string inp) {

return _view.Translate(inp);

}

MAF passes the view to the constructor and the class caches the reference off in a field. It implements the contract (ITranslator) and ContractBase which provides the implementation of the IContract interface for us (remember this was a required interface on our contract). As the host makes calls to the contract interface, this class will translate those calls to the add-in view, which as you will see is the implementation provided by the add-in itself. Note how the [AddInAdapter] attribute is used to mark this class so MAF can discover it.

If it seems like all the above is a lot of repetitious, boilerplate code well.. you’re right! To make it easier on the developer to create the pipeline components, the MAF team has created a pipeline builder available at http://www.codeplex.com/clraddins. It takes the contract assembly and then generates the views and adapters from it:

Creating an add-in

Once the pipeline pieces are built you can create add-ins. Each add-in provides an implementation of the abstract add-in view. For example, we might provide a BabelFish add-in for universalized translation:

[AddIn(“BabelFishTranslator”,

Description=“Universal translator”,

Version=“1.0.0.0”, Publisher=“Zaphod Beeblebrox”]

public class BabelFishAddIn : TranslatorAddInView

{

public string Translate(string input)

{

...

}
}

The add-in implements the AddInView, providing a concrete implementation of the Translate method. It is decorated with the [AddIn] attribute that allows it to provide a name, version, description and other data the host can use to decide whether it is a useful translator.

Putting the pieces together – the directory structure

To properly identify each of the components necessary, MAF enforces a particular directory structure you need to follow for deployment. Each piece is stored in a sub-directory off the root of the pipeline directory (this is typically the APPBASE where the host executable is stored).

The directory names are required, but not case sensitive – each directory holding a single piece of the pipeline that MAF will load dynamically when it is loading the add-in. The host-side view is always located in the same directory as the host executable itself so it does not need a dedicated directory. When creating your Visual Studio project, it is important to set the output directories appropriately so that you create the above directory structure. In addition, all references between the components should be marked as CopyLocal = “false” in Visual Studio to ensure a local copy of the assembly isn’t placed into the sub-directory:

Discovering and loading add-ins from the host

The last piece of the puzzle is the actual loading of add-ins from the host. This is done in three basic steps:

1. Identify and catalog the add-ins

2. Retrieve the list of specific add-ins based on view or name

3. Activate and use the add-in

The first step is to identify the available add-ins for the host. This is done through the System.AddIn.AddInStore class:

string[] errorList = AddInStore.Rebuild(

PipelineStoreLocation.ApplicationBase);

Calling Rebuild forces MAF to walk the directory structure and create the pipeline database. It stores this information in the pipeline root directory and returns a list of errors, if any occur. The most common errors are missing pipeline components – where MAF is unable to locate some required portion of the pipeline such as a View or Adapter.

Next, the host will retrieve a list of add-ins based on the host view through the FindAddIns method – these will be the add-ins conforming to a specific contract (whatever the view/adapter is bound to):

Collection<AddInToken> addinTokens = AddInStore.FindAddIns(

typeof(TranslatorHostView),

PipelineStoreLocation.ApplicationBase);

The first parameter is the host view type – so MAF knows what add-ins we are looking for, the second is the pipeline root directory, which is the same directory passed to the Rebuild method and indicates where the pipeline database is stored.

The returning collection represents a series of tokens that are used to identify and control the add-ins. This is how the host can examine, activate and unload the add-in. To activate a specific add-in, you can take the token and call Activate:

Collection<AddInToken> addinTokens;

...

foreach (AddInToken token in addinTokens)
{

TranslatorHostView view =

token.Activate<TranslatorHostView>(

AddInSecurityLevel.Internet);

string hello = view.Translate(“Bonjour”);

...
}

The Activate call loads all the required components, instantiates the add-in type and returns the host view used to interact with the add-in. Calls made to this object will be marshaled back and forth to the add-in using the pipeline.

Notice that the parameter passed to Activate indicates the security level required. There are several overrides that allow you to dictate exactly how the add-in is loaded. You can load the add-in into a specific AppDomain – the current one for best performance:

token.Activate<TranslatorHostView>(AppDomain.CurrentDomain)

You can specify a specific permission set to restrict the things the add-in can do on your behalf:

PermissionSet pset = ...;

token.Activate<TranslatorHostView>(pset);

Or you can specify a known permission set based on the CAS zones:

Once the add-in is activated, the host can call it just like any other object – but never forget that you are likely crossing an isolation barrier! Make sure to design your contracts so that you minimize the number of calls between the host and add-ins to ensure your performance doesn’t suffer.

When you are finished with the add-in, you can tell MAF to unload it through the AddInController associated with the view:

AddInController ctrl =

AddInController.GetAddInController(view);

ctrl.Shutdown();

This will unload the add-in side pipeline and then destroy the containing AppDomain so you release the resources associated with it.

There are many other capabilities MAF provides such as versioning, passing collections and WPF visuals, passing non-serializable types, etc. You can get a pretty decent overview of the features from MSDN and the MAF team has a blog at http://blogs.msdn.com/clraddins/ that has some great, practical information on using this new framework.

Adding Pause/Resume into SynchronizationContext based components

Mark — Wed, 17 Jan 2007 16:11:04 GMT

About a year ago, I blogged on using the .NET 2.0 facility SynchronizationContext to create thread-aware components which could be hosted and would "do the right thing" for callbacks. Here's the Link to that post.

Recently, I got an email from a reader who wanted to use that paradigm, but wanted to be able to suspend the operation for some period of time and then resume it later on. He asked if there was an easy way to modify the sample I presented, and it turns out that there is.

It essentially involves three steps:

1. In the AsyncStateData structure, add a ManualResetEvent object. This will be used to trigger the pause/resume behavior. It must be initially set as signaled.

2. In the actual asynchronous operation loop, add the event into the loop condition by calling event.WaitOne(). Make sure you have released any locks and the code can safely be halted at that point in the logic otherwise you may create deadlock scenarios and other difficult debugging issues.

3. Add a Pause and Continue method into the class which signals and resets the event.

Modifying my previous code example, here's what I come up with:

Step 1: Add the event to AsyncStateDate

 1: class AsyncStateData

 2: {

 3:  private ManualResetEvent
pauseEvent = new ManualResetEvent(true);

 4:  ...

 5: }

Step 2: Add event into the loop at a safe point

 1: public partial class Calculator

 2: {

 3:  private void InternalCalculatePi(int digits,
AsyncStateData

 4: asyncData)

 5:  {

 6:  int completedDigits
= 0;

 7:  for (; !asyncData.canceled
&& pauseEvent.WaitOne(-1, true) &&

 8: completedDigits
< digits; completedDigits++)

 9:  { ... }

 10:  }

11:

 12: ...

 13: }

Step 3: Add a Pause and Continue API

 1: public partial class Calculator

 2: {

 3:  public void PauseAsync(object asyncTask)

 4:  {

 5:  AsyncStateData
asyncData = asyncTask as AsyncStateData;

 6:  if (asyncData
!= null && asyncData.running == true)

 7:  asyncData.pauseEvent.Reset();

 8:  }

9:

 10:  public void ContinueAsync(object asyncTask)

 11:  {

 12:  AsyncStateData
asyncData = asyncTask as AsyncStateData;

 13:  if (asyncData
!= null && asyncData.running == true)

 14:  asyncData.pauseEvent.Set();

 15:  }

 16: ...

 17: }

There are certainly other ways to do this as well, for example, create a dedicated Thread object instead of using an Async delegate and then call Suspend on the thread (and Resume to continue). That can be dangerous if you take locks while running your async operation so the ManualResetEvent is probably a better solution from that perspective (since you are blocking in a known location vs. just suspending the thread arbitrarily and potentially holding onto shared resources.

Living in an OS X world

Mark — Fri, 01 Sep 2006 16:15:50 GMT

A DM colleague of mine, Neils Berglund noted his required applications under OS X and asked for feedback on what everyone else is using (assuming you are on OS X anyway).

The original question was what RSS reader are you using.. my preference is NetNewsWire as I was used to the Newsgator line of products when using Windows all the time so it made an easy transition for me. I also use MarsEdit to update posts on this blog -- it's not WYSIWYG but it keeps my HTML skills (if I ever had any) from drying up.

So, adding to Neils list, I also use --

Final Cut Express - for editing movies, I found iMovie to be too limiting.
CHMOX - to open and read .CHM (Windows Help) files.
DivX codecs - to play my 300+ .divx movies.
Smultron - a decent, quick color-coded text editor
Virtue Desktops - an awesome window manager.
Remote Buddy - allows the Apple Remote to be used to control other applications such as Power Point.
Microsoft Office - I just can't live without Word/Excel/Power Point. I know Apple has iWork, but it just isn't quite the same.
Flip4Mac - Codecs for .WMV (Windows Media), essential in a 99.999% Windows dominated world.
Coverflow - this isn't critical, but it's a fun app to show off - it let's you view your music through the album covers in a cool, flip-3d style fashion.

I'm also trying out a .mac membership for backup purposes. Not sure I'll stay with that, but it is automatic and seems to work even when I travel.

So, what things do you think are necessary applications under OS X?

Cheating on my computer..

Mark — Thu, 25 Aug 2005 16:51:18 GMT

I feel like I've committed adultery.. I've gone and purchased a Mac-Mini.

I pushed my wife to get into email and the Internet. She's not really into technology or computers; she'd much rather have a conversation with someone or meet face to face than to use email as a communication tool. I kept espousing the value of email, how easy it is to keep in touch, how quick it is, how you can search for things on the Internet. She finally bit the bullet and started using Windows XP on an old computer we had. That's when the trouble started. No sooner had I gotten her connected and she had contacts and relied on the daily communication, the computer started flaking out. First it was the Internet connection itself.. it would just disappear off the network. The printer would stop working for no reason - a reboot would generally fix it. Next it was IE and Outlook Express not working properly, finally the motherboard just died. Each of these problems required me to come in and save the day.

Now mind you, I'm not at all adverse to looking like the hero to my wife, but the problem is that she has now gotten it firmly in her head that technology sucks. You have to be some kind of geek to keep it running. Enter the Mac. I've got several friends who proselytize to me constantly about the stability and reliability of their computing platform, how it doesn't have viruses, how they just leave it running all the time. I'm hoping these claims are true because I'm tired of keeping the computers running here.

So, I go down and buy the thing and bring it home in this ridiculously small box. Unpacked the thing is about the size of two bricks set side-by-side. Incredible. It's also amazingly quiet. You can't hear it unless you are using the DVD - unlike the PC tower which sounded like a jet taking off most of the time. Now I know you can get quiet towers, but it's hard to beat the form factor here - I've not seen anything quite like it in the PC world.

The setup was pretty smooth. I plugged in the monitor/keyboard/mouse and then turned it on. It detected my wireless network and even allowed me to connect to the NAS drive I use for backups. I was able to get my wife's email exported from Outlook Express into the iMail program pretty easily, her address book moved over too. Programs seem slower, especially the DVD installs but overall I'm impressed.

Appearance-wise, OSX is very pretty. The icons are very nice, and I generally like the interface. It's not too drastically different from XP, although I miss my "tasks" pane which allowed me to do various things without searching through the interface. I introduced it t my wife and showed her how to start Safari, the email program, and Word. That's pretty much all she needs. Most of the kids games installed onto it (funny enough, little kid games are mostly cross-platform since they use Flash and Quicktime). So far so good.

I have had a couple of quirky things that are unsettling. Starting up multiple programs really bogs the thing down - I presume that's because it's a slow G4 with only 512M of memory. I had Finder lock up on me and I had to figure out the CTRL+ALT+DEL equivalent on this machine (OPT+APPLE+ESC for those who are interested). It didn't recover and I had to power cycle it.. hopefully that won't be a common occurrence.

I want to try Mono on it and see if I can move some basic .NET code onto the box.. although I've found that computers tend to be more unreliable if I keep messing with them.

Here's hoping that my laptop won't get jealous and start crashing on me.

Welcome to my Blog

Mark — Thu, 11 Aug 2005 21:45:46 GMT

I'm Mark Smith (yeah right!), and I work in the Microsoft .NET and Telephony space. I do consulting in the Dallas Texas area and also teach for DevelopMentor.