Domain Specific Language for WWW with Irony

Recently I have published article at CodeProject. I was influenced by the idea of Domain Specific Languages for some specific tasks. In the article I have presented DSL used to automate some WWW operations (GET, POST, file download). To solve the problem I used Irony as DSL interpreter. More available here. More regarding great project Irony here.

C# 4.0 (dynamic types, optional parameters, named arguments, covariants and contravariants handling)

New features of C# 4.o are presented in several places.
Ironically the most exposed is Doug Holland's at Intel's blog site :) - nice and very brief overview.
More info at Channel9 video provided by C# GoF :). The most striking facts are:

1. dynamic typing is for better Office interaction,
2. optional parameters and named arguments are evil, provided for VB developers to ease Office development in C#,
3. covariance and contravariance are only interesting (good) changes in language, but it should have appeared in previous versions of C#.

Additionally more about covariance and contravariance is on Eric Lippert's blog and more about dynamic programming on Chris Burrows' blog.

volatile - what does it mean? (in C++, C, .NET and Java)

Well it is good felling that my interests are also Herb Sutter interests (in programming area of course:). In his article on Dr. Dobb's he presents differences between volatile meanings in different words (C++/C vs. .NET/Java).

Main points:

• volatile in C++ is connected to optimization during access to variable - no optimization is allowed. Operations on nearby non volatile variables depends on compiler (can be move before or after volatile operation).
• operations on volatile in C++ does not guarantee atomicity - resolution is atomic in C++ (or e.g. atomic_int in C) available in Boost (but I cannot find it) and will be in C++0x.
• volatile in managed environments (.Net, Java) does not allow to move some operations on nearby non volatile variables - "ordinary reads and writes can't move upward across (from after to before) an ordered atomic read, and can't move downward across (from before to after) an ordered atomic write. In brief, that could move them out of a critical section of code, and you can write programs that can tell the difference.".

volatile in .Net/Java:

• keeps order - lock free programming,
• allows some code optimizations - not good for interactions with hardware - but BTW managed environments does not allow to treat memory as a resource to write somewhere programmer might like to - unmanaged code have to be used (e.g. C++ with its volatile:).

volatile in C++/C:

• might not keep order - depends on compiler,
• does not allow code optimization - good for interactions with hardware.
  

Generally speaking volatile in C++ means unoptimizable variable (optimizations in access are not allowed) and in managed worlds it means ordered atomics (other operations cannot move before/after and operations on volatile are atomic - sometimes?).

Not sure points:

• atomicity of volatile operations in .Net/Java - for what variables, e.g. what about architectures 32, 64?,
• "These (volatile) are suitable for nearly all lock-free code uses, except for rare examples similar to Dekker's algorithm. .NET is fixing these remaining corner cases in Visual Studio 2010" - what are the problems?

Optex, fairness v.s. convoys and critical regions

Interesting, but quit old article concerning synchronization (with fairness, convoys considerations) by Jeffrey Richter can be found at http://msdn.microsoft.com/en-us/magazine/cc163642.aspx.

According to author fairness cannot be assured in .NET - main reason is GC. GC has to suspend threads before execution but sometimes suspended is a thread that invokes unmanaged Win32 WaitForSingleObject or WaitForMultipleObjects function. After wake up order in waiting queue can be different than original. Other sources of lacking fairness are APC and debugger.

Optex synchronization mechanism:

• point of the presented solutions is to avoid switching into kernel mode (interlocked is used for this task),

• latter of the presented solutions allows new coming threads pass before threads waiting in semaphore queue (waiting in kernel mode). This unfair solution, in case of constant stream of threads requesting enter, has better throughput than former kernel unlocking solution,

• fairness is completely sacrificed to avoid convoy problem.

Also some words concerning Critical Regions appear in the article:

• "hand made" synchronization mechanisms shall invoke BeginCriticalRegion in Enter and EndCriticalRegion in Exit,

• the mechanism informs CLR host that thread was interrupted (aborted) in critical region and some special action shall be performed (e.g. closing appdomain)

Question - how host is informed?

More Jeffrey Richter on MSDN Magazine - http://msdn.microsoft.com/en-us/magazine/cc301191.aspx

asynchronous exceptions, threads and problems

Aborting thread is evil (lock, using - generally speaking asynchronous exceptions). Abort is ok when thread invokes it on its own (synchronously).

Aborting other thread is ok when we want to destroy also its appdomain.
Interesting article concerning asynchronous exceptions and lock on Eric Lippert's Blog:
"You don't want to take a thread down unless you are doing so in the process of taking down its appdomain. If you cannot write a thread that can shut itself down cleanly, then the appdomain is the sensible structure to use for work that may be arbitrarily cancelled." -- Eric Lippert

Info on thread's abort (from http://www.albahari.com/threading/part4.aspx):

• Static class constructors are never aborted part-way through (so as not to potentially poison the class for the remaining life of the application domain)
• All catch/finally blocks are honored, and never aborted mid-stream
• If the thread is executing unmanaged code when aborted, execution continues until the next managed code statement is reached.

As we can see from above exception is delayed when happens during e.g. finally - it means that thread might never end (if it hangs in finally).

Additionally Suspend (and Resume) is even worse than Abort - can lead to deadlock (suspended in lock). Suspend can be safely called on thread's own. Other thread can invoke Resume - but it is not easy and error prone (more at http://www.albahari.com/threading/part4.aspx).

volatile, execution order, memory model, semantics

Here is my little investigation on volatile, execution order and everything what comes next...

From C# language specification - 3.10 Execution order:
"Execution of a C# program proceeds such that the side effects of each executing thread are preserved at critical execution points. A side effect is defined as a read or write of a volatile field, a write to a non-volatile variable, a write to an external resource, and the throwing of an exception. The critical execution points at which the order of these side effects must be preserved are references to volatile fields (§10.5.3), lock statements (§8.12), and thread creation and termination. The execution environment is free to change the order of execution of a C# program, subject to the following constraints:

• Data dependence is preserved within a thread of execution. That is, the value of each variable is computed as if all statements in the thread were executed in original program order.

• Initialization ordering rules are preserved (§10.5.4 and §10.5.5).

• The ordering of side effects is preserved with respect to volatile reads and writes (§10.5.3). Additionally, the execution environment need not evaluate part of an expression if it can deduce that that expression’s value is not used and that no needed side effects are produced (including any caused by calling a method or accessing a volatile field). When program execution is interrupted by an asynchronous event (such as an exception thrown by another thread), it is not guaranteed that the observable side effects are visible in the original program order."

"The memory model in .NET talks about when reads and writes "actually" happen compared with when they occur in the program's instruction sequence. Reads and writes can be reordered in any way which doesn't violate the rules given by the memory model. As well as "normal" reads and writes there are volatile reads and writes. Every read which occurs after a volatile read in the instruction sequence occurs after the volatile read in the memory model too - they can't be reordered to before the volatile read. A volatile write goes the other way round - every write which occurs before a volatile write in the instruction sequence occurs before the volatile write in the memory model too." (from http://www.yoda.arachsys.com/csharp/threads/volatility.shtml)

"Since field finished has been declared volatile, the main thread must read the actual value from the field result." (from http://msdn.microsoft.com/en-us/library/aa645755(VS.71).aspx)

According to above execution order in .NET memory model is kept with volatile!
Does it mean that former writes to variables that are not volatile will be finished (probably yes) and visible to other threads?
How volatile influences operations on memory (especially cache)? What if each core has its own cache or multiprocessor is used?
Am I asking about volatile v.s. cache coherence?
Some reference to this problem:
"The JIT is responsible to maintain correct semantics for
a given target processor by emitting the necessary instruction for that
processor, including processor-specific memory ordering ops like
load-acquire, fences ..." (from http://www.eggheadcafe.com/conversation.aspx?messageid=30644089&threadid=30643832)
"The Whidbey memory model (Framework V2) targets both IA-32 and IA-64, this
memory model assumes that every shared write (ordinary as well as
interlocked) becomes globally visible to all other processors
simultaneously. This is implicitly true because all writes have release
semantics on IA-32 and X64 CPU's, on IA-64 it's just a matter of emitting a
st.rel (store release) instruction for every write to perform each
processor's stores in order and to make them visible in the same order to
other processors (that is, the execution environment is Processor
Consistent).
Above means that the JIT has to emit a st.rel for _name = name; when run on
IA-64 (64 bit managed code), so the other thread will actually see p.Name
pointing to the string.
Add to that that a thread creation implies a full barrier (fence), so in
this particular case it's not required to include a MemoryBarrier in the
thread procedure. Note that the CLR contains other services that implicitly
raise memory barriers, think of Monitor.Enter, Monitor.Exit, ThreadPool
services, IO services..... So I think that for all except the extreme cases,
you can live without thinking about MemoryBarriers in managed code even
when compiled for IA-64." (from http://www.eggheadcafe.com/conversation.aspx?messageid=30644255&threadid=30643832)

Well, It seems that everything is perfect:) We should rely on JIT to generate proper code for underlying hardware.

Double-Check Locking in C# using volatile (from http://msdn.microsoft.com/en-us/library/ms998558.aspx)

More about memory models,semantics on http://www.diag.com/ftp/Memory_Models.pdf

Article about volatile in C# at http://www.codeproject.com/KB/threads/volatile_look_inside.aspx

Producers-Consumers (ordered and unordered, bounded buffers) in C# on CodeProject

Recently I have published reworked version of producers-consumers library.
Available at Multithreaded buffer poll in .NET
Code contains examples: encryption-compression and decompress-decrypt programs.