Reflecting Java into Scheme

Kenneth R. Anderson, BBN Technologies, Cambridge, MA
KAnderson@bbn.com

Timothy J. Hickey, Brandeis University, Waltham, MA
tim@cs.brandeis.edu

Abstract

We describe our experience with SILK, a Scheme dialect written in Java. SILK grazes symbiotically on Java's reflective layer, enriching itself with Java's classes and their behavior. This is done with two primitive procedures, (constructor) and (method) that provide access to a specific Java constructor or method respectively. A recent extension allows an entire class's behavior to be imported using (import). The extension also provides generic functions that take methods written in Java or Scheme.

In return, SILK provides Java applications with an interactive development and debugging environment that can be used to develop graphical applications from the convenience of your web browser. In Java terms, SILK is a Java bean box with Scheme as a runtime scripting language. However, because SILK has introspective access into Java, it can also be used for compile time metaobject scripting. For example, it can generate a new class using an old one as a template. 

1 Scheme in Java feels like SILK

Java's reflective layer provides access to metaobjects that reflect primitive, class, array, field, constructor, and method objects. There are obvious limitations to its reflective capabilities. For example, one can only affect the behavior of the running program by invoking methods and accessing fields of objects. One cannot define or redefine a new class or method. Also, one cannot subclass any metaobjects.

Despite these restrictions, Java's reflective capabilities are used successfully by many Java facilities, such as serialization, remote method invocation (RMI), and Java Beans. However, programming in base Java, is different from programming in the reflective layer. While Java is a statically typed language, the reflective layer is more dynamic. This suggests that a dynamic language, such as Scheme, could be ideal for programming the reflective layer. Scheme could be used as a dynamic language at runtime, and as a dynamic metalanguage at compile time.

Here we describe SILK (Scheme in about 50 K), a compact Scheme dialect written in Java. Because Java is reflective, considerable power can be gained relatively easily. For example, only two functions, (constructor) and (method), need to be added to Scheme to provide complete access to the underlying Java.

Following Common Lisp, SILK provides generic functions. These generic functions accept several method types:

By arranging for Java to do most of the work, method dispatch is efficient. For most generic functions, the method dispatch overhead is only one Java method invocation beyond the normal overhead of invoking a reflected Java method through Scheme.

The foreign function interface between Scheme and Java is virtually transparent to the user. The function (import <class name>) makes the methods and final static fields of the Java class named class name accessible to Scheme.

SILK started out simply as a Scheme implementation in Java. However, as its use of reflection has grown, it has become a powerful tool for controlling Java applications. The following sections describe what programming in SILK is like focusing on issues related to reflection and the implementation of generic functions.

As this paper is concerned with Scheme, and Java reflection, we begin with a very brief introduction to the essential features of these topics. Anyone familiar with these topics and simply skip to the next section. Anyone with some knowledge of object oriented programming and reflection should be able to follow this paper easily.

1.1 Java reflection

Java is a staticly typed language, with a syntax similar to C [Java]. A Java program is built out of a set of classes. Java classes support single inheritance of implementation (extends) and multiple inheritance of interfaces (implements). An interface is a class that describes an object in terms of the protocol (set of methods) it implements, but provides no implementation itself.

A class definition can contain other class definitions, static and instance fields, and static and instance methods.

A Java method has a signature which is its name, and the types of each of its parameters. The signature is used at compile time to narrow the choice of which method to be invoked.

Java has two types of methods:

  1. An "instance method" has a distinguished first argument as well as possibly some parameters. When such a method is invoked, the dynamic type of this argument is used to select the appropriate method to invoke at runtime, while the declared types of the parameters are used to choose a signature at compile time.
  2. A "static method" does not have a distinguished argument, but may have other parameters. It fills the role of a function in other languages. The choice of which static method to invoke is determined completely at compile time.

The java.lang and java.lang.reflect packages provide metaclasses that reflect the class and its components. Here's an example of using the reflection layer to construct a hash table of size 10 and then invoke a put method on it:

1234567890123456789012345678901234567890123456789012345
package elf;
import java.lang.reflect.*;
import java.util.Hashtable;

public class Reflect {
  public static void main(String[] args) {
    try {
      // Hashtable ht = new Hashtable(10);
      Class[] types1 = new Class[] { Integer.TYPE };
      Constructor c = 
        Hashtable.class.getConstructor(types1);
      Object[] args1 = new Object[] {new Integer(10)};
      Hashtable ht = (Hashtable) c.newInstance(args1);

      // ht.put("Three", new Integer(3))
      Class[] types2 = new Class[] {Object.class, 
                                    Object.class };
      Method m = Hashtable.class.getMethod("put", 
                                           types2);
      Object[] args2 = 
        new Object[] { "Three", new Integer(3) };
      m.invoke(ht, args2);

      System.out.println(ht);  // Prints: {Three=3}
    } catch (Exception e) { e.printStackTrace(); }}}

1.2 Scheme

Scheme is a dynamically typed language that uses simple Lisp syntax. A Scheme program is built out of a sequence of expressions. Each expression is evaluated in turn.

Scheme provides a set of primitive types. The following exhibit shows the Scheme type name and its Java implementation used in SILK:

Scheme       Java
boolean      Boolean
symbol       silk.Symbol
char         Character
vector       Object[]
pair         silk.Pair
procedure    silk.Procedure
number       Number
  exact      Integer
  inexact    Double
string       char[]
port     
 inputport   silk.InputPort
 outputport  java.io.PrintWriter

A Scheme procedure takes arguments of any type. Type predicates can be used to identify the type of an object at runtime.

While Scheme would not be considered "object oriented", the Scheme community has developed SLIB, a standard library that provides more complex data structures built from these types, including several object oriented extensions.

The power of Scheme comes from such features as:

  1. A compact an clear definition, requiring only 50 pages, including its denotational semantic, example, references, and index.
  2. a procedure can construct and return a new procedure, as you would expect from a functional language.
  3. the syntax is so simple that new minilanguages can extend the language easily using functions or Scheme's macro facility, (define-syntax).
Both Java and Scheme provide garbage collection.

2 SILK started out small

SILK stands for "Scheme In about 50 K". The original versions, up to SILK 1.0, were developed by Peter Norvig. The initial version of SILK was written in about 20 hours with about 650 lines of code. The primary goals were to develop a Lisp that was small, fast to load (even over the web), easy to understand and modify, and that could interface to java. SILK expanded to about 50KB of Java code over the next few months as it was extended to pass all of the tests in Aubrey Jaffer's online r4rstest.scm [4T] test suite which tests Scheme compliance with the R4RS standard.

After Peter made this implementation available on the web several people made useful (and sometimes almost identical extensions). This version was a tiny, straightforward Scheme interpreter.

Tim Hickey, who had his own Scheme in Java, adopted SILK, and added JLIB, a library that provides convenient access to the Java AWT. At Brandeis University, JLIB has been in used in an undergraduate/graduate level Computer graphics course (CS155, Spring 1998), and an "Introduction to computers" course (CS2a, Autumn 1997, Autumn 1998) for non computer science majors. Over 1,000 applets have been developed by the students.

The SILK 2.0 version compiled Scheme syntactic expressions into Code objects that could be more efficiently evaluated. This version was started by Peter Norvig and completed by Tim Hickey.

Most recently, generic functions have been added as an extension.

3 Primitive access to Java is easy

Originally, SILK provided two primitive procedures for accessing Java behavior:
(constructor CLASSNAME ARGTYPE1 ...)
(method METHODNAME CLASSNAME ARGTYPE1 ...)
The (constructor) procedure is given a specification of a Java constructor, in terms of a class name and a list of argument type names, and returns a procedure implementing that constructor. The (method) is given a specification of a Java method, in terms of a method name, class name, and a list of argument type names, and returns a procedure implementing that method.

So, for example, we can use the java.math.BigInteger package to see if the number, 12345678987654321, is probably prime (with a probability of error of less than 1 part in 2 -10 when the result is #t):

1234567890123456789012345678901234567890123456789012345
> (define isProbablePrime
    (method "isProbablePrime" "java.math.BigInteger"
             "int"))
isProbablePrime
> (define BigInteger 
    (constructor "java.math.BigInteger"
                 "java.lang.String"))
BigInteger
> (isProbablePrime (BigInteger "12345678987654321") 10)
#f
> 
It is useful to have additional access to Java's reflection layer. Here we define the procedure (class) that returns a class object given its full name string:
1234567890123456789012345678901234567890123456789012345
> (define class (method "forName" "java.lang.Class" 
                        "java.lang.String"))
class
> (define HT (class "java.util.Hashtable"))
HT
> HT
class java.util.Hashtable
>
Here we define a procedure, (get-static-value) that given a class and a string naming a static field, returns the value of the corresponding static field. We then ask for the value of the TYPE field of the class Void. In Java, that would be simply Void.TYPE.
1234567890123456789012345678901234567890123456789012345
> (define get-field 
    (method "getField" "java.lang.Class" 
            "java.lang.String"))
get-field
> (define get-field-value 
    (method "get" "java.lang.reflect.Field" 
                  "java.lang.Object"))
get-field-value
>(define (get-static-value class field-name)
   (get-field-value (get-field class field-name) '()))
get-static-value
> (get-static-value (class "java.lang.Void") "TYPE")
void
>

5 But, procedures aren't generic enough

While JLIB is useful, its implementation reveals two problems with SILK's access to Java:
  1. Procedures are not generic. Procedures must be carefully named apart to avoid name conflicts. There are many potential conflicts, such as:
  2. For each Java method or constructor one needs, it must be named and defined separately. Even if only a few methods of a class are used, this can be a fair amount of work.
An obvious Scheme solution to this problem is to do type tests on the arguments to choose the appropriate method to invoke. For example, a (get) procedure that works both on classes Hashtable and Field might look something like:
1234567890123456789012345678901234567890123456789012345
(define get
  (let ((hashtable-class (class "java.util.Hashtable"))
        (hashtable-get-method 
         (method "get" "java.util.Hashtable"
                 "java.lang.Object"))
        (field-class (class "java.lang.reflect.Field"))
        (field-get-method 
         (method "get" "java.lang.reflect.Field" 
                 "java.lang.Object"))
        (isInstance 
         (method "isInstance" "java.lang.Class" 
                 "java.lang.Object")))
    (lambda (table key)
      ((cond
        ((isInstance hashtable-class table)
         hashtable-get-method)
        ((isInstance field-class table)
         field-get-method)
        (else (lambda (table key)
                (error "no method for " table))))
       table key))))
Macros can help generate such code automatically and we used that approach for a while.

5.2 (import) lifts Java classes into SILK wholesale

Now, an (import) function is used to import the static and instance methods of a class. The goal was to make this similar to the import statement in Java. Here we import Hashtable:

1234567890123456789012345678901234567890123456789012345
> (import "java.util.Hashtable")
importing java.util.Hashtable in 161 ms.
#t
> 
The result of the (import) is that: We can immediately start using Hashtables:
1234567890123456789012345678901234567890123456789012345
> (define ht (Hashtable 20))
ht
> ht
{}
> (put ht 'clone 1)
()
> ht
{clone=1}
> (put ht 'zone 2)
()
> ht
{clone=1, zone=2}
> (get ht 'clone)
1
>
The procedure (import) creates generic functions. For example, (get) is a generic function with three methods:
1234567890123456789012345678901234567890123456789012345
> get
{silk.Generic get[3]}
> (for-each print (methods get))
{silk.InstanceMethod Object Map.get(Object)}
{silk.InstanceMethod Object silk.GlobalEnv.get(Symbol)}
{silk.InstanceMethod Object Field.get(Object)}
#t
>
Here's an example with static methods:
> (import "java.lang.Float")
importing java.lang.Float in 81 ms.
#t
> (Float.parseFloat "17.42")
17.42
>
When a class is imported, a Name.class variable is bound to the class:
> Hashtable.class
class java.util.Hashtable
>

5.3 Here's an example of using (import)

To show what programming with (import) and generic functions is like, here's a simple applet:

To use it, type a number into the second textfield. The top textfield contains the current estimate of its square root. Each time the "iterate" button is pressed, an iteration of Newton's method is performed to better approximate the square root.

Here's the code. The EasyWin class is borrowed from JLIB.

1234567890123456789012345678901234567890123456789012345
(import "java.lang.Double")
(import "java.awt.Color")
(import "java.awt.Button")
(import "java.awt.TextField")
(import "jlib.EasyWin")

(define (test1)
  ;; Construct a test window.
  (let ((win (EasyWin "test1.scm" this-interpreter))
        (g (TextField "1" 20))
        (x (TextField "16" 20))
        (go (Button "Iterate")))
    (define (action e)           ; Define call back.
      (setText g
               (toString
                (f 
                 (Double.valueOf (getText g))
                 (Double.valueOf (getText x))
                 ))))
    (define (f g x)              ; Newton's method.
      (/ (+ g (/ x g)) 2.0))
    (resize win 200 200)         ; Size the window.
    (add win g)                  ; Add the components.
    (add win x)
    (add win go)
    (addActionCallback win action)
    (setBackground win (Color 200 200 255))
    (show win)))

(test1)                          ; Try it out.

6 The generic function protocol is simple

Compared to Common Lisp, or even Tiny CLOS, the SILK generic function protocol is simple. Because the protocol is defined in terms of non generic Java methods, metacircularity is not an issue [GK]. Here are the essential abstract classes and methods:
public abstract class Procedure extends SchemeUtils {
  // Apply the Procedure to a list of arguments.
  public abstract Object apply(Pair args, Engine eng);
}

public class Generic extends Procedure {
  // Add a method to the generic.
  public void addMethod(GenericMethod m); 
}

public abstract class GenericMethod extends Procedure {
  // Two GenericMethod's match if they have equal 
  // lists of parameterTypes.
  public boolean match(GenericMethod x);

  // Is the method applicable to the list of arguments?
  public boolean isApplicable(Pair args);

  // Returns the method that is more applicable.
  public GenericMethod moreApplicableMethod
                        (GenericMethod m2);
}

A Procedure is an object that can be applied to a list of arguments. It is the basis for function calling in SILK. SchemeUtil simply provides many convenient utilities (a common Java idiom). Engine is used to execute code fragments that allows tail call optimization and need not be considered further here.

A Generic is a Procedure defined in terms of a set of GenericMethod's. The addMethod() method is used to add a GenericMethod to the set. No two GenericMethod's in the set are allowed to match().

Its apply() method must choose the most applicable GenericMethod to invoke and apply it to the list of arguments. The isApplicable() and moreAppllicableMethod() mehods of GenericMethod are used to make this choice.

There are currently four classes of GenericMethod. ConstructorMethod, StaticMethod, and InstanceMethod are wrapper classes for each of the corresponding Java metaobjects. SchemeMethod is used to define methods who's behavior is written in Scheme.

Since (import) assigns constructors, static methods and instance methods to different generic functions, the methods in a generic function tend to be of only one type. However, a generic function can have any subclass of GenericMethod. This allows Scheme methods to be added to any Generic, for example.

Choosing the applicable method

To choose the applicable method, we follow Java semantics closely. This may seem suprising since Java chooses a method based on the dynamic type of its first argument, and the static (compile time) type of its other arguments. SILK simply makes this choice at runtime based on the types of arguments passed to the Generic.

For example, here is what the (list#) Generic looks like after importing java.io.file and javax.swing.JFrame:

1234567890123456789012345678901234567890123456789012345
> list#
{silk.Generic list#[7]}
> (for-each print (methods list#))
{InstanceMethod String[] File.list(FilenameFilter)}
{InstanceMethod String[] File.list()}
{InstanceMethod void Component.list(PrintStream, int)}
{InstanceMethod void Component.list(PrintWriter, int)}
{InstanceMethod void Component.list()}
{InstanceMethod void Component.list(PrintWriter)}
{InstanceMethod void Component.list(PrintStream)}

To list the contents of a directory, the second method would be chosen:

1234567890123456789012345678901234567890123456789012345
> (define f (File "d:/java/jlib3/src/silk/"))
f
> (for-each* print (list# f))
Closure.java
Code.java
ConstructorMethod.java
...
>

The only complication with this approach is Scheme datatypes must be mapped to an appropriate Java type during method invocation. Currently, there are two issues.

The first is that SILK's numeric types include only Integer and Double. So, methods such as java.util.Hashtable(int, float) can't be invoked directly. One must first convert the required float, using (Float 0.75) for example. While SILK does not use Float or Long objects, once such objects are constructed it treats them as normal numbers.

The second issue that both SILK symbols and strings (represented as Java char[]) are mapped to class String during method invocation. This can lead to an ambiguity, such as in the append() method of java.lang.StringBuffer:

1234567890123456789012345678901234567890123456789012345
> (import "java.lang.StringBuffer")
importing java.lang.StringBuffer in 70 ms.
#t
> append#
{silk.Generic append#[10]}
> (for-each print (methods append#))
{InstanceMethod StringBuffer.append(char[], int, int)}
{InstanceMethod StringBuffer.append(char[])}   ; ***
{InstanceMethod StringBuffer.append(boolean)}
{InstanceMethod StringBuffer.append(String)}   ; ***
{InstanceMethod StringBuffer.append(Object)}
{InstanceMethod StringBuffer.append(char)}
{InstanceMethod StringBuffer.append(long)}
{InstanceMethod StringBuffer.append(int)}
{InstanceMethod StringBuffer.append(float)}
{InstanceMethod StringBuffer.append(double)}
Since this Generic has methods on both String and char[], SILK can't decide which to use. In such a case, the user must invoke a particular method using (method).

6.1 Use only the most general method

To minimize the method lookup overhead, for Java instance methods we let Java's single argument dispatch do most of the work. To do that, we only store the most general Java methods in a generic function. So, for example, for the generic function (toString) we only need the Object.toString() method: 1234567890123456789012345678901234567890123456789012345

> (methods toString)
({InstanceMethod String Object.toString()})
> 
We call such a method, a "most general method".

The feasibility of this approach was studied using the 494 classes reachable from the class javax.swing.Jframe. Here are some statistics:

1234567890123456789012345678901234567890123456789012345
Count What
 494 classes
 458 public classes
  52 Exception classes
 176 static most general methods
2759 instance most general methods.
2935 total most general methods.
There were 134 generic functions that contain only static methods. 93% of them have two or fewer methods:
1234567890123456789012345678901234567890123456789012345
  Count Methods Cumulative % and examples
    110       1 82.1%
     15       2 93.3%
      6       3
      1       4 createPackedRaster
      1       5 getKeyStroke
      1       9 valueOf
The 2,759 most general instance methods fall into 1525 generic functions. 91% of these have three or fewer methods:
1234567890123456789012345678901234567890123456789012345
  Count Methods Cumulative % and examples
   1058       1  69.4%
    255       2  86.1%
     75       3  91.0%
     39       4
     24       5
     23       6
     17       7
     10       8
      6       9
      1      10
      5      11
      1      12
      1      13
      2      16
      1      17 get
      1      18 contains
      3      20 clone insert print
      1      22 println
      1      24 remove
      1      36 add
So, most generic functions have one or two methods so our discriminator approach favors such situations.

Methods are only added to a generic function when a class is imported. So, the above statistics reflect what you get if you imported all 494 classes. The number of actual methods is likely to be substantially lower than this. For example, when using only javax.swing.JFrame, (add) only has six methods, not 36.

6.2 A few discriminator states are adequate

Since the number of methods per generic function tends to be small, we focus on optimizing the method lookup for such cases. We use a discriminator function with a small number of states. The state of the discriminator is recomputed whenever a method is added to the generic function. The states are chosen based on the static statistical analysis above. In contrast, Common Lisp chooses discriminator states dynamically so performance is adapted to each run of an application [KR]. Here is a description of the states:

1 NOMETHODS
No methods, an error occurs if invoked. The discriminator is in this state when the generic function is first consructed, before any methods have been added to it.
2 ONEMETHOD
Simply invoke the method.
3 TWOMETHODINSTANCEFIXED
Two instance methods with the same number of arguments. Check the first argument of the first method. If it is appliable, apply it, otherwise apply the second method. This works because the types of the first arguments are disjoint.
4 TWOMETHODNOTFIXED
Two methods of any type with different numbers of arguments. Choose the method based on the number of arguments.
5 GENERAL
Most general lookup. Compute the most applicable method based on all of the arguments of all methods.
For the most likely case of a generic function only having one or two methods, discrimination is little more than a switch jump and a subclass test or two. For cases where an error would occur, we simply invoke the wrong method and let Java signal the error.

Most of the cost of invoking a generic function is in crossing the Scheme/Java frontier to actually apply the method. Currently this involves converting a list of arguments from the Scheme side to an array of arguments on the Java side. String and symbol arguments are also converted to appropriate Java types. The return value must also be converted. For example, a boolean result must be inverted to either #t or #f.

Scheme methods

Besides using Java methods in generic functions, it is quite useful to define methods directly in Scheme, using the (define-method) macro:

(define-method name ((arg class) ...) 
  (form) ..)
Where class names a Java class.

For example, here we define the generic function (iterate collection action) that maps the function action over the elements of collection:

1234567890123456789012345678901234567890123456789012345
(import "java.util.Iterator")
(import "java.util.Collection")
(import "java.util.Map")
(import "java.util.Vector")
(import "java.util.Hashtable")

(define-method iterate ((items java.util.Iterator) 
                        action)
  (if (hasNext items)
      (begin (action (next items))
             (iterate items action))))

(define-method iterate ((items java.util.Collection)
                        action)
  (iterate (iterator items) action))

(define-method iterate ((items java.util.Map) action)
  (iterate (entrySet items) action))

(define-method iterate ((items silk.Pair) action)
  (action (car items))
  (let ((items (cdr items)))
    (if (pair? items) (iterate items action))))

(define-method iterate ((items java.lang.Object[])
                        action)
  (let loop ((i 0) 
             (L (vector-length items)))
    (if (< i L) (begin (action (vector-ref items i))
                       (loop (+ i 1) L)))))

(define-method iterate ((items java.lang.Object[])
                        action)
  (let loop ((i 0) 
             (L (vector-length items)))
    (if (< i L) (begin (action (vector-ref items i))
                       (loop (+ i 1) L)))))

The collection argument can be an array, a Scheme list (of type silk.Pair), or any of the Java 1.2 collection types. This type of integration is not easy in Java because new instance methods cannot be added to existing classes. Here's an example use:

> (define h (Hashtable 50))
h
> (put h 'fred 3)
()
> (put h 'mary 4)
()
> (iterate h print)
fred=3
mary=4
()
> 

Scheme methods are essentially treated like Java static methods (of class silk.StaticMethod). Currently, there is no provision for (call-next-method). One must invoke such a method directly using (method).

Java classes can be defined directly in Scheme using a (define-class) macro, using the compiling technique described below. Such classes only have constructor and field accessor methods. Scheme methods can be added to the class using (define-method).

7 SILK creates new code from old

It should be clear that SILK fulfills one of Scheme's important roles as an embedded scripting and prototyping language [BB]. Perhaps a greater strength is that Scheme can be used as a compile time scripting language to generate new code, perhaps in another language, such as C or Java. Two examples of this are described in references [BW] and [BF] where Scheme is used to set up a complex numerical problem such as a complex computer visualization. Partial evaluation is then used to generate an efficient algorithm to compute the solution of the problem, in C.

Java development environments, such as Sun's Java Bean box, compile small glue classes automatically. This generated code usually follows a standard template and requires some introspection of an existing class. We can do the same thing in SILK. Essentially, Scheme becomes a macro language for Java.

For example, the normal Java runtime environment does not provide a way to trace individual methods. Here we sketch how to add such tracing capability. The basic idea is to generate a subclass of an existing class which allows its methods to be traced. If SILK has enough access to the Java application, an instance of this traceable class can be substituted into the application without changing any Java code. As part of this process, we would have a method, m2, say the method Hashtable.put(), and generate a traced method from it using (gen-trace-method m2):

1234567890123456789012345678901234567890123456789012345
> m2
public synchronized Object Hashtable.put(Object,
                                         Object)
> (emit (gen-trace-method m2))
public synchronized java.lang.Object 
    put(java.lang.Object a1, java.lang.Object a0) {
  if(trace) Trace.enter(this + ".put(" + a1 + ", " + 
                        a0 + "}");
  java.lang.Object result = super.put(a1, a0);
  if(trace) Trace.exit(result);
  return result;
  }
#t
Here's some Scheme code that does this:
(define-method method-return-type
  (m java.lang.reflect.Method)
  (getName (getReturnType m)))

(define-method gen-trace-method
  (m java.lang.reflect.Method)
  `(,(gen-method-signature m) 
    { ,(gen-trace-method-body m) }))
  
(define-method gen-method-signature 
  (m java.lang.reflect.Method)
  `(,(Modifier.toString (getModifiers m)) 
    ,(method-return-type m)
    ,(getName m) "(" 
    ,(map-args 
      (lambda (arg) `(,(arg-type arg) ,(arg-name arg)))
      "," 
      (arg-types m))
    ")"))
        
(define-method gen-trace-method-body
  (m java.lang.reflect.Method)
  `(if "(" trace ")" 
       Trace.enter "(" this + 
       ,(quotify "." (getName m) "(")
       ,(map-args (lambda (arg) `(+ ,(arg-name arg)))
                  "+ \", \"" 
                  (arg-types m))
       + ,(quotify "}") ")" ";"
       ,(method-return-type m) result = super. 
       ,(getName m) "("
       ,(map-args arg-name "," (arg-types m))
       ")" ";"
       if "(" trace ")" "Trace.exit(result)" ";"
       return result ";"))

(define arg-type car)            ; Accessors
(define arg-name cadr)
(define (arg-types m)            
  ;; method -> ((type name) ...)
  (let ((c 0))
    (map* 
     (lambda (a) 
       (list (getName a)
             (make-arg-name (set! c (+ c 1)))))
     (getParameterTypes m))))

(define (make-arg-name c)
  ;; (make-arg-name 1) -> "a1"
  (string->symbol
   (string-append "a" (number->string c))))

(define (quotify . args)
  ;; (quotify 'fred 3) -> "\"fred3\""
  (let ((double-quote "\""))
    (apply string-append double-quote
           (append args (list double-quote)))))
1234567890123456789012345678901234567890123456789012345
This code generation system is extremely basic. (emit) takes a list structure, produced by (gen-trace-method)here, and formats it into readable Java code. Scheme's backquote macro characters, `(, ,@) are used to contruct the necessary list structures. (map-args) is like (map) but adds a separator between each argument, to generate a comma separated argument list, for example.

SILK can automatically compile Java files using:

(define (compile-file file)
  ;; Compile the *.java file, file, 
  ;; using the current CLASSPATH.
  (let ((as (Array.newInstance String.class 3))
        (main (Main (get-field System.class 'out) 
                    'silkc)))
    (vector-set! as 0 "-classpath")
    (vector-set! as 1 (System.getProperty
                       "java.class.path"))
    (vector-set! as 2 file)
    (compile main as)))
To compile and load the new traced class we just do:
1234567890123456789012345678901234567890123456789012345
> (compile-file "elf/TracedHashtable.java")
#t
>
In this simple example, Java's Method metaclass was used directly to generate the traced code. A more realistic example would provide a compile time metaobject protocol that would be used to do code generation more formally. While this example is simple, it should be clear that SILK can become a useful compile time software development environment without a substantial amount of work.

Related work

5.1 Skij has an interesting solution

Skij [MT] is an interesting Scheme dialect that is similar to SILK, but much more designed as a Java scripting language, than as a Scheme implementation. It provides generic operations, (new), (invoke), and (invoke-static).

For example, here is how it would create a hashtable and access it, and invoke a static method:

1234567890123456789012345678901234567890123456789012345
(define ht (new 'java.lang.Hashtable 100))

(invoke 'put ht "hi" "bye")
(invoke 'get ht "hi")
(invoke-static 'java.lang.Class 'forName
               'java.lang.Hashtable)
While the Skij solution works for incorporating Java methods into Skij, Scheme procedures are still not generic. Using the keywords new, invoke, and invoke-static seemed verbose, since they would be used often. Also, a straight forward implemenation would require the method name as well as the arugments to be used during method selection. With a generic function approach the method name is effectively curried into the generic function.

8 Now, let the past reflect on the future

We have tried to show that SILK makes effective use of Java's reflective capabilites. However, even though we were well aware of reflection from the beginning, SILK was originally developed using a relatively traditional approach; one that you might use to implement a Scheme in a non-reflective language, such as C or Scheme. First the essentials of a runtime environment were defined, then the essential Scheme primitives were written in terms of the capabilities of the runtime environment. After that, the Scheme listener is written to bootstrap itself into existence.

The result is a Scheme dialect that is relatively compact, though it has grown over the past 18 months. Even so, it still fits in a 76KB jar file. The following exhibit shows statistics from Scheme dialects we know of.

Scheme implementation statistics
Implementation java files lines Scheme files lines
Silk 1.0 12 1905 0 0
Silk 2.0 20 2778 0 0
Generic Silk 2.0 28 3508 5 510
Skij [MT] 27 2523 44 2844
Jaja [CQ] 66 5760 ? ?
Kawa [PB] 273 16629 14 708

About a third of the code, 988 lines that relate to 191 Scheme primitives, can be generated by Scheme.

We speculate that by making a more determined effort to use reflection as a building material, SILK could be even smaller. Imagine a runtime environment with the following characteristics:

Since, by definition, generic functions do argument type checking, the type check code of the previous SILK kernel is unnecessary.

Now, we can use Java as a supermarket adding its methods to generic functions to create the Scheme environment we want. For example, if we are willing to accept Java's String class as the representation of a Scheme string, we can simply load the corresponding methods into the appropriately named generics and we are in business. Making this implementation decision as Skij does, simply means that only the (string-set!) procedure is no longer available. Scheme must provide some classes, such as Pair, but their constructors and methods become become the obvious procedures, (cons), (car), and (cdr). Some primitives may require type conversion, but they can be constructed from functional composition with functions in the runtime base.

The size and performance characteristics of a Scheme dialect built this way would certainly be different than SILK's. But, in either case, most of the performance is lost jumping across the Scheme/Java boundary so the differences are likely to be small.

Performance can be regained using the compile time technique described above. Once a set of generic functions have been identifed, their contents can be crystalized into a Java class that constructs them when the Scheme application starts.

So, in summary, there is a reflective bootstrapping process where a Scheme runtime enviornment grazes over the Java runtime environment to construct the behavior it wants. Then, introspection of the Scheme environment is used to compile it into a brick of Java code that is the Scheme environment users actually see over the web.

Thus by proper use of reflection, software gets smaller and faster.

Acknowledgments

The authors wish to thank Peter Norvig for developing the first version of SILK and for keeping it simple enough that we could easily use and extend it. We thank Geoffrey S. Knauth, Richard Shapiro, Tom Mitchell, Bruce Roberts, and Jeff Tustin for reviewing drafts of this paper. We thank Rusty Bobrow who thought SILK was simple enough that he suggested it to his daughter for programming a High School science project. And we'd like to thank Rusty's brother, Danny Bobrow for long ago teaching us that "Meta is Beta!". That two generations of one family use reflection, reflects well on reflection.

References

[BF] Clifford Beshers Steven Feiner, Generating efficient virtual worlds for visualization using partial evaluation and dynamic compilation, ACM SIGPLAN Symposium on Partial Evaluation and Semantics-Based Program Manipulation (PEPM'97), p. 107-115.

[4T] Aubrey Jaffer, r4rstest.scm, ftp://ftp-swiss.ai.mit.edu/pub/scm/r4rstest.scm.

[BW] A. Berlin and D. Weise, Compiling scientific code using partial evaluation. Technical Report CSL-TR 90-422, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1990.

[GK] Gregor Kiczales, Tiny CLOS, file://parcftp.xerox.com/pub/mops/tiny/

[KR] Gregor Kiczales and Luis Rodriguez, Efficient method dispatch in PCL, proceedings 1990 ACM Conference on LISP and Functional Programming, Nice, France, June 1990, 99-105.

[PB] Per Bothner, Kawa the Java-based Scheme System, http://www.cygnus.com/~bothner/kawa.html

[PN] Peter Norvig, SILK: Scheme in Fifty K, http://www.norvig.com/SILK.html

[TH] Tim Hickey, JLIB: A Declarative GUI-building library for SILK, http://www.cs.brandeis.edu/~tim/Packages/jlib/jlib.html

[MT] Mike Travers, Skij, IBM alphaWorks archive, http://www.alphaworks.ibm.com/formula/Skij

[CQ] Christian Queinnec, JaJa: Scheme in Java, http://www-spi.lip6.fr/~queinnec/WWW/Jaja.html

[R5] Richard Kelsey, William Clinger, and Jonathan Rees (Editors), Revised(5) Report on the Algorithmic Language Scheme, http://www-swiss.ai.mit.edu/~jaffer/r5rs_toc.html


[BB] Brian Beckman, A scheme for little languages in interactive graphics, Software Practice and Experience, 21, 2, p. 187-208, Feb, 1991.

[CQ1] Christian Queinnec, Fast and compact dispatching for dynamic object-oriented languages, May, 1997.

[ROS91] J. Rose, A minimal metaobject protocol for dynamic dispatch. In Proceedings of the OOPSLA '91 Workshop on Reflection and Metalevel Architectures in Object-Oriented Programming, October 1991.

[HC92] Shih-Kun Huang and Deng-Jyi Chen, Efficient algorithms for method dispatch in object-oriented programming systems, Journal of Object-Oriented Programming, 5(5):43-54, September 1992.

[AGS94] E.Amiel, O. Gruber, and E. Simon, Optimmizing multi-method dispatch using compressed dispatch tables. In OOPSLA '94, October 1994.

[VH94] Jan Vitek and R. Nigel Horspool, Taming message passing: Efficien method lookup for dynamically typed languages. ECOOP '94 - 8th European Conference on Object-Oriented Programming, Bologna (Italy), 1994.

[CT95] Weimin Chen and Volker Turau, Multiple-dispatching base on automata, Theory and Practice of Object Systems, 1(1):41-60, 1995.

[Java] Java spec reference?