What is XTRAN?

 What Can You Do with XTRAN?

XTRAN is capable of automating any software engineering task that can be described in its powerful rules language.  We divide such tasks into several broad categories:  Analysis, re-engineering, translation and code generation, and text processing.

Analysis

For analysis, we configure XTRAN with one or more input language parsers but no output language renderers.  XTRAN's rules language provides sophisticated analysis of any language supported by an XTRAN parser.  This powerful analysis capability can also produce documentation and program descriptive information suitable for input into CASE and modeling systems.  In fact, XTRAN's analysis capability, specified through its rules language, allows you to extract any information that is present in the code, at any level of detail or abstraction, and in any form that's required.

XTRAN is usually run on a single module at a time.  However, its ability to persist information across runs allows the collection and reporting of system-wide information.  This is a frequently used approach to global analysis of a software system using XTRAN.  We often write XTRAN analysis rules that append information from each module to a file, then digest and report the accumulated information.

Analysis with XTRAN can be either production analysis, as part of the normal software engineering cycle, or ad hoc analysis needed to address specific issues.

XTRAN analysis is especially useful in assessing a body of code in terms of:

• Code quality (cyclomatic complexity, structuring, etc.)
• Modularity
• Architecture
• Calling tree
• "Include" dependencies
• Symbol usage (global, local, or both)
• Cloned code
• Dead code
• Translatability
• Portability
• Style
• Conformance to coding standards
• Code defects that can be identified using pattern matching and/or rules
• Code visualization, e.g. as HTML with color coding of statement types
• Forensic code analysis
• Anything you can think of!

Such an assessment is an important prerequisite to any migration or re-engineering project.  In addition, ongoing assessment of code provides to programming management the information it needs to keep software development on the right track.

Re-Engineering

For re-engineering and code standardization, we configure XTRAN with a parser and a renderer for the same language.  You can then use XTRAN's powerful rules language to apply, across a body of code, any set of systematic changes that can be specified using rules.  A few examples include:

• Enforce programming standards.
  
  
• Structure the code by automatically eliminating "gotos" in many cases, substituting functionally equivalent "if", "else", "for", "do", and "while" constructions.
  
  
• Find operating system dependencies and change them to a different operating system or standard library such as Posix.
  
  
• Change code to use a different API, such as a graphics library or DBMS.
  
  
• Find common low-level patterns of language usage and decompile them into higher-level constructions, thereby raising the level of the code.
  
  
• Requalify structure members to reflect changes to a structure's definition.
  
  
• Make large numbers of changes in symbol names.

Since XTRAN makes the changes to the code's XIR, which it then renders on output, the changed code is automatically restyled in terms of indentation, curly braces, line breaking, comment tabbing, etc.  Since styling parameters are under user control, you can use XTRAN to restyle code as desired.

Translation, Code Generation

For translation or code generation, we configure XTRAN with one or more input language parsers and one or more output language renderers.  (You can also use XTRAN's powerful re-engineering capabilities in conjunction with translation; see below for examples.)

Translation combinations we have implemented with XTRAN include:

• Encore (SEL, Gould) 32 assembler to C
• Fortran to C and C++
• HP (Digital, Compaq) PDP-11 MACRO-11 assembler to C
• HP (Digital, Compaq) VAX MACRO assembler to C
• IBM PL.8 / PL.9 / PL/ix to C++
• IBM Series/1 EDL to C
• IBM Series/1 assembler to C
• Intel PL/M to C
• Intel x86 assembler to C
• Motorola 680x assembler to Texas Instruments TMS370 assembler
• NEC 78C10 assembler to C
• Norsk Data NPL to C
• Pascal to C
• PL/I to C

Text Processing

For text processing, we configure XTRAN with only a meta-code parser, and no language renderers.  You can then use the powerful text manipulation capabilities of XTRAN's rules language.  The rules language allows you to read and write text files as desired.

XTRAN's regular expression and delimited list manipulation capabilities, along with the other capabilities of its rules language, make it an extremely powerful text processor.

Combining XTRAN's Capabilities

Automating a complex software engineering task often requires a combination of XTRAN's analysis, re-engineering, translation, code generation, and text processing capabilities.

XTRAN and Legacy Modernization

Many existing legacy software systems represent major investments that must be modernized in order to provide the agility required to remain competitive:

• Allow the use of modern software development tools, to increase the productivity of the software development department
  
  
• Attract and keep the best programmers, to supplement existing IT staff
  
  
• Improve the quality of the code, to reduce maintenance costs and allow timely enhancements of the system
  
  
• Rationalize disparate systems, to make them work together better and to provide a common software engineering language and platform
  
  
• Prepare systems for transition to a Service Oriented Architecture (SOA), and automate that transition

The best modernization strategy will likely involve one or more of the following alternatives:

• Re-engineer the existing code, to improve its quality, repurpose it, and rearchitect it for modern use
  
  
• Replace the code with commercial off-the-shelf software ("COTS")
  
  
• Totally reimplement the application from the ground up

When it comes time to modernize your legacy applications, XTRAN can play a vital role in automating virtually every aspect of the modernization process.  Achieving a high level of software engineering automation is critical to the success of the project, to reduce the number of bugs introduced by the modernization process, and ultimately, to reduce the risk of failure.

• Automated Analysis
  
  
  • Verify the accuracy of an existing functional specification against the code, or, in the worst case, extract such a specification from the code itself.  Such a specification is required in order to determine the best modernization strategy and then implement that strategy.
    
    
  • Determine the quality of the legacy code, to decide if it is worth re-engineering, and assess the need for quality improvement prior to modernization.
    
    
  • Find and extract business rules from the code.  If you decide to replace the application with COTS or to totally reimplement it, you must know what business rules to implement in the replacement.  If you decide to re-engineer, you will need to know what the business rules are and where they are, so they can be exposed as services to be reused.
    
    
  • Assess the impact of a legacy modernization project on the code body to be modernized.  For example, if you anticipate a change to a DBMS or other third-party product, all calls to the product's API may have to be changed.  Analysis with XTRAN can find and catalog information about all such calls, and assist in determining the best strategy for automating the changes.
    
    
  • Assess the impact of a port on the code body to be ported.  For example, all operating system dependencies may have to be changed.  Analysis with XTRAN can find and catalog information about all such dependencies, and assist in determining the best strategy for automating the changes.
    
    
  • Assess the quality and adherence to coding standards of the re-engineered or ported code, on an ongoing basis.
    
    
  • Answer specific questions about what's in the code, on an ad hoc basis.
    
    
• Automated Re-Engineering
  
  
  • Improve the quality and maintainability of the code, and raise its level, prior to modernization.  This can significantly reduce the negative impact of a modernization effort, as well as improving the quality of the result.
    
    
  • Expose the business rules in the code as services to be reused.  This can include extracting such rules as components.
    
    
  • Implement an API change, for example a change to a different DBMS or other third-party software product.  This can involve changing every call to the product's API; XTRAN rules for automating such changes can actually take advantage of the API usage information catalogued during the analysis phase.
    
    
  • Implement specific changes to the code, on an ad hoc basis.
    
    
  • Move disparate systems to a common platform, for easier maintenance and sharing of skills and code.
    
    
• Automated Translation
  
  
  • Move legacy code from a proprietary language to an open language, in order to reduce dependence on a specific vendor.
    
    
  • Move legacy code from an obsolete language, for which it is increasingly difficult to find experienced programmers and modern development tools, to a modern language for which programmers and modern tools are available.
    
    
  • Move legacy code from a non-portable language to a portable language, in order to move the application from an older platform, with low price/performance and increasingly higher maintenance costs and difficulty, to a modern, cost-effective platform with higher price/performance and lower maintenance costs.
    
    
  • Move disparate systems to a common programming language, for easier maintenance and sharing of skills and code.

XTRAN and Forensic Code Analysis

In legal proceedings involving computer code, it is sometimes necessary to analyze that code to determine its implications for the legal case.  XTRAN's powerful analysis capabilities are ideal for this, and can be "tuned" to specific requirements using XTRAN's rules language.

Forensic code analysis may involve:

• Analyzing the code's architecture.  XTRAN can provide this, at any level of abstraction or detail.
  
  
• Determining the code's quality.  XTRAN provides a number of popular quality measures, and others can be added as needed.
  
  
• Comparing two bodies of code to determine if one was copied from the other.  XTRAN provides powerful code comparison capabilities, which can be "tuned" to any level of detail.
  
  
• Determining whether the code satisfies contractual requirements.  XTRAN can be "tuned" to search the code for constructs that imply either satisfaction of, or failure to satisfy, such requirements.
  
  
• Determining whether contractually required documentation accurately describes the code.  XTRAN can be "tuned" to search for constructs that verify, or fail to verify, the accuracy of the documentation.
  
  
• Determining whether the code contains any "back doors", "trap doors", or "bombs".  XTRAN's pattern matching facilities can be used to look for common patterns that imply such problems in the code.

One problem commonly encountered with such analysis is the sheer bulk of code that must be analyzed, often within a limited amount of time.  XTRAN provides the automation to reduce this problem to manageable proportions, saving both time and money.

Additional XTRAN Applications

XTRAN's capabilities are applicable to a wide variety of additional problems, including Euro, code dialect translations, CASE tool interfaces, verification and enforcement of coding standards and styles, programming training and tutoring, and other rule-driven manipulation of computer languages and text data.

Ultimately, the only limit to the ways you can use XTRAN is your imagination.

 What Computer Languages Does XTRAN Handle?

XTRAN currently accommodates a wide variety of computer languages, including:

• Assemblers (2GLs)
  • Encore (SEL, Gould) 32
  • HP (Digital, Compaq) PDP-11 MACRO-11
  • HP (Digital, Compaq) VAX MACRO
  • IBM mainframe (360, 370)
  • IBM Series/1
  • Intel x86
  • Motorola 680x
  • NEC 78C10
  • Texas Instruments TMS370
• 3^rd generation languages (3GLs)
  • C, including K&R and ANSI dialects
  • C++
  • COBOL
  • Fortran, including IBM, Mark III, and VMS dialects
  • Pascal, including HP (Apollo) Domain, IBM, Microsoft, OmegaSoft, and VMS dialects
  • PL/I, including IBM and VMS dialects
• 4^th generation languages (4GLs)
  • Adabas Natural
  • RPG
  • SAS
• Proprietary languages
  • IBM Series/1 EDL
  • IBM PL.8 / PL.9 / PL/ix
  • Intel PL/M
  • Norsk Data NPL
• Markup languages
  • HTML
• Meta-data languages
  • XML

 XTRAN's Architecture

XTRAN consists of:

• A powerful and sophisticated rules language, which we call meta-language or meta-code because it is used to define and manipulate other languages.
  
  
• A language-independent inference engine for evaluating rules written in meta-code.
  
  
• Language-specific parsers that read computer language text files and convert them to:
  
  
• A proprietary XTRAN Internal Representation (XIR) that XTRAN uses to represent all computer languages (and meta-code) during its processing, and which meta-code manipulates as its data.
  
  
• Language-specific renderers that render and output the processed XIR as computer language text files.

Here's a graphical look at XTRAN's code and data architecture:

The left side represents XTRAN's code, and reflects the sequence of phases XTRAN executes when it runs.  The right side represents the data that XTRAN keeps in memory as it runs.  The arrows indicate the production and consumption of XIR by XTRAN's various execution phases.

The dark blue parts of the code (language parsers and renderers) are language-specific; the remainder of the code, and XTRAN's rules language, are language-independent.

This unique architecture means that a new language (or language combination) requires only the development of a parser and/or renderer, plus a set of rules, in order to apply the power of XTRAN to a new language manipulation problem.

All of the XIR in XTRAN is organized into code trees.  XTRAN has many built-in code trees; you can also create and use your own.

Internally, XTRAN is highly object-oriented.  "Computer language" is a data class, as are "parser" and "renderer".  XTRAN can be configured with multiple parsers and/or renderers as needed, to handle a mixture of input and/or output languages.  (XTRAN is always configured with at least a parser for its own rules language.)

 XTRAN's Rules Language

XTRAN's powerful rules language (which we call meta-language or meta-code) is an evaluated, interpretive language.  Its syntax is like C, but its semantics are more like Lisp.  It includes meta-statements, meta-expressions, meta-variables, meta-functions, and meta-comments.

Interpretive evaluation of meta-code is so fast that it allows processing of very large amounts of code in reasonable time.  For example, we recently performed substantial re-engineering and analysis, involving intensive XTRAN rules evaluation, on more than 600,000 code lines of PL/I in just over four hours on a laptop computer.

The many capabilities offered by XTRAN through its rules language include:

 XTRAN's Pattern Matching and Replacement Facilities

XTRAN provides, via its rules language, an extremely powerful suite of pattern matching and replacement facilities you can apply to XIR at both the statement and expression levels:

• You can specify a pattern in a host language, in meta-code, or in a combination of both.
  
  
• Any element of a pattern can be "wild".
  
  
• You can optionally qualify each such "wild" element using an arbitrarily complex condition comprising additional meta-code to be evaluated for each match attempt.  Such a condition can, through navigation, explore the context in which the match is being attempted.
  
  
• XTRAN will, if requested, capture a copy of what matched each "wild" element, at match time.
  
  
• These copies can then be reported, reused later in the same pattern, or reused in a replacement pattern.
  
  
• Since pattern matching is done on XIR, it is totally independent of physical aspects of the code such as line breaks, indentation, and comment tabbing.

 XTRAN's Language Parsers and Renderers; XBNF

An XTRAN language parser has the job of reading a computer language's text source code and creating the XTRAN Internal Representation (XIR) that represents that code.  XTRAN's parsing engine allows construction of a parser in XTRAN's rules language (meta-code) by describing the grammar to be parsed, using a modified form of Backus-Naur Form (BNF) that we call XTRAN BNF (XBNF).  XBNF, as used for parsing, references a small number of hard-coded parsing primitives, and allows recursive productions.  You can construct additional parsing primitives in meta-code using XBNF.

Similarly, an XTRAN language renderer has the job of rendering a computer language's XIR as source code text and putting it out, including all of the code styling issues that implies.  XTRAN's language rendering engine allows construction of a renderer in meta-code by describing the grammar to be output, using a rendering version of XBNF.  XBNF, as used for rendering, references a small number of hard-coded output primitives, and allows recursive productions.  You can construct additional output primitives in meta-code using XBNF.

XBNF includes facilities for parsing and rendering nonpositional (free-format) languages, such as C and PL/I; positional (column-oriented) languages, such as RPG, job control languages, and some assemblers; and languages that are a mixture of positional and nonpositional, such as COBOL.

XTRAN's XBNF-driven parsing and rendering capabilities include fully integrated dialect control, allowing a parser or renderer to be conditioned on the language dialect being parsed.  This dialect control is dynamic, allowing you to switch dialects during parsing or rendering if needed.

Unlike "compiler compilers", which generate parsers as their output, XTRAN effectively executes XBNF dynamically at language parsing or rendering time, including a "fastback" parsing feature.  XTRAN provides an XBNF trace facility to help debug XBNF.

XBNF is integrated with meta-code, so a language parser or renderer can be written to be "tuned" with additional rules, dynamically if appropriate.

XTRAN's XBNF facilities are fairly recent, so for historical reasons, XTRAN's older language parsers and renderers are hard-coded.  However, they can be enhanced and/or overridden using parsing or rendering XBNF.

 @DBG:  XTRAN's Meta-Debugger

XTRAN includes a powerful, full-featured, interactive meta-debugger, called @DBG, which allows you to control XTRAN's execution and debug your meta-code.  @DBG's many features include:

• Breaks to @DBG on any of:
  
  
  • Reference to a specified source or target statement, optionally limited to one or more specified XTRAN processing phases (parse, process, etc.)
    
    
  • Evaluation of a specified meta-statement
    
    
  • Occurrence of a specified event
    
    
  • Start of a specified XTRAN processing phase
    
    
  • Reference to a specified symbol
    
    
  • Parse of a specified atom
    
    
  • Attempt to parse a specified XBNF text element
    
    
  • Attempt and/or success of a specified statement pattern match/replacement
    
    
  • Evaluation of XTRAN's built-in "break to @DBG" meta-function anywhere in the rules being evaluated, with optional conditionalization of the break and @DBG commands to be executed
    
    
• Break actions (@DBG commands to execute at time of break)
  
  
• Meta-variable value change watchpoints, with optional action such as a break to @DBG
  
  
• Meta-function call traceback
  
  
• Stepping of meta-code evaluation:  Over, into, and out of user meta-functions; to a specified meta-statement
  
  
• Dynamic meta-code evaluation at @DBG command level
  
  
• Command history
  
  
• Invocation of XTRAN's graphical, interactive XIR browsing mode
  
  
• Indirect @DBG command files

Many of these features are also accessible via XTRAN's command line flags or built-in meta-functions.

 XTRAN Documentation

When we license and deliver XTRAN, we provide with it a variant of the XTRAN User Manual appropriate to the licensed activity and computer languages.  The XTRAN User Manual comprises about 50,000 lines of HTML, organized into more than 60 chapters.  It provides a thorough reference to XTRAN, with many usage examples.

 Where did XTRAN Come From?

In 1983, we needed to port one of our products, XFORM, from the Digital VAX computer to the PC.  XFORM was originally written in Digital PDP-11 assembler, but we had previously translated it to VAX assembler, creating our CONPAX translation tool to automate the process.  (CONPAX was so successful at this that we brought it to market, and it helped over forty licensees translate millions of lines of assembler code.)  So the requirement was to translate the VAX assembler into C.

 Questions?  Comments?