How Nushell Code Gets Run

In Thinking in Nu, we encouraged you to "Think of Nushell as a compiled language" due to the way in which Nushell code is processed. We also covered several code examples that won't work in Nushell due that process.

The underlying reason for this is a strict separation of the parsing and evaluation stages that disallows eval-like functionality. In this section, we'll explain in detail what this means, why we're doing it, and what the implications are. The explanation aims to be as simple as possible, but it might help if you've programmed in another language before.

Interpreted vs. Compiled Languages

Interpreted Languages

Nushell, Python, and Bash (and many others) are "interpreted" languages.

Let's start with a simple "Hello, World!" Nushell program:

# hello.nu

print "Hello, World!"

Of course, this runs as expected using nu hello.nu. A similar program written in Python or Bash would look (and behave) nearly the same.

In "interpreted languages" code usually gets handled something like this:

Source Code → Interpreter → Result

Nushell follows this pattern, and its "Interpreter" is split into two parts:

1. Source Code → Parser → Intermediate Representation (IR)
2. IR → Evaluation Engine → Result

First, the source code is analyzed by the Parser and converted into an intermediate representation (IR), which in Nushell's case is just a collection of data structures. Then, these data structures are passed to the Engine for evaluation and output of the results.

This, as well, is common in interpreted languages. For example, Python's source code is typically converted into bytecode before evaluation.

Compiled Languages

On the other side are languages that are typically "compiled", such as C, C++, or Rust. For example, here's a simple "Hello, World!" in Rust:

// main.rs

fn main() {
    println!("Hello, World!");
}

To "run" this code, it must be:

1. Compiled into machine code instructions
2. The compilation results stored as a binary file on the disk

The first two steps are handled with rustc main.rs.

3. Then, to produce a result, you need to run the binary (./main), which passes the instructions to the CPU

So:

1. Source Code ⇒ Compiler ⇒ Machine Code
2. Machine Code ⇒ CPU ⇒ Result

Dynamic vs. Static Languages

Eval Function

Most dynamic, interpreted languages have an eval function. For example, Python eval (also, Python exec) or Bash eval.

The argument to an eval is "source code inside of source code", typically conditionally or dynamically computed. This means that, when an interpreted language encounters an eval in source code during Parse/Eval, it typically interrupts the normal Evaluation process to start a new Parse/Eval on the source code argument to the eval.

Here's a simple Python eval example to demonstrate this (potentially confusing!) concept:

# hello_eval.py

print("Hello, World!")
eval("print('Hello, Eval!')")

When you run the file (python hello_eval.py), you'll see two messages: "Hello, World!" and "Hello, Eval!". Here is what happens:

1. The entire program is Parsed
2. (Line 3) print("Hello, World!") is Evaluated
3. (Line 4) In order to evaluate eval("print('Hello, Eval!')"):
   1. print('Hello, Eval!') is Parsed
   2. print('Hello, Eval!') is Evaluated

Notice how the use of eval here adds a new "meta" step into the execution process. Instead of a single Parse/Eval, the eval creates additional, "recursive" Parse/Eval steps instead. This means that the bytecode produced by the Python interpreter can be further modified during the evaluation.

Nushell does not allow this.

As mentioned above, without an eval function to modify the bytecode during the interpretation process, there's very little difference (at a high level) between the Parse/Eval process of an interpreted language and that of the Compile/Run in compiled languages like C++ and Rust.

We'll dig deeper into what it means in the next section.

Implications

Consider this Python example:

exec("def hello(): print('Hello eval!')")
hello()

During interpretation:

1. The entire program is Parsed
2. In order to Evaluate Line 1:
   1. def hello(): print('Hello eval!') is Parsed
   2. def hello(): print('Hello eval!') is Evaluated
3. (Line 2) hello() is evaluated.

Note, that until step 2.2, the interpreter has no idea that a function hello even exists! This makes static analysis of dynamic languages challenging. In this example, the existence of the hello function cannot be checked just by parsing (compiling) the source code. The interpreter must evaluate (run) the code to discover it.

• In a static, compiled language, a missing function is guaranteed to be caught at compile-time.
• In a dynamic, interpreted language, however, it becomes a possible runtime error. If the eval-defined function is conditionally called, the error may not be discovered until that condition is met in production.

The Nushell REPL

As with most any shell, Nushell has a "Read→Eval→Print Loop" (REPL) that is started when you run nu without any file. This is often thought of, but isn't quite the same, as the "commandline".

> some code...

> print "Hello world!"
# => Hello world!

> ls
# => prints files and directories...

When you press Enter after typing a commandline, Nushell:

1. (Read): Reads the commandline input
2. (Evaluate): Parses the commandline input
3. (Evaluate): Evaluates the commandline input
4. (Evaluate): Merges the environment (such as the current working directory) to the internal Nushell state
5. (Print): Displays the results (if non-null)
6. (Loop): Waits for another input

In other words, each REPL invocation is its own separate parse-evaluation sequence. By merging the environment back to the Nushell's state, we maintain continuity between the REPL invocations.

Compare a simplified version of the cd example from "Thinking in Nu":

cd spam
source-env foo.nu

There we saw that this cannot work (as a script or other single expression) because the directory will be changed after the parse-time source-env keyword attempts to read the file.

Running these commands as separate REPL entries, however, works:

> cd spam
> source-env foo.nu
# Yay, works!

To see why, let's break down what happens in the example:

1. Read the cd spam commandline.
2. Parse the cd spam commandline.
3. Evaluate the cd spam commandline.
4. Merge environment (including the current directory) into the Nushell state.
5. Read and Parse source-env foo.nu.
6. Evaluate source-env foo.nu.
7. Merge environment (including any changes from foo.nu) into the Nushell state.

When source-env tries to open foo.nu during the parsing in Step 5, it can do so because the directory change from Step 3 was merged into the Nushell state during Step 4. As a result, it's visible in the following Parse/Eval cycles.

Multiline REPL Commandlines

Keep in mind that this only works for separate commandlines.

In Nushell, it's possible to group multiple commands into one commandline using:

• A semicolon:

  cd spam; source-env foo.nu

• A newline:

  > cd span
    source-env foo.nu

  Notice there is no "prompt" before the second line. This type of multiline commandline is usually created with a keybinding to insert a Newline when Alt+Enter or Shift+ Enter is pressed.

These two examples behave exactly the same in the Nushell REPL. The entire commandline (both statements) are processed a single Read→Eval→Print Loop. As such, they will fail the same way that the earlier script-example did.

Parse-time Constant Evaluation

While it is impossible to add parsing into the evaluation stage and yet still maintain our static-language benefits, we can safely add a little bit of evaluation into parsing.

By their nature, constants and constant values are known at Parse-time. This, of course, is in sharp contrast to variable declarations and values.

As a result, we can utilize constants as safe, known arguments to parse-time keywords like source, use, and related commands.

Consider this example from "Thinking in Nu":

let my_path = "~/nushell-files"
source $"($my_path)/common.nu"

As noted there, we can, however, do the following instead:

const my_path = "~/nushell-files"
source $"($my_path)/common.nu"

Let's analyze the Parse/Eval process for this version:

1. The entire program is Parsed into IR.

   1. Line 1: The const definition is parsed. Because it is a constant assignment (and const is also a parser-keyword), that assignment can also be Evaluated at this stage. Its name and value are stored by the Parser.
   2. Line 2: The source command is parsed. Because source is also a parser-keyword, it is Evaluated at this stage. In this example, however, it can be successfully parsed since its argument is known and can be retrieved at this point.
   3. The source-code of ~/nushell-files/common.nu is parsed. If it is invalid, then an error will be generated, otherwise the IR results will be included in evaluation in the next stage.

2. The entire IR is Evaluated:

   1. Line 1: The const definition is Evaluated. The variable is added to the runtime stack.
   2. Line 2: The IR result from parsing ~/nushell-files/common.nu is Evaluated.

Also keep in mind that the evaluation allowed during parsing is very restricted. It is limited to only a small subset of what is allowed during a regular evaluation.

For example, the following is not allowed:

const foo_contents = (open foo.nu)

Put differently, only a small subset of commands and expressions can generate a constant value. For a command to be allowed:

• It must be designed to output a constant value
• All of its inputs must also be constant values, literals, or composite types (e.g., records, lists, tables) of literals.

In general, the commands and resulting expressions will be fairly simple and without side effects. Otherwise, the parser could all-too-easily enter an unrecoverable state. Imagine, for instance, attempting to assign an infinite stream to a constant. The Parse stage would never complete!

help commands | where is_const

For example, the path join command can output a constant value. Nushell also defines several useful paths in the $nu constant record. These can be combined to create useful parse-time constant evaluations like:

const my_startup_modules =  $nu.default-config-dir | path join "my-mods"
use $"($my_startup_modules)/my-utils.nu"

Conclusion

Nushell operates in a scripting language space typically dominated by "dynamic", "interpreted" languages, such as Python, Bash, Zsh, Fish, and many others. Nushell is also "interpreted" since code is run immediately (without a separate, manual compilation).

However, is not "dynamic" in that it does not have an eval construct. In this respect, it shares more in common with "static", compiled languages like Rust or Zig.

This lack of eval is often surprising to many new users and is why it can be helpful to think of Nushell as a compiled, and static, language.