Nushell 0.94.0

Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your command line. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful command line pipelines.

Today, we're releasing version 0.94.0 of Nu. This release adds case-preserving environment, changes to path handling, raw string literals, and improvements to streaming!

Where to get it

Nu 0.94.0 is available as pre-built binariesopen in new window or from crates.ioopen in new window. If you have Rust installed you can install it using cargo install nu.

As part of this release, we also publish a set of optional plugins you can install and use with Nu. To install, use cargo install nu_plugin_<plugin name>.

WARNING

The dataframes cargo feature has been removed in this release, after being deprecated in 0.93.0. You may need to modify your build scripts if you have been building Nushell from source.

Table of content

Themes of this release / New features [toc]

Fixing path and PWD handling [toc]

One of our contributors, @YizhePKUopen in new window, has started an effort to refactor how Nushell internally handles the current working directory and paths in general. Namely, Nushell aggressively canonicalizes paths instead of using logical paths which can lead to unintuitive or annoying behavior (#2175open in new window). Thanks to their work, $env.PWD and the pwd command now support logical paths. With time, we intend to make similar to changes to other commands to make them more intuitive and consistent with the rest of Nushell. Another goal with these changes is to eliminate bugs in commands, as some commands incorrectly use the Nushell process's current working directory instead of $env.PWD. So far, the grid, path type, and touch --reference commands have been fixed, and we are aiming to bring more fixes in the next release. We kindly ask for your patience as we rework this part of Nushell, and we would appreciate if you would report any issues you encounter!

Case-preserving environment [toc]

Breaking change

See a full overview of the breaking changes

In an effort to allow more ergonomic cross-platform scripting, we have changed the environment in Nushell to be case-preserving (#12701open in new window). That is, environment variables keep their initial casing but can be accessed in a case-insensitive manner within Nushell. This means that accessing $env.PATH or $env.Path will now work on both unix-based systems and Windows systems. To get environment variables in a case-sensitive manner, you can use get -s.

$env | get -s PATH

Please note that there are still a few things that have to be ironed out with this new behavior. For example, it is currently unclear what load-env { FOO: BAR, foo: bar } should do. So, there may be more changes in the near future.

Streaming all the things [toc]

Another major effort with this release was to make streaming more consistent and widespread (#12774open in new window, #12897open in new window, #12918open in new window, and more). For example, Nushell now passes operating system pipe descriptors directly to external commands, meaning there should be an improvement to throughput for some pipelines. Additionally, several commands were reworked so that they are now able to stream their output in more cases instead of having to collect it all in memory beforehand. The commands in question include from csv, to csv, from tsv, to tsv, from json --objects, take, skip, first, last, bytes ends-with, bytes collect, str join, into string, and into binary. Of course, the plan is to add more commands to this list in the future, so this is only the beginning!

New language feature: raw strings [toc]

This release adds a new type of string literal to the language: raw strings (#9956open in new window). These are just like single quoted strings, except that raw strings may also contain single quotes. This is possible, since raw strings are enclosed by a starting r#' and an ending '#. This syntax should look familiar to users of Rust.

r#'some text'# == 'some text' # true

r#'contains 'quoted' text'# == "contains 'quoted' text"

Additional # symbols can be added to the start and end of the raw string to enclose one less than the same number of # symbols next to a ' symbol in the string.

r###'this text has multiple '## symbols'###

Removal of deprecated features [toc]

Breaking change

See a full overview of the breaking changes

Last release, we deprecated the dataframe commands and feature in favor of the new polars plugin. See the previous release notesopen in new window for more information and migration steps. In this release, the dataframe commands have been removed as planned alongside the dataframes cargo feature. Similarly, this release removes lazy records from the language following their deprecation. This includes the removal of the lazy make command and the --collect-lazyrecords flag on describe. One consequence of this change is that the sys command will now return a regular record and will take a minimum of 400ms to complete (to sample CPU usage). As such, the sys command has been deprecated, and new subcommands have been added in its place (sys mem, sys cpu, sys host, sys disks, sys net, sys temp, and sys users).

Shell integration config [toc]

Breaking change

See a full overview of the breaking changes

With this release, we have changed the shell_integration setting in the config to be a record of multiple settings (#12629open in new window):

shell_integration: {
  # osc2 abbreviates the path if in the home_dir, sets the tab/window title, shows the running command in the tab/window title
  osc2: true
  # osc7 is a way to communicate the path to the terminal, this is helpful for spawning new tabs in the same directory
  osc7: true
  # osc8 is also implemented as the deprecated setting ls.show_clickable_links, it shows clickable links in ls output if your terminal supports it
  osc8: true
  # osc9_9 is from ConEmu and is starting to get wider support. It's similar to osc7 in that it communicates the path to the terminal
  osc9_9: false
  # osc133 is several escapes invented by Final Term which include the supported ones below.
  # 133;A - Mark prompt start
  # 133;B - Mark prompt end
  # 133;C - Mark pre-execution
  # 133;D;exit - Mark execution finished with exit code
  # This is used to enable terminals to know where the prompt is, the command is, where the command finishes, and where the output of the command is
  osc133: true
  # osc633 is closely related to osc133 but only exists in visual studio code (vscode) and supports their shell integration features
  # 633;A - Mark prompt start
  # 633;B - Mark prompt end
  # 633;C - Mark pre-execution
  # 633;D;exit - Mark execution finished with exit code
  # 633;E - NOT IMPLEMENTED - Explicitly set the command line with an optional nonce
  # 633;P;Cwd=<path> - Mark the current working directory and communicate it to the terminal
  # and also helps with the run recent menu in vscode
  osc633: true
  # reset_application_mode is escape \x1b[?1l and was added to help ssh work better
  reset_application_mode: true
}

This change provides greater granularity, allowing you to disable any integrations that may cause issues in certain terminals.

Hall of fame [toc]

Bug fixes [toc]

Thanks to all the contributors below for helping us solve issues and bugs 🙏

authortitleurl
@IanManskeopen in new windowFix panic when redirecting nothing#12970open in new window
@YizhePKUopen in new windowFix touch --reference using PWD from the environment#12976open in new window
@YizhePKUopen in new windowFix path type using PWD from the environment#12975open in new window
@IanManskeopen in new windowClear environment for child Commands#12901open in new window
@IanManskeopen in new windowPreserve metadata in more places#12848open in new window
@WindSoilderopen in new windowallow define it as a variable inside closure#12888open in new window
@IanManskeopen in new windowReplace ExternalStream with new ByteStream type#12774open in new window
@WindSoilderopen in new windowallow passing float value to custom command#12879open in new window
@IanManskeopen in new windowFix sys panic#12846open in new window
@IanManskeopen in new windowFix char panic#12867open in new window
@ExaltedBagelopen in new windowFix panic when exploring empty dictionary#12860open in new window
@ExaltedBagelopen in new windowFix improperly escaped strings in stor insert#12820open in new window
@IanManskeopen in new windowFix custom converters with save#12833open in new window
@IanManskeopen in new windowFix pipe redirection into complete#12818open in new window
@IanManskeopen in new windowFix syntax highlighting for not#12815open in new window
@IanManskeopen in new windowFix list spread syntax highlighting#12793open in new window
@WindSoilderopen in new windowallow raw string to be used inside subexpression, list, and closure#12776open in new window
@WindSoilderopen in new windowAllow ls works inside dir with [] brackets#12625open in new window
@ExaltedBagelopen in new windowEnable columns with spaces for into_sqlite by adding quotes to column names#12759open in new window
@WindSoilderopen in new windowallow raw string to be used inside subexpression, list, and closure#12776open in new window
@IanManskeopen in new windowBump base64 to 0.22.1#12757open in new window
@devynopen in new windowFix trailing slash in PWD set by cd#12760open in new window
@lavafrothopen in new windowfix: prevent relative directory traversal from crashing#12438open in new window
@merelymyselfopen in new windowPrevent each from swallowing errors when eval_block returns a ListStream#12412open in new window

Enhancing the documentation [toc]

Thanks to all the contributors below for helping us making the documentation of Nushell commands better 🙏

authortitleurl
@NotTheDr01dsopen in new windowFixed small error in the help-examples for the get command#12877open in new window
@NotTheDr01dsopen in new windowFixed a nitpick usage-help error - closure v. block#12876open in new window
@fdncredopen in new windowmake it clearer what is being loaded with --log-level info#12875open in new window
@WindSoilderopen in new windowmake better messages for incomplete string#12868open in new window
@NotTheDr01dsopen in new windowSearch terms for compact command#12864open in new window
@NotTheDr01dsopen in new windowAdd example and search term for 'repeat' to the fill command#12844open in new window
@devynopen in new windowMake the message when running a plugin exe directly clearer#12806open in new window
@amtoineopen in new windowimprove NUON documentation#12717open in new window
@sholderbachopen in new windowUpdate PLATFORM_SUPPORT regarding feature flags#12741open in new window

Our set of commands is evolving [toc]

New commands [toc]

sys subcommands [toc]

This release adds new sys subcommands corresponding to each column of the record returned by sys (#12747open in new window):

  • sys host
  • sys cpu
  • sys mem
  • sys net
  • sys disks
  • sys temp
  • sys users (new!)

The output of these commands should be the same as before as the corresponding sys column except for a few differences:

  • sys host does not have a sessions column. This table has been moved to the new sys users command (#12787open in new window). (The old sys command still has the sessions column.)
  • The boot_time column in sys host and sys | get host is now a date value instead of a formatted string (#12846open in new window).

ps support on BSD systems [toc]

In #12892open in new window, @devynopen in new window has added support for the ps command on FreeBSD, NetBSD, and OpenBSD!

debug profile --lines [toc]

The debug profile command now has a --lines flag with #12930open in new window. Providing this will flag will add a file and line column to the output of debug profile.

Changes to existing commands [toc]

Making range semantics consistent [toc]

Breaking change

See a full overview of the breaking changes

Ranges are inclusive in the upper bound by default in Nushell (e.g., 1..2 or 1..=2 gives [1, 2]). However, some commands treated the upperbound as exclusive. With #12894open in new window this has been fixed for the following commands:

  • str substring
  • str index-of
  • detect columns

To specify an exclusive upperbound, use the ..< form for ranges (e.g., 1..<2).

parse [toc]

Breaking change

See a full overview of the breaking changes

To support streaming output, parse now operates/matches on a single line at a time only if provided with the streaming output of an external command, file, or raw byte stream. For example, these snippets will try to match each line separately:

^cat file.txt | parse -r "some regex"
^cat file.txt | take 1024 | parse -r "some regex"

open file.txt | parse -r "some regex"
open --raw file.txt | parse -r "some regex"
open --raw file.txt | skip 1024 | parse -r "some regex"

The old behavior was to collect all of the output the external command or byte stream and then run the regex across the whole string. To mimic this behavior, you can use the collect command or store the output in a variable:

^cat file.txt | collect | parse -r "some regex"

let text = open file.txt
$text | parse -r "some regex"

Note that this change does not affect normal value streams like:

[(open foo.txt) (open bar.txt)] | parse -r "some regex"

In this case, the regex is run separately over each string value in the stream. With the proper regex flags, the regex is able to match across multiple lines within the same string value.

Note that parse may see more breaking changes in the next release to simplify these behaviors.

scope commands [toc]

Breaking change

See a full overview of the breaking changes

After #12832open in new window, several columns from the output of scope commands have been removed:

  • is_builtin
  • is_plugin
  • is_custom
  • is_keyword
  • is_extern

All of these columns are mutually exclusive (or one column always implies another), so these columns have instead been replaced with a single type column. It will contain one of the following string values:

  • built-in
  • plugin
  • custom
  • keyword
  • external
  • alias

which [toc]

Breaking change

See a full overview of the breaking changes

Related to the changes for scope commands, the type column reported by which can now be:

  • plugin or keyword instead of built-in
  • external instead of custom for declared extern commands

describe [toc]

Breaking change

See a full overview of the breaking changes

With the --detailed flag, the stdout, stderr, and exit_code columns have been removed for external streams / byte streams. Instead, the origin column now reports unknown, file, or external for byte streams, and the type column will be byte stream, string (stream), or binary (stream).

Streaming more commands [toc]

This release changes a bunch of commands to stream where possible (#12897open in new window):

  • first and last now support byte stream input and will return the first or last specified number of bytes (without collecting the whole stream into memory at once).
  • take and skip also now support byte streams input and will return new byte streams.
  • str join will now stream string output if it is given a stream of string values.
  • bytes collect can also stream its output now.
  • bytes ends-with now supports byte streams and does not need to collect the entire stream output in memory at once (#12887open in new window).
  • from json --objects now streams its output if its input is a stream (#12949open in new window).
  • into binary and into string will now apply a type hint to byte streams and will simply return the input stream without consuming or draining it.

cd and pwd [toc]

Thanks to @YizhePKUopen in new window in #12603open in new window, the cd command now sets $env.PWD to a logical path by default. Instead, there is now a --physical flag for cd which will canonicalize the path before setting it as the PWD. Similarly, pwd now has a --physical flag to resolve symlinks before reporting the current working directory.

collect [toc]

After #12788open in new window, the closure parameter for collect is now optional. If no closure is provided, the collected pipeline input is simply returned as a value.

complete [toc]

Previously, if an external command was terminated by a signal, the exit code returned from complete would be -1. Also, if the external command core dumped, then an error value would be present in the exit_code column. Now with #12774open in new window, if an external command was terminated by a signal (this includes core dumps), then the exit_code column will contain the negation of the signal number (to differentiate it from a regular exit code).

Also, the o+e>| combined pipe redirection now works properly with complete. That is, the stdout and stderr of the preceding external command are merged prior to be passed to complete, and the combined output will be present in the stdout column of the record returned by complete (#12818open in new window).

each [toc]

Thanks to @merelymyselfopen in new window, the each command now bubbles up errors in certain cases where it would previously ignore them (#12412open in new window).

ls [toc]

In a similar change, the ls command will now return errors instead of empty entries if it fails to list a path (#12033open in new window). Additionally, ls has received a bug fix with #12625open in new window so that it works in a directory with [ or ] in its path.

bytes build [toc]

With #12685open in new window, the bytes build command now accepts integers as rest arguments (each integer number is treated as a byte).

from nuon and to nuon [toc]

The nuon commands can now serialize and deserialize cell path values after #12718open in new window.

save [toc]

With #12833open in new window, custom converters now work again for the save command. E.g., save output.custom now uses the to custom command if it exists (note that this is currently not done for singular string value inputs).

hide-env [toc]

A long-standing bug with hide-env is that it would not hide environment variables from external commands / child processes in certain cases. With #12901open in new window, this issue has finally been fixed.

PWD fixes [toc]

Some commands are incorrectly using the current working directory of the shell process, instead of the internal $env.PWD tracked in the engine state. This release fixes a few of these commands:

nu-highlight [toc]

nu-highlight has received two bug fixes. Namely, highlighting not has been fixed (#12815open in new window) and the extra output present when highlighting list spreads has been fixed (#12793open in new window).

into sqlite [toc]

Columns with spaces are now supported by into sqlite, since it will wrap the columns with backticks (#12759open in new window).

stor insert [toc]

Thanks to @ExaltedBagelopen in new window in #12820open in new window, stor insert now escapes single quotes.

decode base64 and encode base64 [toc]

The alphabet used for --character-set binhex has been corrected in #12757open in new window.

char [toc]

In #12867open in new window, an issue with char was fixed where it can panic if a list is spread after the --integer or --unicode flags.

Deprecated commands [toc]

sys [toc]

With the removal of lazy records, the sys command will now return a regular record. This means it has to compute all of its columns before returning. Since the sys command samples CPU usage over a 400ms period, then this would be unideal if the only information you needed from the command was, e.g., the host information. Because of this, the command has been deprecated in favor of the new sys subcommands as listed in new commands.

str contains --not [toc]

This release deprecates the --not flag for str contains, since none of the other string commands have a similar flag (#12837open in new window). Instead of the --not flag, you can use the not operator in one of the following ways:

# using parentheses
not ("foobar" | str contains foo)

# using a pipe with the `$in` variable
"foobar" | str contains foo | not $in

Removed commands [toc]

lazy make [toc]

The lazy make command has been removed in #12682open in new window. See removal of deprecated features for more information.

describe --collect-lazyrecords [toc]

The --collect-lazyrecords flag has been removed from describe, since lazy records have been removed from the language (#12682open in new window).

List of environment variables support in with-env [toc]

This release removes support for the list of environment variable form of with-env that was deprecatedopen in new window in the previous release. Going forward, only the record form is supported.

For plugin developers [toc]

Breaking API changes [toc]

  • The ExternalStream type has been replaced entirely with ByteStreamopen in new window, which represents a single (possibly typed) stream of bytes. Through the plugin layer, only the Read variant is used. It is no longer possible to get stderr or exit_code from an external command by piping it into a plugin command directly.
  • The ListStreamopen in new window type now requires a Span.
  • Configopen in new window's shell_integration field has been modified and split into several fields. It was previously a single bool.
  • LazyRecord has been removed, but this was not allowed in plugins anyway.

Breaking protocol changes [toc]

Breaking changes [toc]

Full changelog [toc]