Nushell 0.98.0

Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your command line. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful command line pipelines.

Today, we're releasing version 0.98.0 of Nu. This release changes non-zero exit codes from external commands to be handled as errors, makes the IR evaluator the default, removes support for the system clipboard, and includes many command changes and bugfixes.

Where to get it

Nu 0.98.0 is available as pre-built binaries or from crates.io. If you have Rust installed you can install it using cargo install nu.

As part of this release, we also publish a set of optional plugins you can install and use with Nu. To install, use cargo install nu_plugin_<plugin name>.

Table of content

Highlights and themes of this release
Changes
Notes for plugin developers
Hall of fame
Full changelog

Highlights and themes of this release [toc]

Non-zero exit codes are now errors [toc]

Breaking change

See a full overview of the breaking changes

New error behavior

This release adds a new error when an external exits with a non-zero exit code. In the default config, this error is suppressed in the REPL. However, if you are upgrading with an existing config, you must add this section to your $env.config to suppress this error:

display_errors: {
    exit_code: false
    # Core dump errors are always printed, and SIGPIPE never triggers an error.
    # The setting below controls message printing for termination by all other signals.
    termination_signal: true
}

One long-requested feature for nushell is to make non-zero exit codes throw an error. Before this release, non-zero exit codes would quietly stop execution. In some cases, execution would not even stop, but rather only skip over the remaining commands in the current code block. With this release, non-zero exit codes (and termination due to unix signals) are now errors just like any other error, so nushell now runs almost as if bash's set -e option was enabled (#13515).

Errors due to non-zero exit codes can be caught in a try/catch block, and the error record passed to the catch closure will contain an exit_code column in this case.

try {
  nu -c 'exit 42'
} catch {|e|
  print $e.exit_code? # prints 42
}

As such, $env.LAST_EXIT_CODE is no longer necessary in most cases. Rather, try/catch should suffice for both internal and external errors, unifying and simplifying error handling.

Note that the complete command still works the same as it does today. If the previous command fails, its exit code and output will still be captured and returned by complete.

Some other things to note about the current implementation:

Only the exit code of the last command of a pipeline is checked and can cause an error. The exit codes of all the previous commands in a pipeline are ignored. In the future, we may check and track the exit codes of all commands in a pipeline to provide an equivalent to bash's set -o pipefail, but this would require a major refactor. So for now, this functionality is only a potential future addition.
If a command was terminated by a unix signal (e.g., SIGTERM, SIGINT), then the exit_code value in the catch error record will contain the negation of the signal number. For example, if SIGINT is signal number 2, then exit_code would be reported as -2. This is similar to how Python reports exit codes.
The one exception is SIGPIPE. If a command is terminated due to SIGPIPE, then it is not treated as an error.

One downside of having non-zero exit codes as errors is that uncaught errors will print a nushell error message to the terminal which may make it annoying to find any error messages printed by the external command that failed. Similarly, some commands use exit codes to indicate other things besides failure (e.g., diff). Having all these extra error message in the REPL can be annoying, so two new config options have been added as well:

$env.config.display_errors.exit_code
$env.config.display_errors.termination_signal

With no config (e.g., when running a script), these config options are both true and the corresponding error cases will be printed as a nushell error message. However, the default config (i.e., config nu --default) sets display_errors.exit_code to false. So, if you use the default config, or set this option in your config, then an error message won't be printed for non-zero exit codes in the REPL.

One last thing to note is that core dumps are treated as a separate error case and an error message will always be printed if the error is uncaught (i.e., there is no config option for this).

IR is now the default evaluator [toc]

Nushell 0.96.0 introduced internal representation (IR), with a new compiler and evaluator to take some of the work away from runtime, enable future optimizations, and create a well-known target for our ongoing parser replacement work.

The IR evaluator has now been battle-tested by many Nushell contributors and power users and passes our entire test suite, so we've decided to make it the default now and hopefully remove the old evaluator in the next release if all goes well.

This means that the NU_USE_IR environment variable is no longer respected, and instead we've added NU_DISABLE_IR which will go back to the old evaluator if present. If you find that you need to disable IR to make code that was previously working function, please file a bug report so that we can fix it before the next release.

This may change the debug profile switches you need to use for useful output - expressions are not a valid mode for IR, you need to debug instructions instead.

Removing support for the system clipboard [toc]

Breaking change

See a full overview of the breaking changes

With #13694, support for the system clipboard in keybinds has been removed, as this is not something that we think we can provide good, cross-platform support for in a secure manner. Namely,

Tests for the clipboard feature are lacking due to the difficulty in setting up tests on each platform.
The clipboard feature is known to be problematic on certain systems, potentially affecting startup times. Due to the lack of tests, other unfound issues may be lurking.
The system clipboard adds a potential attack surface.
The clipboard feature adds 1MB to the binary size.

So, going forward, users that want the system clipboard functionality will have to compile a custom build of nushell by enabling the system-clipboard cargo feature. As an alternative, we recommend users delegate system clipboard handling to their terminal.

Record parsing changes [toc]

Breaking change

See a full overview of the breaking changes

Previously, it was impossible to use datetime literals with colons in record literals without adding parentheses. But thanks to @cyradotpink in #13413, this is now possible.

# Previously, parentheses were necessary:
{ foo: (2024-08-13T22:11:09) }

# But this now works!
{ foo: 2024-08-13T22:11:09 }

To make this possible, a few changes were made to record parsing. This should not affect most code, but some previously valid code no longer parses successfully. For example:

{ :: 1 }
# This used to parse as { ':': 1 }, but it is now an error.

Changes [toc]

Additions [toc]

`metadata access` [toc]

Thanks to @Bahex in #13785, a new metadata access command was added. It takes a closure argument, and the first argument to the closure will be the metadata record. Additionally, the pipeline input to metadata access will be forwarded to the closure as is. So, unlike the metadata command, this new command allows you to get the metadata of a pipeline without consuming the pipeline.

`split cell-path` [toc]

In #13705, @Bahex also added the split cell-path command. It is essentially the opposite of into cell-path and returns a list of the path members of the input cell path.

`split column --numbered` [toc]

Thanks to @nome in #13831, a --numbered flag was added to split column which limits the number of times to split the input string (just like split row --numbered).

`encode base32`, `decode base32` [toc]

In #13428, @KAAtheWiseGit added this common variant of Base32 which uses letters A-Z followed by t he digits 2-7, specifically chosen to avoid letters and digits that look similar (RFC 4648 section 6).

`encode base32hex`, `decode base32hex` [toc]

In #13428, @KAAtheWiseGit added this variant of Base32 which only uses the 22 letters past 0-9 sequentially (RFC 4648 section 7).

`encode new-base64`, `decode new-base64` [toc]

In #13428, @KAAtheWiseGit added the encode new-base64 and decode new-base64 commands, which will eventually be renamed back to encode base64 and decode base64. These commands are more standards-conformant and rather than taking a custom alphabet option, they support --url for the URL-safe base64 variant, and --nopad to handle Base64 without padding.

Breaking changes [toc]

`into record` [toc]

In #13637, several changes/improvements were made to list inputs for into record:

A list of pairs of values will be converted into a record. For example:
```
[[a 1] [b 2]] | into record
# { a: 1, b: 2 }
```
A list of records are merged into a single record (records at the end of the list have precedence):
```
[{ a: 1 } { b: 2 } { a: 3 }] | into record
# { a: 3, b: 2 }
```

As a consequence, the previous behavior for list inputs has been removed (the index of each item would be used as the key). Instead, this old behavior can now be accomplished through either of the following:

0.. | zip $list | into record

$list | enumerate | transpose -r -d | into record

Additionally, into record no longer supports range values as input.

After #13650, into record also now outputs millisecond, microsecond, and nanosecond columns when provided with a datetime value.

`clear` [toc]

In #13821, @T3sT3ro changed the clear command to clear the entire scrollback buffer. As such, the --all flag was removed, since it is now the default behavior. Instead, a --keep-scrollback flag was added which, when used, makes clear clear only the terminal (the previous default behavior).

`path exists` [toc]

After #13763, if an I/O error occurs when checking if a path exists, false is now returned instead of throwing an error (this includes errors due to insufficient permissions).

`scope commands` and `help commands` [toc]

With #13598, two output columns have been renamed for scope commands and help commands:

usage has been renamed to description
extra_usage is now extra_description

`tee` [toc]

tee now works as expected for non-collection input (#13652), and will no longer implicitly handle the input as if it were a singleton list. This does create some hairy situations for error handling, namely:

For string or binary input, the input will be converted to a stream (even if it wasn't one), and the closure provided to tee must finish in order for the output of tee to end. If an error is produced by the closure, it will be forwarded to the output stream.
For any other non-collection input, any error in the tee closure will be logged to standard error, but there isn't really anything else we can do because we can't wait for that closure to finish.

`polars concatenate` renamed to `polars str-join` [toc]

In #13781, polars concatenate was renamed to polars str-join to match the name of the Nushell str join command. It also now has support for expressions.

Deprecations [toc]

`encode base64`, `decode base64` [toc]

In #13428, @KAAtheWiseGit added encode new-base64 / decode new-base64 to replace these.

Removals [toc]

`str deunicode` [toc]

In #13693, the str deunicode command (added in v0.96.0) has been removed due to concerns with stability and support post v1.0.

`$env.config.use_grid_icons` [toc]

After #13788, the use_grid_icons config option has been removed. Instead, the grid command now has an --icons flag that has the same effect.

Bug fixes and other changes [toc]

`find` [toc]

After #13848, the find command no longer strips tabs from its input.

Also, regex patterns are now properly escaped after #13792.

`encode hex` [toc]

In #13428, @KAAtheWiseGit added the --lower flag to encode hex, to produce lowercase hex.

Various Polars commands now support expressions [toc]

In #13724, #13726, and #13799, these Polars plugin commands were updated to support expressions:

polars uppercase
polars lowercase
polars replace
polars replace-all
polars cumulative

Notes for plugin developers [toc]

Breaking plugin changes [toc]

usage in signatures has been changed to description (#13598). This affects both the Rust API and the plugin protocol.

Rust API improvements [toc]

The IntoValue and FromValue traits now work well with Vec<u8> byte arrays and bytes::Bytes (#13641), converting to binary and from string/binary.
IntoValue is now supported for several Nushell types, as well as HashMap, DateTime, and &str (#13744)
The derive macros for IntoValue and FromValue now also take #[nu_value(type_name = "...")] to set a type name for the expected type. (#13647)

Other changes [toc]

The nu-plugin library now uses MessagePack instead of bincode internally for custom value serialization. This is not a protocol change as the representation of custom values is intentionally not defined, and they're treated as a black box by the engine. (#13745)

All breaking changes [toc]

#13821 Improve #12008 UX, clear scrollback by default on clear
#13763 fix path exists on a non-directory file
#13515 Error on non-zero exit statuses
#13788 remove config use_grid_icons, move to parameter of grid command
#13781 Renamed polars concatenate and added expression support.
#13652 Make tee work more nicely with non-collections
#13745 Remove bincode and use MessagePack instead for plugin custom values
#13694 Remove system-clipboard from the default build
#13693 Remove str deunicode
#13428 encode/decode for multiple alphabets
#13598 Change the usage misnomer to "description"
#13637 Change behavior of into record on lists to be more useful

Hall of fame [toc]

Thanks to all the contributors below for helping us solve issues and improve documentation 🙏

author	title	PR
@fdncred	allow tab to be retained with `find`	#13848
@zhiburt	Fix padding issue with header_on_border	#13808
@IanManske	Fix IR for `try`	#13811
@ayax79	String values should pass their content-type correctly on http requests with a body (e.g. http post)	#13731
@devyn	Remove bincode and use MessagePack instead for plugin custom values	#13745
@ysthakur	Use right options in custom completions	#13698
@KAAtheWiseGit	Fix encode/decode todo's	#13683
@sholderbach	Fix `bits ror`/`bits rol` implementation	#13673
@sholderbach	Fix bugs and UB in bit shifting ops	#13663

Nushell 0.98.0

Where to get it

Table of content

Highlights and themes of this release [toc]

Non-zero exit codes are now errors [toc]

IR is now the default evaluator [toc]

Removing support for the system clipboard [toc]

Record parsing changes [toc]

Changes [toc]

Additions [toc]

metadata access [toc]

split cell-path [toc]

split column --numbered [toc]

encode base32, decode base32 [toc]

encode base32hex, decode base32hex [toc]

encode new-base64, decode new-base64 [toc]

Breaking changes [toc]

into record [toc]

clear [toc]

path exists [toc]

scope commands and help commands [toc]

tee [toc]

polars concatenate renamed to polars str-join [toc]

Deprecations [toc]

encode base64, decode base64 [toc]

Removals [toc]

str deunicode [toc]

$env.config.use_grid_icons [toc]

Bug fixes and other changes [toc]

find [toc]

encode hex [toc]

Various Polars commands now support expressions [toc]

Improved script help output [toc]

Bit operations and commands [toc]

detect columns [toc]

Notes for plugin developers [toc]

Breaking plugin changes [toc]

Rust API improvements [toc]

Other changes [toc]

All breaking changes [toc]

Hall of fame [toc]

Full changelog [toc]

`metadata access` [toc]

`split cell-path` [toc]

`split column --numbered` [toc]

`encode base32`, `decode base32` [toc]

`encode base32hex`, `decode base32hex` [toc]

`encode new-base64`, `decode new-base64` [toc]

`into record` [toc]

`clear` [toc]

`path exists` [toc]

`scope commands` and `help commands` [toc]

`tee` [toc]

`polars concatenate` renamed to `polars str-join` [toc]

`encode base64`, `decode base64` [toc]

`str deunicode` [toc]

`$env.config.use_grid_icons` [toc]

`find` [toc]

`encode hex` [toc]

`detect columns` [toc]