Nushell 0.91.0

Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your command line. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful command line pipelines.

Today, we're releasing version 0.91.0 of Nu. This release adds changes to globing, an overhaul to the plugin protocol, support for piping command stderr, and new commands!

Where to get it

Nu 0.91.0 is available as pre-built binaries or from crates.io. If you have Rust installed you can install it using cargo install nu.

Note

The optional dataframe functionality is available by cargo install nu --features=dataframe.

As part of this release, we also publish a set of optional plugins you can install and use with Nu. To install, use cargo install nu_plugin_<plugin name>.

Table of content

• Themes of this release / New features
  • Handling globs for variables
  • Plugin protocol overhaul
  • Stderr pipeling support
  • REPL stability and panic recovery
  • Hall of fame
    • Bug fixes
  • Our set of commands is evolving
    • New commands
      • tee
      • interleave
      • is-not-empty
      • commandline
    • Changes to existing commands
      • zip supports closures
      • bits supports binary values
      • into int --signed
      • Migrating more commands to use uutils
      • List spreading for filesystem commands
      • Duplicate record keys now error
      • Removing list of cell path support
    • Deprecated commands
    • Removed commands
• Breaking changes
• Full changelog

toc]

toc]

Breaking change

See a full overview of the breaking changes

From this release, if you pass a string variable to commands that support glob patterns, then Nushell won't auto-expand the glob pattern (#11886, #11946).

For example, given let f = "a[ab]c.txt", then ls $f will only list a file named a[ab]c.txt. But if you want to auto-expand the glob pattern on variables, there are 3 ways:

1. Use the glob command with spreading:

   let f = "a*c.txt"
   rm ...(glob $f)
   

2. Use the into glob command:

   let f = "a*c.txt"
   rm ($f | into glob)
   

3. Annotate the variable with glob type:

   let f: glob = "a*c.txt"
   rm $f
   

In that case, str escape-glob command is no longer useful, and it has been deprecated (#12018).

You can check the guide and book for more information.

toc]

Breaking change

See a full overview of the breaking changes

The plugin protocol has been redesigned to support plugins that operate on streams (#11911). A new StreamingPlugin trait is provided for plugins that want to take advantage of this functionality.

The API for existing plugins written in Rust has not changed, but they do need to be recompiled with the latest version of nu-plugin to use the new protocol.

As part of this effort, several other aspects of the protocol have been improved:

• Plugins will now automatically refuse to run with an incompatible version of Nushell. This should greatly improve error messages in that case.

• Plugin custom values are now accepted nested within other values and as arguments. The previous protocol treated them as a special case, and they could only be used as input or output from a plugin command, and could not be nested within other values.

• Plugins are now expected to keep running until Nushell tells them to stop. Currently, we still run a new plugin executable for each command, but this opens the door for that to change in the future.

• The bidirectional communication is abstract enough to allow for plugins to have much tighter integration with the engine. Expect further improvements to plugins in future releases!

For details about how the new protocol works, please refer to the updated plugins section of the contributor book, as well as the new plugin protocol reference.

As a great example of a streaming plugin in the wild, @cablehead has already created the nu_plugin_from_sse plugin for parsing server-sent events in realtime from an http get call. Kudos!

toc]

Nushell now supports piping an external command's stderr to another command (#11708).

Let's say you want to pass stderr output to the less command, you can do this:

cmd e>| less

In case you want to combine both stdout and stderr output to less, you can do this:

cmd o+e>| less

For more information, you can check the guide.

toc]

Thanks to the work of @ayax79 in #11860, #11935, and #11953, the Nushell REPL should no longer crash if a panic occurs and should no longer exit if some other error is encountered. Besides being very convenient, this should also make Nushell safer to use a login shell, since panics would previously cause Nushell to crash on an attempted login. Similarly, if a panic was triggered when loading a config file, then this used to prevent the REPL from starting. Now, the REPL falls back to loading the default config files in this case.

Note that panics are still important errors/bugs, so please open issues and bug reports if you encounter any!

toc]

toc]

Thanks to all the contributors below for helping us solve issues and bugs 🙏

author	title	PR
@PanGan21	fix: process empty headers in to md command	#12012
@devyn	Add Goodbye message to ensure plugins exit when they are no longer needed	#12014
@zhiburt	nu-table: Improve table -a	#11905
@ayax79	wrapping run_repl with catch_unwind and restarting the repl on panic	#11860
@WindSoilder	make stderr works for failed external command	#11914
@kit494way	separate commandline into subcommands	#11877
@kit494way	Fix panic in seq date	#11871
@kit494way	Fix commandline --cursor to return int	#11864
@dannou812	Fixed to/from toml date format	#11846
@IanManske	Prevent duplicate keys for lazy make	#11808
@IanManske	Prevent duplicate records keys when decoding from nuon	#11807
@kit494way	Allow comments in match blocks	#11717
@WindSoilder	Fix file completions which contains glob pattern	#11766
@TrMen	Enforce call stack depth limit for all calls	#11729

toc]

toc]

Thanks to the work of @devyn, this release adds two new commands related to streaming!

tee

Inspired by the Unix tee command, this command allows you to make a copy of a stream to a closure in the middle of your pipeline (#11928).

Examples:

# Get the sum of numbers from 1 to 100, but also save those numbers to a text file
seq 1 100 | tee { save numbers.txt } | math sum
# The exact opposite: keep the numbers, but save the sum to a file
seq 1 100 | tee { math sum | save sum.txt }
# Run an external command, and save a copy of its log output on stderr
do { cargo run } | tee --stderr { save err.txt }
# Filter the log output before saving it
do { cargo run } | tee --stderr { lines | find WARN | save warnings.txt }

The closure will run in a background thread in parallel, and it will only get a copy of values that actually make their way to the end of the pipeline. For example, if you cut the stream short:

seq 1 100 | tee { save numbers.txt } | first 5

then "numbers.txt" will only contain the first 5 numbers as well.

interleave

This command supports consuming multiple streams in parallel, and combining the streams into a single stream (#11955).

In contrast to zip, the values are added to the final stream as soon as they're ready, without any regard for fairness. There is no way to know which stream the values came from unless that information is embedded into the values by the closures being executed.

For example, the following zip-based pipeline will always produce output that looks like this:

> seq 1 50 | wrap a | zip { seq 1 50 | wrap b } | flatten
╭────┬────┬────╮
│  # │ a  │ b  │
├────┼────┼────┤
│  0 │  1 │ ❎ │
│  1 │ ❎ │  1 │
│  2 │  2 │ ❎ │
│  3 │ ❎ │  2 │
│  4 │  3 │ ❎ │
│  5 │ ❎ │  3 │
...

Each number from a is always paired with a number from b. However, if interleave is used instead, it is not predictable in which order a values will appear with respect to b:

> seq 1 50 | wrap a | interleave { seq 1 50 | wrap b }
╭────┬────┬────╮
│  # │ b  │ a  │
├────┼────┼────┤
│  0 │  1 │ ❎ │
│  1 │ ❎ │  1 │
│  2 │  2 │ ❎ │
│  3 │  3 │ ❎ │
│  4 │ ❎ │  2 │
│  5 │ ❎ │  3 │
│  6 │  4 │ ❎ │
│  7 │  5 │ ❎ │
│  8 │  6 │ ❎ │
│  9 │  7 │ ❎ │
│ 10 │ ❎ │  4 │
...

One advantage of this is that it is not necessary for both streams to produce the same amount of output, and whatever output is produced will be immediately available. This is particularly helpful for running external commands and interleaving lines from all of them:

interleave ...(
  (ls projects).name | each { |project|
    {
      cd $project
      make | lines | each { |line| {project: $project, out: $line} }
    }
  }
)

This example would run the make command in every subdirectory of the "projects" directory, and gather the output lines in parallel.

is-not-empty

As a quality of life improvement, we have added the is-not-empty command in #11991. It is the same as the is-empty command but negates the result. This should hopefully eliminate the need for users to hand-roll their own is-not-empty command.

commandline

In #11877, @kit494way improved the signatures for the commandline command. Instead of using flags to perform different operations, commandline now has subcommands:

• commandline edit: to append, insert, or replace the cursor line (returns nothing)
• commandline get-cursor: to get the cursor position (returns an integer)
• commandline set-cursor: to set the cursor position (returns nothing)

These subcommands make certain flags unnecessary, and so these have been marked as deprecated:

• --cursor
• --cursor-end
• --append
• --insert
• --replace

toc]

zip supports closures

When zip is passed a closure now, it will run that closure and zip its output as a stream (#11924). With this change, it is no longer necessary for the argument to zip to complete before the result is available:

seq 1 10 | zip { 1.. | each { $in * 2 } }

This would not have completed before: the infinite stream would have had to be on the input side of zip, because a subexpression would have been (attempted to be) fully consumed:

# never completes, and probably eats up all of your memory!
seq 1 10 | zip (1.. | each { $in * 2 })
# works fine
1.. | each { $in * 2 } | zip (seq 1 10)

Migrating more commands to use uutils

Continuing the integration with uutils, this release migrates the mv and mkdir commands. In #12022, we renamed the umv command to mv, removing the old implementation of mv. Similarly, we removed the old mkdir command in #12007 and renamed the umkdir command to take its place.

bits supports binary values

Despite their name, the bits commands used to only support integers and did not work with binary values. @astral-l has remedied this in #11854, and binary values now work with the bits family of commands. Where appropriate, some of these commands (bits and, bits or, and bits xor) now have an --endian flag to specify the endianness when operating on binary values of different lengths.

into int --signed

Thanks to @astral-l's work in #11902, the into int command now has a --signed flag for interpreting binary values as signed integers. (The default behavior when converting binary values is to zero-extend.) Additionally, binary values longer than 8 bytes will now error, as this could overflow a 64-bit integer which is what Nushell uses internally for integers.

List spreading for filesystem commands

With #11858, rm, open, and touch no longer have a required positional parameter and now only have a rest parameter. This should make spreading a list of files/paths to these commands more ergonomic.

Duplicate record keys now error

With #11807, from nuon now errors if duplicate record keys are found. Similarly, with #11808, lazy make now errors if duplicate keys are provided.

Removing list of cell path support

select and reject used to allow lists of cell paths as positional arguments like:

let cols = [name size]
ls | select $cols

In #11859, this was removed in favor of the spread operator:

ls | select ...$cols

Or, one can still provide the cell paths directly:

ls | select name size

toc]

• str escape-glob has been deprecated in #12018.
• Certain flags for commandline were deprecated in #11877.

toc]

• umv has been renamed to mv in #12022.
• umkdir has been renamed to mkdir in #12007.

toc]

• #11902 add --signed flag for binary into int conversions
• #11911 Bidirectional communication and streams for plugins
• #11972 Rename spans in the serialized form of Value
• #11886 open, rm, umv, cp, rm and du: Don't globs if inputs are variables or string interpolation
• #11877 separate commandline into subcommands
• #11864 Fix commandline --cursor to return int
• #11859 Remove list of cell path support for select and reject
• #11857 Disallow spreading lists automatically when calling externals

toc]

• fdncred created
  • fix --table-name parameter for into sqlite
  • fix: allow view source to view aliases again
  • remove old mv command in favor of umv (renamed to mv)
  • add is-not-empty command as a QOL improvement
  • bump reedline to test ExecuteHostCommand changes
  • allow current day to be highlighted
  • remove unused dependencies
  • fixes debug info not populating process information
  • allow last to work with ranges
  • update to the latest reedline and rusqlite
  • update default_config with new defaults
  • update to latest reedline
  • update to latest reedline
  • bump rust toolchain to 1.74.1
  • make char command const
  • allow ansi strip to work better on other nushell values
  • for ints, provide an option to convert all of them to filesizes with the into value command
  • bump to dev release of nushell 0.90.2
• robjwells created
  • Adjust examples in date commands
• geekvest created
  • Fix typos in comments
• Dorumin created
  • Improve sleep resolution
• ysthakur created
  • Canonicalize default-config-dir and plugin-path
  • Fix const string interpolation test
  • Add ConfigDirNotFound error
  • Specify which file not found in error
  • Disallow spreading lists automatically when calling externals
• jordanst3wart created
  • add examples for adding paths to PATH, and to load from a custom file in default_env.nu
• devyn created
  • Replace panics with errors in thread spawning
  • Add interleave command for reading multiple streams in parallel
  • Plugin StreamReader: fuse the iterator after an error
  • Add Goodbye message to ensure plugins exit when they are no longer needed
  • Add tee command for operating on copies of streams
  • Bidirectional communication and streams for plugins
  • Rename spans in the serialized form of Value
  • Print stderr streams to stderr in pipeline_data::print_if_stream()
  • Replace debug_assert! with assert! in Signature::check_names
  • Add support for the argument to zip being a closure
• klesh created
  • fix: start command should break on the first succeeded command result
• FilipAndersson245 created
  • replace Criterion with Divan
  • Benchmark changes
  • Fixes test 1 year more.
• kik4444 created
  • Fix touch to allow changing timestamps on directories, remake from #11760
• Olilin1 created
  • Fix: string_to_table in ssv.rs now filters comments. (issue #11997)
• WindSoilder created
  • deprecate str escape-glob command
  • Glob: don't allow implicit casting between glob and string
  • Don't expanding globs if user pass variables.
  • open, rm, umv, cp, rm and du: Don't globs if inputs are variables or string interpolation
  • make stderr works for failed external command
  • refactor: move du from platform to filesystem
  • Support redirect stderr and stdout+stderr with a pipe
  • Fix file completions which contains glob pattern
• PanGan21 created
  • fix: process empty headers in to md command
• IanManske created
  • Disable flaky network tests
  • Add upsert example
  • Move typos config to repo root
  • Add Value::coerce_str
  • Remove Record::from_raw_cols_vals_unchecked
  • Name the Value conversion functions more clearly
  • Remove required positional arg for some file system commands
  • Remove list of cell path support for select and reject
  • Fix error message for headers
  • Prevent duplicate keys for lazy make
  • Prevent duplicate records keys when decoding from nuon
  • Fix #11750: LazyRecord error message
  • Record cleanup
• ZzMzaw created
  • Allow clear command to clear terminal's history
• astral-l created
  • add binary data handling to bits commands
  • add --signed flag for binary into int conversions
• hustcer created
  • Making coreutils umkdir as the default mkdir
  • Fix Windows msvc *.msi builds
  • Upgrade openssl-src to fix riscv64 target building error
  • Upgrade hustcer/setup-nu action to v3.9 and Nu to v0.90.1 for workflows
• ayax79 created
  • Add columns to dataframe that are present in the schema but not present the Dataframe when applying schema.
  • Handling errors instead of propagating them up and killing the REPL
  • Handle configuration panics
  • wrapping run_repl with catch_unwind and restarting the repl on panic
  • Ability to cast a dataframe's column to a different dtype
• 132ikl created
  • Fix completions for directories with hidden files
• sholderbach created
  • Fix future lint by truncate(false) in touch
  • Fix cargo b -p nu-command --tests
  • Remove some unnecessary static Vecs
  • Remove unused Index(Mut) impls on AST types
  • Prune unused ShellError variants
  • Prune unused ParseError variants
  • Bump roxmltree to 0.19
  • Bump calamine to 0.24
  • Fix workspace.members for dependabot
• nils-degroot created
  • Add date support in from xlsx
• zhiburt created
  • nu-table: Improve table -a
• dmatos2012 created
  • fix format date based on users locale
• dannou812 created
  • move-Command Tests, Refactor, Fix
  • Fixed to/from toml date format
• lpnh created
  • style: correct keybinding name and improve formatting in default_config.nu
• kit494way created
  • separate commandline into subcommands
  • Fix panic in seq date
  • Fix commandline --cursor to return int
  • Allow comments in match blocks
• yurivict created
  • FreeBSD compatibility patches
• kubouch created
  • Remove unnecessary engine state clone
• CAD97 created
  • Simplify prompt tilde substitution
• nibon7 created
  • Bump polars from 0.36 to 0.37
  • Bump miette from 5.10.0 to 7.0.0
  • Apply clippy fixes
• 0323pin created
  • Allow building on NetBSD
• crides created
  • colored file-like completions
• KAAtheWiseGit created
  • Fix a panic when parsing empty file
• abusch created
  • Upgrade to ratatui 0.26
  • Allow specifying a cellpath in input list to get the value to display
• TrMen created
  • Enforce call stack depth limit for all calls