Nushell 0.94.0

Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your command line. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful command line pipelines.

Today, we're releasing version 0.94.0 of Nu. This release adds case-preserving environment, changes to path handling, raw string literals, and improvements to streaming!

Where to get it

Nu 0.94.0 is available as pre-built binaries or from crates.io. If you have Rust installed you can install it using cargo install nu.

As part of this release, we also publish a set of optional plugins you can install and use with Nu. To install, use cargo install nu_plugin_<plugin name>.

Table of content

• Themes of this release / New features
  • Fixing path and PWD handling
  • Case-preserving environment
  • Streaming all the things
  • New language feature: raw strings
  • Removal of deprecated features
  • Shell integration config
  • Hall of fame
    • Bug fixes
    • Enhancing the documentation
  • Our set of commands is evolving
    • New commands
      • sys subcommands
      • ps support on BSD systems
      • debug profile --lines
    • Changes to existing commands
      • Making range semantics consistent
      • parse
      • scope commands
      • which
      • describe
      • Streaming more commands
      • cd and pwd
      • collect
      • complete
      • each
      • ls
      • bytes build
      • from nuon and to nuon
      • save
      • hide-env
      • PWD fixes
      • nu-highlight
      • into sqlite
      • stor insert
      • decode base64 and encode base64
      • char
    • Deprecated commands
      • sys
      • str contains --not
    • Removed commands
      • lazy make
      • describe --collect-lazyrecords
      • List of environment variables support in with-env
  • For plugin developers
    • Breaking API changes
    • Breaking protocol changes
• Breaking changes
• Full changelog

Themes of this release / New features [toc]

Fixing path and PWD handling [toc]

One of our contributors, @YizhePKU, has started an effort to refactor how Nushell internally handles the current working directory and paths in general. Namely, Nushell aggressively canonicalizes paths instead of using logical paths which can lead to unintuitive or annoying behavior (#2175). Thanks to their work, $env.PWD and the pwd command now support logical paths. With time, we intend to make similar to changes to other commands to make them more intuitive and consistent with the rest of Nushell. Another goal with these changes is to eliminate bugs in commands, as some commands incorrectly use the Nushell process's current working directory instead of $env.PWD. So far, the grid, path type, and touch --reference commands have been fixed, and we are aiming to bring more fixes in the next release. We kindly ask for your patience as we rework this part of Nushell, and we would appreciate if you would report any issues you encounter!

Case-preserving environment [toc]

In an effort to allow more ergonomic cross-platform scripting, we have changed the environment in Nushell to be case-preserving (#12701). That is, environment variables keep their initial casing but can be accessed in a case-insensitive manner within Nushell. This means that accessing $env.PATH or $env.Path will now work on both unix-based systems and Windows systems. To get environment variables in a case-sensitive manner, you can use get -s.

$env | get -s PATH

Please note that there are still a few things that have to be ironed out with this new behavior. For example, it is currently unclear what load-env { FOO: BAR, foo: bar } should do. So, there may be more changes in the near future.

Streaming all the things [toc]

Another major effort with this release was to make streaming more consistent and widespread (#12774, #12897, #12918, and more). For example, Nushell now passes operating system pipe descriptors directly to external commands, meaning there should be an improvement to throughput for some pipelines. Additionally, several commands were reworked so that they are now able to stream their output in more cases instead of having to collect it all in memory beforehand. The commands in question include from csv, to csv, from tsv, to tsv, from json --objects, take, skip, first, last, bytes ends-with, bytes collect, str join, into string, and into binary. Of course, the plan is to add more commands to this list in the future, so this is only the beginning!

New language feature: raw strings [toc]

This release adds a new type of string literal to the language: raw strings (#9956). These are just like single quoted strings, except that raw strings may also contain single quotes. This is possible, since raw strings are enclosed by a starting r#' and an ending '#. This syntax should look familiar to users of Rust.

r#'some text'# == 'some text' # true

r#'contains 'quoted' text'# == "contains 'quoted' text"

Additional # symbols can be added to the start and end of the raw string to enclose one less than the same number of # symbols next to a ' symbol in the string.

r###'this text has multiple '## symbols'###

Removal of deprecated features [toc]

Last release, we deprecated the dataframe commands and feature in favor of the new polars plugin. See the previous release notes for more information and migration steps. In this release, the dataframe commands have been removed as planned alongside the dataframes cargo feature. Similarly, this release removes lazy records from the language following their deprecation. This includes the removal of the lazy make command and the --collect-lazyrecords flag on describe. One consequence of this change is that the sys command will now return a regular record and will take a minimum of 400ms to complete (to sample CPU usage). As such, the sys command has been deprecated, and new subcommands have been added in its place (sys mem, sys cpu, sys host, sys disks, sys net, sys temp, and sys users).

Shell integration config [toc]

With this release, we have changed the shell_integration setting in the config to be a record of multiple settings (#12629):

shell_integration: {
  # osc2 abbreviates the path if in the home_dir, sets the tab/window title, shows the running command in the tab/window title
  osc2: true
  # osc7 is a way to communicate the path to the terminal, this is helpful for spawning new tabs in the same directory
  osc7: true
  # osc8 is also implemented as the deprecated setting ls.show_clickable_links, it shows clickable links in ls output if your terminal supports it
  osc8: true
  # osc9_9 is from ConEmu and is starting to get wider support. It's similar to osc7 in that it communicates the path to the terminal
  osc9_9: false
  # osc133 is several escapes invented by Final Term which include the supported ones below.
  # 133;A - Mark prompt start
  # 133;B - Mark prompt end
  # 133;C - Mark pre-execution
  # 133;D;exit - Mark execution finished with exit code
  # This is used to enable terminals to know where the prompt is, the command is, where the command finishes, and where the output of the command is
  osc133: true
  # osc633 is closely related to osc133 but only exists in visual studio code (vscode) and supports their shell integration features
  # 633;A - Mark prompt start
  # 633;B - Mark prompt end
  # 633;C - Mark pre-execution
  # 633;D;exit - Mark execution finished with exit code
  # 633;E - NOT IMPLEMENTED - Explicitly set the command line with an optional nonce
  # 633;P;Cwd=<path> - Mark the current working directory and communicate it to the terminal
  # and also helps with the run recent menu in vscode
  osc633: true
  # reset_application_mode is escape \x1b[?1l and was added to help ssh work better
  reset_application_mode: true
}

This change provides greater granularity, allowing you to disable any integrations that may cause issues in certain terminals.

Hall of fame [toc]

Bug fixes [toc]

Thanks to all the contributors below for helping us solve issues and bugs 🙏

Enhancing the documentation [toc]

Thanks to all the contributors below for helping us making the documentation of Nushell commands better 🙏

Our set of commands is evolving [toc]

New commands [toc]

sys subcommands [toc]

This release adds new sys subcommands corresponding to each column of the record returned by sys (#12747):

• sys host
• sys cpu
• sys mem
• sys net
• sys disks
• sys temp
• sys users (new!)

The output of these commands should be the same as before as the corresponding sys column except for a few differences:

• sys host does not have a sessions column. This table has been moved to the new sys users command (#12787). (The old sys command still has the sessions column.)
• The boot_time column in sys host and sys | get host is now a date value instead of a formatted string (#12846).

ps support on BSD systems [toc]

In #12892, @devyn has added support for the ps command on FreeBSD, NetBSD, and OpenBSD!

debug profile --lines [toc]

The debug profile command now has a --lines flag with #12930. Providing this will flag will add a file and line column to the output of debug profile.

Changes to existing commands [toc]

Making range semantics consistent [toc]

Ranges are inclusive in the upper bound by default in Nushell (e.g., 1..2 or 1..=2 gives [1, 2]). However, some commands treated the upperbound as exclusive. With #12894 this has been fixed for the following commands:

• str substring
• str index-of
• detect columns

To specify an exclusive upperbound, use the ..< form for ranges (e.g., 1..<2).

parse [toc]

To support streaming output, parse now operates/matches on a single line at a time only if provided with the streaming output of an external command, file, or raw byte stream. For example, these snippets will try to match each line separately:

^cat file.txt | parse -r "some regex"
^cat file.txt | take 1024 | parse -r "some regex"

open file.txt | parse -r "some regex"
open --raw file.txt | parse -r "some regex"
open --raw file.txt | skip 1024 | parse -r "some regex"

The old behavior was to collect all of the output the external command or byte stream and then run the regex across the whole string. To mimic this behavior, you can use the collect command or store the output in a variable:

^cat file.txt | collect | parse -r "some regex"

let text = open file.txt
$text | parse -r "some regex"

Note that this change does not affect normal value streams like:

[(open foo.txt) (open bar.txt)] | parse -r "some regex"

In this case, the regex is run separately over each string value in the stream. With the proper regex flags, the regex is able to match across multiple lines within the same string value.

Note that parse may see more breaking changes in the next release to simplify these behaviors.

scope commands [toc]

After #12832, several columns from the output of scope commands have been removed:

• is_builtin
• is_plugin
• is_custom
• is_keyword
• is_extern

All of these columns are mutually exclusive (or one column always implies another), so these columns have instead been replaced with a single type column. It will contain one of the following string values:

• built-in
• plugin
• custom
• keyword
• external
• alias

which [toc]

Related to the changes for scope commands, the type column reported by which can now be:

• plugin or keyword instead of built-in
• external instead of custom for declared extern commands

describe [toc]

With the --detailed flag, the stdout, stderr, and exit_code columns have been removed for external streams / byte streams. Instead, the origin column now reports unknown, file, or external for byte streams, and the type column will be byte stream, string (stream), or binary (stream).

Streaming more commands [toc]

This release changes a bunch of commands to stream where possible (#12897):

• first and last now support byte stream input and will return the first or last specified number of bytes (without collecting the whole stream into memory at once).
• take and skip also now support byte streams input and will return new byte streams.
• str join will now stream string output if it is given a stream of string values.
• bytes collect can also stream its output now.
• bytes ends-with now supports byte streams and does not need to collect the entire stream output in memory at once (#12887).
• from json --objects now streams its output if its input is a stream (#12949).
• into binary and into string will now apply a type hint to byte streams and will simply return the input stream without consuming or draining it.

cd and pwd [toc]

Thanks to @YizhePKU in #12603, the cd command now sets $env.PWD to a logical path by default. Instead, there is now a --physical flag for cd which will canonicalize the path before setting it as the PWD. Similarly, pwd now has a --physical flag to resolve symlinks before reporting the current working directory.

collect [toc]

After #12788, the closure parameter for collect is now optional. If no closure is provided, the collected pipeline input is simply returned as a value.

complete [toc]

Previously, if an external command was terminated by a signal, the exit code returned from complete would be -1. Also, if the external command core dumped, then an error value would be present in the exit_code column. Now with #12774, if an external command was terminated by a signal (this includes core dumps), then the exit_code column will contain the negation of the signal number (to differentiate it from a regular exit code).

Also, the o+e>| combined pipe redirection now works properly with complete. That is, the stdout and stderr of the preceding external command are merged prior to be passed to complete, and the combined output will be present in the stdout column of the record returned by complete (#12818).

each [toc]

Thanks to @merelymyself, the each command now bubbles up errors in certain cases where it would previously ignore them (#12412).

ls [toc]

In a similar change, the ls command will now return errors instead of empty entries if it fails to list a path (#12033). Additionally, ls has received a bug fix with #12625 so that it works in a directory with [ or ] in its path.

bytes build [toc]

With #12685, the bytes build command now accepts integers as rest arguments (each integer number is treated as a byte).

from nuon and to nuon [toc]

The nuon commands can now serialize and deserialize cell path values after #12718.

save [toc]

With #12833, custom converters now work again for the save command. E.g., save output.custom now uses the to custom command if it exists (note that this is currently not done for singular string value inputs).

hide-env [toc]

A long-standing bug with hide-env is that it would not hide environment variables from external commands / child processes in certain cases. With #12901, this issue has finally been fixed.

PWD fixes [toc]

Some commands are incorrectly using the current working directory of the shell process, instead of the internal $env.PWD tracked in the engine state. This release fixes a few of these commands:

• path type (#12975)
• touch --reference (#12976)
• grid (#12947)

nu-highlight [toc]

nu-highlight has received two bug fixes. Namely, highlighting not has been fixed (#12815) and the extra output present when highlighting list spreads has been fixed (#12793).

into sqlite [toc]

Columns with spaces are now supported by into sqlite, since it will wrap the columns with backticks (#12759).

stor insert [toc]

Thanks to @ExaltedBagel in #12820, stor insert now escapes single quotes.

decode base64 and encode base64 [toc]

The alphabet used for --character-set binhex has been corrected in #12757.

char [toc]

In #12867, an issue with char was fixed where it can panic if a list is spread after the --integer or --unicode flags.

Deprecated commands [toc]

sys [toc]

With the removal of lazy records, the sys command will now return a regular record. This means it has to compute all of its columns before returning. Since the sys command samples CPU usage over a 400ms period, then this would be unideal if the only information you needed from the command was, e.g., the host information. Because of this, the command has been deprecated in favor of the new sys subcommands as listed in new commands.

str contains --not [toc]

This release deprecates the --not flag for str contains, since none of the other string commands have a similar flag (#12837). Instead of the --not flag, you can use the not operator in one of the following ways:

# using parentheses
not ("foobar" | str contains foo)

# using a pipe with the `$in` variable
"foobar" | str contains foo | not $in

Removed commands [toc]

lazy make [toc]

The lazy make command has been removed in #12682. See removal of deprecated features for more information.

describe --collect-lazyrecords [toc]

The --collect-lazyrecords flag has been removed from describe, since lazy records have been removed from the language (#12682).

List of environment variables support in with-env [toc]

This release removes support for the list of environment variable form of with-env that was deprecated in the previous release. Going forward, only the record form is supported.

For plugin developers [toc]

Breaking API changes [toc]

• The ExternalStream type has been replaced entirely with ByteStream, which represents a single (possibly typed) stream of bytes. Through the plugin layer, only the Read variant is used. It is no longer possible to get stderr or exit_code from an external command by piping it into a plugin command directly.
• The ListStream type now requires a Span.
• Config's shell_integration field has been modified and split into several fields. It was previously a single bool.
• LazyRecord has been removed, but this was not allowed in plugins anyway.

Breaking protocol changes [toc]

• The ExternalStream pipeline data header type has been replaced with ByteStream.
• The ListStream pipeline data header type now requires a span.

Breaking changes [toc]

• #12939 Remove list support in with-env
• #12894 fix range semantic in detect_columns, str substring, str index-of
• #12930 Small improvements to debug profile
• #12889 Remove dataframes crate and feature
• #12897 Add string/binary type color to ByteStream
• #12832 Use CommandType in more places
• #12774 Replace ExternalStream with new ByteStream type
• #12846 Fix sys panic
• #12837 remove --not flag for 'str contains'
• #12747 Add sys subcommands
• #12669 Polars lazy refactor
• #12524 ListStream touchup
• #12682 Remove lazy records
• #12629 overhaul shell_integration to enable individual control over ansi escape sequences

Full changelog [toc]

• IanManske created
  • Fix panic when redirecting nothing
  • Do not propagate glob creation error for external args
  • Allow byte streams with unknown type to be compatible with binary
  • Revert "Polars lazy refactor (#12669)"
  • Use cwd in grid
  • Remove list support in with-env
  • Remove dataframes crate and feature
  • Take owned Read and Write
  • Make get_full_help take &dyn Command
  • Clear environment for child Commands
  • Use CommandType in more places
  • Preserve metadata in more places
  • Support ByteStreams in bytes starts-with and bytes ends-with
  • Add Span merging functions
  • Bytestream touchup
  • Replace ExternalStream with new ByteStream type
  • Remove stack debug assert
  • Remove usages of Call::positional_nth
  • Fix sys panic
  • Fix char panic
  • Update PR template
  • Fix custom converters with save
  • Fix pipe redirection into complete
  • Fix raw strings as external argument
  • Refactor the CLI code a bit
  • Fix syntax highlighting for not
  • Pass Stack ref to Completer::fetch
  • Refactor message printing in rm
  • Fix/simplify cwd in benchmarks
  • Refactor parse command
  • Add sys users command
  • Fix list spread syntax highlighting
  • describe refactor
  • Add sys subcommands
  • ListStream touchup
  • Refactor flattening to reduce intermediate allocations
  • Bump base64 to 0.22.1
  • Fix typo
  • Remove some macros
  • Remove lazy records
• YizhePKU created
  • Fix touch --reference using PWD from the environment
  • Fix path type using PWD from the environment
  • Fixes . expanded incorrectly as external argument
  • Rewrite run_external.rs
  • Remove std::env::set_current_dir() call from EngineState::merge_env()
  • Implement PWD recovery
  • Migrate to a new PWD API (part 2)
  • Fix PWD cannot point to root paths
  • Migrate to a new PWD API
  • Path expansion no longer removes trailing slashes
  • Change environment variables to be case-preserving
• fdncred created
  • make polars plugin use mimalloc
  • Revert "Remove std::env::set_current_dir() call from EngineState::merge_env()"
  • add math min and math max to bench command
  • sync up with reedline changes
  • make it clearer what is being loaded with --log-level info
  • cleanup osc calls for shell_integration
  • overhaul shell_integration to enable individual control over ansi escape sequences
  • add raw-string literal support
  • change wix install method from perMachine to perUser
• devyn created
  • Make from json --objects streaming
  • Implement streaming I/O for CSV and TSV commands
  • Add support for the ps command on FreeBSD, NetBSD, and OpenBSD
  • Update mimalloc to 0.1.42
  • Add string/binary type color to ByteStream
  • collect: don't require a closure
  • Exclude polars from ensure_plugins_built(), for performance reasons
  • Fix the way the output of table is printed in print()
  • Make the message when running a plugin exe directly clearer
  • Fix trailing slash in PWD set by cd
  • Upgrade to interprocess 2.0.0
  • Flush on every plugin Data message
  • Add toolkit release-pkg windows for Windows release pkg builds
  • Make bytes build accept integer values as individual bytes
  • Bump version to 0.93.1
  • Fix Windows Terminal profile installation
  • Bump version to 0.93.0
  • Bump reedline to 0.32.0
• rgwood created
  • explore refactoring+clarification
  • explore cleanup: remove+move binary viewer config
  • explore: consolidate padding config, handle ByteStream, tweak naming+comments
  • Remove accidentally committed file
  • explore: adopt anyhow, support CustomValue, remove help system
• WindSoilder created
  • fix std help
  • add some completion tests
  • fix range semantic in detect_columns, str substring, str index-of
  • allow define it as a variable inside closure
  • allow passing float value to custom command
  • make better messages for incomplete string
  • allow raw string to be used inside subexpression, list, and closure
  • Allow ls works inside dir with [] brackets
• kubouch created
  • Fix leftover wrong column name
  • Small improvements to debug profile
• app/dependabot created
  • Bump shadow-rs from 0.27.1 to 0.28.0
  • Bump actions/checkout from 4.1.5 to 4.1.6
  • Bump rust-embed from 8.3.0 to 8.4.0
  • Bump interprocess from 2.0.1 to 2.1.0
  • Bump softprops/action-gh-release from 2.0.4 to 2.0.5
  • Bump actions/checkout from 4.1.4 to 4.1.5
  • Bump rmp-serde from 1.2.0 to 1.3.0
  • Bump actions/checkout from 4.1.3 to 4.1.4
  • Bump crate-ci/typos from 1.20.10 to 1.21.0
• NotTheDr01ds created
  • Add completer for std help
  • Fixed small error in the help-examples for the get command
  • Fixed a nitpick usage-help error - closure v. block
  • Search terms for compact command
  • Add example and search term for 'repeat' to the fill command
• sholderbach created
  • Remove unused dependencies
  • Shrink Value by boxing Range/Closure
  • Add SOPs for dealing with adding deps/crates
  • Update interprocess to 2.0.1
  • Specify the required minimum chrono version
  • Pin base64 to the fixed patch version
  • Avoid taking unnecessary ownership of intermediates
  • Update PLATFORM_SUPPORT regarding feature flags
  • Eliminate dead code in nu-explore
  • Fix clippy::wrong_self_convention in polars plugin
  • Rework for new clippy lints
  • Update the bundled readme in release archives
• ayax79 created
  • fixed syntax shape requirements for --quantiles option for polars summary
  • Allow custom value operations to work on eager and lazy dataframes interchangeably.
  • Polars lazy refactor
• cablehead created
  • Add a passing test for interactivity on slow pipelines
  • feat: add an echo command to nu_plugin_example
• ExaltedBagel created
  • Fix panic when exploring empty dictionary
  • Fix improperly escaped strings in stor insert
  • Enable columns with spaces for into_sqlite by adding quotes to column names
• cptpiepmatz created
  • Add Stack::stdout_file and Stack::stderr_file to capture stdout/-err of external commands
• francesco-gaglione created
  • Merged tests to produce a single binary
• Skyppex created
  • remove --not flag for 'str contains'
• FilipAndersson245 created
  • Tango migration
• amtoine created
  • improve NUON documentation
  • add support for cell-paths to NUON
• szepeviktor created
  • Minimize future false positive typos
• lavafroth created
  • fix: prevent relative directory traversal from crashing
• maxim-uvarov created
  • add more tests to the polars plugin
• rtpg created
  • Report errors that occur on file operations in ls
• hustcer created
  • Upgrade Nu to v0.93.0 for nightly and release workflow
• merelymyself created
  • Prevent each from swallowing errors when eval_block returns a ListStream