Nushell 0.107.0

Today, we're releasing version 0.107.0 of Nu. This release has a pretty big variety of improvements to various parts of Nushell. There are some new features for watch, find, and query xml, and changes to over 50 commands!

Where to get it

Nu 0.107.0 is available as pre-built binaries or from crates.io. If you have Rust installed you can install it using cargo install nu.

As part of this release, we also publish a set of optional plugins you can install and use with Nushell.

Table of contents

• Highlights and themes of this release
  • Watch out for new watch features!
  • Making find more cohesive
  • query xml improvements
• Changes
  • Breaking changes
    • find is now case-sensitive by default
    • New behavior for find --multiline
    • each now passes through null input
    • Change the output of format bits to big endian instead of native endian
    • Execution Order of Hooks Changed: env_change before pre_prompt
    • query xml returns scalar results when possible
    • Use fixed column name for query xml output
    • Add active column to overlay list
  • Additions
    • watch streams events
    • New watch --debounce option
    • get, select, reject can --ignore-case of cell-path
    • --endian flag for into binary
    • Spread null into collections or arguments
    • http subcommands now keep track of redirects
    • New random choice command in std-rfc
    • Add str align to std-rfc/str
    • JSON column support for stor and query db
    • Extend nodeset output formats for query xml
    • New keybinding: vichangemode
    • Add -h/--help flag to testbin
    • Other additions
  • Deprecations
    • Built-in random dice deprecated
    • Deprecate watch --debounce-ms
  • Other changes
    • random dice moved to std
    • http commands will now fail on invalid headers
    • http post now sends body serialized as pretty json
    • Improved error messages for misspelled flags
    • Improved default color theme
    • Reset content type for commands returning partial input
    • Additional changes
  • Bug fixes
    • Fix highlighting of aliases to external commands
    • input list Plays Nicely With Styled Input
    • Fixed spacing of help examples
    • Prevent detect columns from creating invalid records with duplicate keys
    • Improvements to errors
    • Other fixes
• Hall of fame
• Full changelog

Highlights and themes of this release [toc]

Watch out for new watch features! [toc]

This release features a couple additions to the watch command. The watch command can now stream events as a table output, which unlocks all new sorts of possibilities for how you can use it. Thank you to @Bahex for this amazing feature!

The watch command also has a "new" flag: --debounce. Fans of the watch command might be scratching their heads, since a very similar option already exists: --debounce-ms. This flag prevents events in rapid succession from being detected. The --debounce (no -ms) flag does the same thing, but with a proper duration value rather than an integer. Thanks to @lucascherzer for this addition!

Read more about watch event streaming, or read more about the --debounce flag and the corresponding deprecation of --debounce-ms.

Making find more cohesive [toc]

Following last release's improvements to the find command, this release has some more changes aimed at making find more cohesive. These include find being case-sensitive by default, and a more useful behavior for the --multiline flag. Thank you to @new-years-eve for your work both this release and last release in making find more useful!

Read more about find being case-sensitive, and more about the new --multiline behavior.

query xml improvements [toc]

The nu_plugin_query plugin ships with Nushell, and has a number of different commands that can help manipulate data formats that might otherwise be unwieldy. Thank you to @weirdan for spending some time to polish the query xml command. These changes should make it much nicer to use!

Read more about these improvements:

• Returning scalar results, rather than putting all output into tables
• Using a fixed column name, instead of a name based on the input expression
• Additional information with nodeset outputs

Changes [toc]

Breaking changes [toc]

find is now case-sensitive by default (#16323) [toc]

The find command is now case-sensitive by default in all modes. Previously, you could use the --ignore-case flag to make find case-insensitive in the --regex mode, but the default "search term mode" would always be case-insensitive. Now, both modes are case-sensitive by default, and you can use the --ignore-case flag to make them case-insensitive.

This change makes the different modes of find more consistent. It also makes find more consistent with other parts of Nushell, such as where's =~ operator and the recent case-sensitive cell-paths change.

alias find = find -i

New behavior for find --multiline (#16323) [toc]

Previously, find would always split multi-line input strings, making it impossible to perform proper multi-line regex matches unless a string was within list, table, or record. Now, the --multiline flag can be used to prevent this splitting, replacing its previous behavior of prepending (?m) to the regex.

> "hello\nworld" | find -mr 'lo\swo'
╭────────────╮
│ empty list │
╰────────────╯

> "hello\nworld" | find -mr 'lo\swo'
hello
worl

each now passes through null input (#16396) [toc]

When null is passed to the each command, it now returns null instead of passing null to the closure. For example, before this change:

> null | each { "something" } | describe
string
> null | each { "something" }
something
>

But after this change:

> null | each { "something" } | describe
nothing
> null | each { "something" }
>

Change the output of format bits to big endian instead of native endian (#16435) [toc]

While the most popular architectures use little endian, many people are used to reading binary numbers as big endian. However, until now, if you were in a little endian system, you would get:

> 258 | format bits
00000010 00000001

If you copied and pasted that as a number, you would get a surprising result:

> 0b00000010_00000001
513

Now, format bits always formats in big endian:

> 258 | format bits
00000001 00000010

Execution Order of Hooks Changed: env_change before pre_prompt (#16356) [toc]

Before this release env_change hooks would execute after pre_prompt hooks, and the prompt would be rendered after env_change hooks.

This meant env_change hooks got in between pre_prompt hooks and PROMPT_COMMAND.

Now, order of execution is as follows:

• env_change hooks
• pre_prompt hooks
• Rendering the prompt with PROMPT_COMMAND

query xml returns scalar results when possible (#16459) [toc]

Previously, query xml always returned a table, even for scalar results. Now scalar results will be returned as scalars.

Here's an example of where this would apply:

open -r tests/fixtures/formats/jt.xml
| query xml 'false()'

Before this change, this would return:

╭───┬─────────╮
│ # │ false() │
├───┼─────────┤
│ 0 │ false   │
╰───┴─────────╯

Now, this will just return false.

Use fixed column name for query xml output (#16461) [toc]

Previously, the query xml command outputs nodeset results in a table with a column name corresponding to the input expression. Now, the column name is fixed. This should make it easier to extract values from the output of query xml.

Before:

open -r tests/fixtures/formats/jt.xml
| query xml '//channel/title|//item/title'
# => ╭───┬───────────────────────────────────────────╮
# => │ # │           //channel/title|/...            │
# => ├───┼───────────────────────────────────────────┤
# => │ 0 │ JT                                        │
# => │ 1 │ Creating crossplatform Rust terminal apps │
# => ╰───┴───────────────────────────────────────────╯

Now:

open -r tests/fixtures/formats/jt.xml
| query xml '//channel/title|//item/title'
# => ╭───┬───────────────────────────────────────────╮
# => │ # │             string_value                  │
# => ├───┼───────────────────────────────────────────┤
# => │ 0 │ JT                                        │
# => │ 1 │ Creating crossplatform Rust terminal apps │
# => ╰───┴───────────────────────────────────────────╯

Add active column to overlay list (#16125) [toc]

overlay list now returns table instead of list of overlays. Before, only active overlays were included in overlay list. Now, hidden overlays will be included as well, and there is a column indicating whether a given overlay is hidden or not.

Additions [toc]

watch streams events (#16428) [toc]

watch command can now be used to return a stream of detected events instead of calling a closure with it's information, though using a closure is still possible and existing uses won't break.

In addition to this:

watch . {|operation, path, new_path|
    if $operation == "Write" and $path like "*.md" {
        md-lint $path
    }
}

Now this is also possible:

watch .
| where operation == Write and path like "*.md"
| each { md-lint $in.path }

New watch --debounce option (#16187) [toc]

The watch command now has --debounce flag, which takes a duration value. This will replace the --debounce-ms flag which takes an int rather than a duration, and will eventually take over its -d short flag.

get, select, reject can --ignore-case of cell-path (#16401) [toc]

get, select, reject commands now have a --ignore-case flag, which makes the commands interpret all cell-path arguments as completely case insensitive.

--endian flag for into binary (#16466) [toc]

Previously, converting values to binary with into binary could only do so in the native endianness of your platform. Using native endianness is still the default, but with the --endian flag, you get to choose:

> 258 | into binary --endian little
Length: 8 (0x8) bytes | printable whitespace ascii_other non_ascii
00000000:   02 01 00 00  00 00 00 00                             ••000000
> 258 | into binary --endian big
Length: 8 (0x8) bytes | printable whitespace ascii_other non_ascii
00000000:   00 00 00 00  00 00 01 02                             000000••

Note that this only affects int, float, filesize, bool and duration (i.e. it does not affect strings, dates and binary). Likewise, only the individual elements in tables and records are affected (not the table or record itself)

Spread null into collections or arguments (#16399) [toc]

null values can be used with the spread operator (...), behaving as if they were empty lists or records (whichever is appropriate for its place)

> [ 1 2 ...(null) ] == [ 1 2 ]
true
> { a:1 b:2 ...(null) } == { a:1 b:2 }
true

http subcommands now keep track of redirects (#16078) [toc]

The http subcommands can now maintain a list of redirects when using the --full flag. This will be stored in a new urls column:

> http get --full http://nushell.sh | get urls
╭───┬────────────────────────╮
│ 0 │ http://nushell.sh/     │
│ 1 │ http://www.nushell.sh/ │
╰───┴────────────────────────╯

{urls: important} | merge (http get --full <...>)

New random choice command in std-rfc (#16270) [toc]

The random choice command has been added as a new candidate for our standard library. This command can randomly sample a number of elements from a list:

> use std-rfc/random
> [1 2 3 4 5] | random choice 2
╭───┬───╮
│ 0 │ 1 │
│ 1 │ 3 │
╰───┴───╯

Add str align to std-rfc/str (#16062) [toc]

The std-rfc/str module has new command in this release, str align. This command will look for a substring (such as a delimiter), and add padding so that it is in the same column in all lines. It can also take a range to only align any number of lines.

> use std-rfc/str
> [ "one = 1", "two = 2", "three = 3", "four = 4", "five = 5" ] | str align '='
one   = 1
two   = 2
three = 3
four  = 4
five  = 5

JSON column support for stor and query db (#16258) [toc]

The stor create/insert/open and query db commands now support JSON and JSONB columns. This lets you store more kinds of structured data directly inside of an SQLite database without an explicit to/from step.

Here's an example of storing a simple table inside a stor database, and retrieving it as structured data with query db:

# create a table named my_table with a JSON column named data
> stor create -t my_table -c {data: json}
╭──────────┬────────────────╮
│ my_table │ [list 0 items] │
╰──────────┴────────────────╯

# inset a row into the data column containing a table
> {data: [1 2 3]} | stor insert -t my_table
╭──────────┬───────────────────╮
│          │ ╭───┬───────────╮ │
│ my_table │ │ # │   data    │ │
│          │ ├───┼───────────┤ │
│          │ │ 0 │ ╭───┬───╮ │ │
│          │ │   │ │ 0 │ 1 │ │ │
│          │ │   │ │ 1 │ 2 │ │ │
│          │ │   │ │ 2 │ 3 │ │ │
│          │ │   │ ╰───┴───╯ │ │
│          │ ╰───┴───────────╯ │
╰──────────┴───────────────────╯

# retrieve the data column from the table as structured data
> stor open | query db "select data from my_table"
╭───┬───────────╮
│ # │   data    │
├───┼───────────┤
│ 0 │ ╭───┬───╮ │
│   │ │ 0 │ 1 │ │
│   │ │ 1 │ 2 │ │
│   │ │ 2 │ 3 │ │
│   │ ╰───┴───╯ │
╰───┴───────────╯

Extend nodeset output formats for query xml (#16465) [toc]

query xml now can output additional information when it returns Nodesets:

• --output-type enables type column. It includes the node type found, e.g. element or comment
• --output-names adds local_name, prefixed_name and namespace columns
• --output-string-value (re)adds string_value column

If you're using any of the --output-* switches, and want string_value column to show up, pass --output-string-value explicitly. In the absence of any --output-* attributes, --output-string-value is assumed to be on.

New keybinding: vichangemode (#16327) [toc]

You can now set bindings which change the Vi mode. To do so send a vichangemode event with the mode field to set normal, insert, or visual

$env.config.keybindings ++=
	[{
	    name: modechangetest
	    modifier: control
	    keycode: "char_["
	    mode: [vi_normal, vi_insert]
	    event: {send: vichangemode, mode: normal}
	}]

The available modifiers and keycodes, remain limited to single character bindings with modifiers. We don't yet provide access to the key-chord parsing of the vi mode.

Add -h/--help flag to testbin (#16196) [toc]

nu --testbin has a new flag -h to show available <bins>

> nu --testbin -h
Usage: nu --testbin <bin>
<bin>:
chop -> With no parameters, will chop a character off the end of each line
cococo -> Cross platform echo using println!()(e.g: nu --testbin cococo a b c)
echo_env -> Echo's value of env keys from args(e.g: nu --testbin echo_env FOO BAR)
echo_env_mixed -> Mix echo of env keys from input(e.g: nu --testbin echo_env_mixed out-err FOO BAR; nu --testbin echo_env_mixed err-out FOO BAR)
echo_env_stderr -> Echo's value of env keys from args to stderr(e.g: nu --testbin echo_env_stderr FOO BAR)
echo_env_stderr_fail -> Echo's value of env keys from args to stderr, and exit with failure(e.g: nu --testbin echo_env_stderr_fail FOO BAR)
fail -> Exits with failure code 1(e.g: nu --testbin fail)
iecho -> Another type of echo that outputs a parameter per line, looping infinitely(e.g: nu --testbin iecho 3)
input_bytes_length -> Prints the number of bytes received on stdin(e.g: 0x[deadbeef] | nu --testbin input_bytes_length)
meow -> Cross platform cat (open a file, print the contents) using read_to_string and println!()(e.g: nu --testbin meow file.txt)
meowb -> Cross platform cat (open a file, print the contents) using read() and write_all() / binary(e.g: nu --testbin meowb sample.db)
nonu -> Cross platform echo but concats arguments without space and NO newline(e.g: nu --testbin nonu a b c)
nu_repl -> Run a REPL with the given source lines, it must be called with `--testbin=nu_repl`, `--testbin nu_repl` will not work due to argument count logic
relay -> Relays anything received on stdin to stdout(e.g: 0x[beef] | nu --testbin relay)
repeat_bytes -> A version of repeater that can output binary data, even null bytes(e.g: nu --testbin repeat_bytes 003d9fbf 10)
repeater -> Repeat a string or char N times(e.g: nu --testbin repeater a 5)

Other additions [toc]

• The commandline edit command has a new flag --accept (or -A) which immediately executes the resulting commandline. (#16193)

• The bench command in the standard library now shows a progress bar (or spinner) on terminals which support osc 9;4. (#16245)

• The to html command has a new --raw flag, which doesn't escape HTML tags in the input. (#16373)

• Added information to the run-external and help descriptions that indicates these commands or any with the same names are automatically used for running externals and the --help flags respectively. (#16493)

• std-rfc/kv commands now take an optional --table (-t) argument which allows a custom table name to be used. If not specified, the current std-kv-store table will be used. (#16450)

Deprecations [toc]

Built-in random dice deprecated [toc]

With random dice being moved to std, the built-in command is no longer necessary. See random dice moved to std for more information.

Deprecate watch --debounce-ms (#16187) [toc]

With the new watch --debounce option, the --debounce-ms option is no longer necessary. Use watch --debounce with a duration value instead.

Other changes [toc]

random dice moved to std (#16420) [toc]

The random dice command has been rewritten in Nushell and moved to the standard library. The random dice built-in is still available with a deprecation error, but will be removed in 0.108. The new command can be used as follows:

> use std/random
> random dice
╭───┬───╮
│ 0 │ 6 │
╰───┴───╯

It's behavior, parameters, and defaults are the same.

http commands will now fail on invalid headers (#16078) [toc]

Before, non-UTF-8 headers would be silently ignored. Now, these will cause an error. Here's an example which uses two Nushell instances to demonstrate this.

In one Nushell instance, we create something resembling an HTTP server on port 1234:

["HTTP/1.1 200 OK\r\nContent-Length: 0\r\nx-header: " 0x[1F FF AA AA] "\r\n\r\n"]
| each { into binary }
| bytes collect
| ncat -l 1234

In another instance, we connect to port 1234. The x-header is not valid UTF-8, which now produces an error:

> http get --full --allow-errors http://localhost:1234
Error: nu::shell::network_failure

  × Network failure
   ╭─[entry #1:1:1]
 1 │ http get --full --allow-errors http://localhost:1234
   · ────┬───
   ·     ╰── protocol: http parse fail: invalid header value
   ╰────

http post now sends body serialized as pretty json (#16078) [toc]

Before, http post would serialize values as raw JSON. Now, the JSON will be serialized into the pretty format. Note that this increases the body size.

POST / HTTP/1.1
Host: localhost:1234
User-Agent: nushell
Accept: */*
Content-Type: application/json
accept-encoding: gzip
Content-Length: 13

{"foo":"bar"}

POST / HTTP/1.1
accept-encoding: gzip
content-length: 18
user-agent: nushell
accept: */*
host: localhost:1234
content-type: application/json; charset=utf-8

{
  "foo": "bar"
}

Improved error messages for misspelled flags (#16427) [toc]

Previously, the help text for a missing flag would list all of them, which could get verbose on a single line:

> ls --full-path
Error: nu::parser::unknown_flag

  × The `ls` command doesn't have flag `full-path`.
   ╭─[entry #26:1:4]
 1 │ ls --full-path
   ·    ─────┬─────
   ·         ╰── unknown flag
   ╰────
  help: Available flags: --help(-h), --all(-a), --long(-l), --short-names(-s), --full-paths(-f), --du(-d),
        --directory(-D), --mime-type(-m), --threads(-t). Use `--help` for more information.

The new error message only suggests the closest flag:

> ls --full-path
Error: nu::parser::unknown_flag

  × The `ls` command doesn't have flag `full-path`.
   ╭─[entry #45:1:4]
 1 │ ls --full-path
   ·    ─────┬─────
   ·         ╰── unknown flag
   ╰────
  help: Did you mean: `--full-paths`?

Improved default color theme (#16509) [toc]

We changed the default theme to use the ANSI default color (39m) instead of white (37m). This finally makes the default theme usable in the context of light terminal color settings. On dark terminal palettes this change should have no impact.

Comparison of white vs default color on Solarized Light theme, before and after:

Reset content type for commands returning partial input (#16500) [toc]

The following commands no longer preserve content_type element of the input metadata:

• bytes at - bytes 3..5 of an image/png stream are not valid image/png stream themselves
• bytes remove - when you remove some bytes, the content type likely becomes invalid
• first - first 5 bytes of an image/png stream are not valid image/png stream themselves
• skip - bytes after 5th of an image/png stream are not valid image/png stream
• take - see first
• str substring - a random string slice of application/json isn't likely to be a valid application/json value

Additional changes [toc]

• Trying to execute a non-existent script file used to return a confusing error pointing to an internal source code path. That error now points to the script file argument in the commandline. (#16273)

• format date will respect $env.LANG and override it with $env.LC_ALL. (#16369)

• Required named parameters will now always appear before regular named parameters/flags in the help output. (#16476)

• query xml now has xml: namespace prefix available by default, without the need to specify it via --namespaces. (#16472)

Bug fixes [toc]

Fix highlighting of aliases to external commands (#15408) [toc]

Aliases to external commands will now be properly highlighted as external commands.

> $env.config.highlight_resolved_externals = true
> $env.config.color_config = { shape_external: red_bold, shape_external_resolved: yellow_bold, shape_internalcall: blue_bold }
> alias internal-alias = echo
> alias external-alias = coreutils echo
> alias unresolvable-alias = notacommand
> alias bash = notacommand
> "internal-alias; external-alias; unresolvable-alias; bash;" | nu-highlight
internal-alias; external-alias; unresolvable-alias; bash;

> $env.config.highlight_resolved_externals = true
> $env.config.color_config = { shape_external: red_bold, shape_external_resolved: yellow_bold, shape_internalcall: blue_bold }
> alias internal-alias = echo
> alias external-alias = coreutils echo
> alias unresolvable-alias = notacommand
> alias bash = notacommand
> "internal-alias; external-alias; unresolvable-alias; bash;" | nu-highlight
internal-alias; external-alias; unresolvable-alias; bash;

input list Plays Nicely With Styled Input (#16276) [toc]

input list had some trouble dealing with ANSI styled inputs, such as:

• not resetting terminal styles after each item, leading to items' styles leaking and affecting how the following items are rendered
• in fuzzy mode, resetting the styles a little too much after emphasizing matching sections

Here's a before/after comparison:

	Select	Fuzzy
❌
✅

Fixed spacing of help examples (#16353) [toc]

help command used to trim the outputs of examples, which could result in inconsistent white space:

$env.config.table = {mode: light, padding: {left: 1}, header_on_separator: false}

  Find and replace all occurrences of found string in record using regular expression
  > { KeyA: abc, KeyB: abc, KeyC: ads } | str replace --all --regex 'b' 'z' KeyA KeyC
  KeyA   azc
   KeyB   abc
   KeyC   ads

  Find and replace all occurrences of found string in record using regular expression
  > { KeyA: abc, KeyB: abc, KeyC: ads } | str replace --all --regex 'b' 'z' KeyA KeyC
   KeyA   azc
   KeyB   abc
   KeyC   ads

Prevent detect columns from creating invalid records with duplicate keys (#16527) [toc]

Previously detect columns created records (rows) with duplicate key names under some circumstances. The resulting table behaved inconsistently with different commands:

> let data = "meooooow cat\nkitty kitty woof"
> $data | detect columns
╭───┬──────────┬───────╮
│ # │ meooooow │  cat  │
├───┼──────────┼───────┤
│ 0 │ kitty    │ kitty │
╰───┴──────────┴───────╯
> $data | detect columns | get 0.cat
woof

Now, this will result in an error suggesting using detect columns --guess or parse:

> let data = "meooooow cat\nkitty kitty woof"
> $data | detect columns
Error: nu::shell::failed_to_detect_columns

  × Failed to detect columns
   ╭─[entry #48:1:1]
 1 │ $data | detect columns
   · ──┬──   ───────┬──────
   ·   │            ╰── tried to detect columns here
   ·   ╰── value coming from here
   ╰────

Improvements to errors [toc]

This release has a number of improvements to errors in specific circumstances:

• Improve errors for subcommands without a corresponding parent command (#16529)

  Added missing parent ('namespace') commands to improve error reporting:

  • attr
  • detect
  • error
  • query
  • registry

• Before, attempting to run an external command when $env.PATH was not set could result in a confusing error. This release improves the error. (#16410)

• Errors which reference values originating from command examples in the output of scope commands should now be less confusing. (#16395)

• where should now error at parse-time on row condition expressions which are not booleans, in addition to at run-time (#16175)

• Type errors involving boolean literals are now more precise. (#16408)

• The error shown when attempting to run a non-const command in a const-eval context is now more precise. (#16393)

Other fixes [toc]

• Short flags which require a value should now properly be tracked by the parser. Previously, something like the -a in table -a wouldn't be properly highlighted if no value for -a was provided. This also means that the ast command will now properly account for these flags. (#16376)

• Parse-time pipeline type checking now properly supports commands with multiple pipeline output types for the same pipeline input type. (#16111)

• Previously, a record key = would not be quoted in to nuon, producing invalid nuon output. It wasn't quoted in other string values either, but it didn't cause problems there. Now any string containing = gets quoted. (#16440)

• path relative-to works better for case-insensitive filesystems, this works on Windows or macOS: "/etc" | path relative-to "/Etc" (#16310)

• Stepped range literals where ..= precedes the second value no longer cause a parser panic. e.g: random int 1..=3..5 (#16231)

• Arguments to external commands with subexpressions like ^/bin/echo ProxyCommand=($'hello') are now parsed correctly (#16346)

• The index column in a table will get a highlight color if $env.config.table.header_on_separator = true (#16377)

• In explore, cursor position and search highlighting works better on non-ASCII characters. (#16325)

• The gstat plugin will show correct branch in detached HEAD if commit hash starts with 0 (#16309)

• Interrupting a script with ctrl-c now properly kills background jobs on exit. (#16285)

• Argument variables for custom commands no longer leak outside of their scope. (#16262)

• into sqlite will respect $env.PWD rather than current working directory. (#16349)

• std help displays better on binary examples. (#16354)

• Using polars open --type tsv will now correctly use tabs as the delimiter. (#16524)

• The polars fill-null command previously only allowed dataframes as inputs. Now, polars fill-null also accepts expressions as an input type. (#16444)

Hall of fame [toc]

Thanks to all the contributors below for helping us solve issues, improve documentation, refactor code, and more! 🙏

Full changelog [toc]