Loading Data
Earlier, we saw how you can use commands like ls, ps, date, and sys host to load information about your files, processes, time of day, and the system itself. Each command gives us a table of information that we can explore. There are other ways we can load in a table of data to work with.
Opening files
One of Nu's most powerful assets in working with data is the open command. It is a multi-tool that can work with a number of different data formats. To see what this means, let's try opening a json file:
open editors/vscode/package.json
# => ╭──────────────────┬────────────────────────────────────────────────────────────────╮
# => │ name │ lark │
# => │ description │ Lark support for VS Code │
# => │ author │ Lark developers │
# => │ license │ MIT │
# => │ version │ 1.0.0 │
# => │ │ ╭──────┬─────────────────────────────────╮ │
# => │ repository │ │ type │ git │ │
# => │ │ │ url │ https://github.com/example/lark │ │
# => │ │ ╰──────┴─────────────────────────────────╯ │
# => │ publisher │ vscode │
# => │ categories │ [list 0 items] │
# => │ │ ╭───┬──────╮ │
# => │ keywords │ │ 0 │ lark │ │
# => │ │ ╰───┴──────╯ │
# => │ │ ╭────────┬────────╮ │
# => │ engines │ │ vscode │ ^1.0.0 │ │
# => │ │ ╰────────┴────────╯ │
# => │ │ ╭───┬─────────────────╮ │
# => │ activationEvents │ │ 0 │ onLanguage:lark │ │
# => │ │ ╰───┴─────────────────╯ │
# => │ main │ ./out/extension │
# => │ │ ╭───────────────┬───────────────────╮ │
# => │ contributes │ │ configuration │ {record 0 fields} │ │
# => │ │ │ grammars │ [list 0 items] │ │
# => │ │ │ languages │ [list 0 items] │ │
# => │ │ ╰───────────────┴───────────────────╯ │
# => │ │ ╭───────────────────┬────────────────────────────────────────╮ │
# => │ scripts │ │ compile │ tsc │ │
# => │ │ │ postinstall │ node ./node_modules/vscode/bin/install │ │
# => │ │ │ test │ npm run compile │ │
# => │ │ │ vscode:prepublish │ npm run compile │ │
# => │ │ │ watch │ tsc -w │ │
# => │ │ ╰───────────────────┴────────────────────────────────────────╯ │
# => │ │ ╭───────────────────────┬─────────╮ │
# => │ devDependencies │ │ @types/mocha │ ^2.2.42 │ │
# => │ │ │ @types/node │ ^7.0.43 │ │
# => │ │ │ tslint │ ^5.8.0 │ │
# => │ │ │ typescript │ ^2.6.1 │ │
# => │ │ │ vscode │ ^1.1.6 │ │
# => │ │ │ vscode-languageclient │ ^3.5.0 │ │
# => │ │ ╰───────────────────────┴─────────╯ │
# => ╰──────────────────┴────────────────────────────────────────────────────────────────╯In a similar way to ls, opening a file type that Nu understands will give us back something that is more than just text (or a stream of bytes). Here we open a "package.json" file from a JavaScript project. Nu can recognize the JSON text and parse it to a table of data.
If we wanted to check the version of the project we were looking at, we can use the get command.
open editors/vscode/package.json | get version
# => 1.0.0Nu currently supports the following formats for loading data directly into tables:
- csv
- eml
- ics
- ini
- json
- kdl
- md (Markdown)
- msgpack / msgpackz
- nuon
- ods
- plist
- SQLite databases
- ssv
- toml
- tsv
- url
- vcf
- xlsx
- xml
- yaml / yml
The eml, ics, ini, plist, and vcf formats are provided by the formats core plugin.
Did you know?
Under the hood open will look for a from ... subcommand in your scope which matches the extension of your file. You can thus simply extend the set of supported file types of open by creating your own from ... subcommand.
But what happens if you load a text file that isn't one of these? Let's try it:
open notes.txt
# => Remember to water the plants.
# => Call Mom on Sunday.We're shown the contents of the file.
Below the surface, what Nu sees in these text files is one large string. Next, we'll talk about how to work with these strings to get the data we need out of them.
NUON
Nushell Object Notation (NUON) aims to be for Nushell what JavaScript Object Notation (JSON) is for JavaScript. That is, NUON code is a valid Nushell code that describes some data structure. For example, this is a valid NUON (adapted from the documented configuration file, which you can also view with config nu --doc):
{
menus: [
# Configuration for default nushell menus
# Note the lack of source parameter
{
name: completion_menu
input_mode: cursor_prefix
marker: "| "
type: {
layout: columnar
columns: 4
col_width: 20 # Optional value. If missing all the screen width is used to calculate column width
col_padding: 2
}
style: {
text: green
selected_text: green_reverse
description_text: yellow
}
}
]
}You might notice it is quite similar to JSON, and you're right. NUON is a superset of JSON! That is, any JSON code is a valid NUON code, therefore a valid Nushell code. Compared to JSON, NUON is more "human-friendly". For example, comments are allowed and commas are not required.
One limitation of NUON currently is that it cannot represent all of the Nushell data types. Most notably, NUON does not allow the serialization of closures (to nuon --serialize writes them as strings, which are not turned back into closures when read).
Markdown, KDL, and YAML
Markdown
Markdown is one of the formats that open understands, so opening a .md file doesn't give you a string. Take this README.md, shown here as text with open --raw:
open --raw README.md
# => # My Project
# =>
# => A short description.
# =>
# => ## Install
# =>
# => - Download it
# => - Run itWithout --raw, from md turns it into a table with one row for each heading, paragraph, list item, and so on:
open README.md | select element content
# => ╭───┬─────────┬──────────────────────╮
# => │ # │ element │ content │
# => ├───┼─────────┼──────────────────────┤
# => │ 0 │ text │ My Project │
# => │ 1 │ text │ A short description. │
# => │ 2 │ text │ Install │
# => │ 3 │ text │ Download it │
# => │ 4 │ text │ Run it │
# => ╰───┴─────────┴──────────────────────╯The table also has a content_span column with each element's position in the file and, for elements that have them, attributes such as a heading's level. Use from md --verbose to get the full syntax tree.
KDL
KDL documents are read as a table of nodes. Each row has the node's name, its positional args, its named props, and its children:
'package name=nu version="0.116.0"' | from kdl | get 0.props
# => ╭─────────┬─────────╮
# => │ name │ nu │
# => │ version │ 0.116.0 │
# => ╰─────────┴─────────╯from kdl reads KDL version 2 by default. Use --spec 1 for KDL 1 documents.
YAML
from yaml follows YAML 1.2 by default, so a value like yes stays a string instead of becoming a boolean. Use --spec 1.1 to read a file with the older YAML 1.1 rules:
'enabled: yes' | from yaml
# => ╭─────────┬─────╮
# => │ enabled │ yes │
# => ╰─────────┴─────╯
'enabled: yes' | from yaml --spec 1.1
# => ╭─────────┬──────╮
# => │ enabled │ true │
# => ╰─────────┴──────╯Handling Strings
An important part of working with data coming from outside Nu is that it's not always in a format that Nu understands. Often this data is given to us as a string.
Let's imagine that we're given this data file:
open people.txt
# => Octavia | Butler | Writer
# => Bob | Ross | Painter
# => Antonio | Vivaldi | ComposerEach bit of data we want is separated by the pipe ('|') symbol, and each person is on a separate line. Nu doesn't have a pipe-delimited file format by default, so we'll have to parse this ourselves.
The first thing we want to do when bringing in the file is to work with it a line at a time:
open people.txt | lines
# => ╭───┬──────────────────────────────╮
# => │ 0 │ Octavia | Butler | Writer │
# => │ 1 │ Bob | Ross | Painter │
# => │ 2 │ Antonio | Vivaldi | Composer │
# => ╰───┴──────────────────────────────╯We can see that we're working with the lines because we're back into a list. Our next step is to see if we can split up the rows into something a little more useful. For that, we'll use the split command. split, as the name implies, gives us a way to split a delimited string. We will use split's column subcommand to split the contents across multiple columns. We tell it what the delimiter is, and it does the rest:
open people.txt | lines | split column "|"
# => ╭───┬──────────┬───────────┬───────────╮
# => │ # │ column0 │ column1 │ column2 │
# => ├───┼──────────┼───────────┼───────────┤
# => │ 0 │ Octavia │ Butler │ Writer │
# => │ 1 │ Bob │ Ross │ Painter │
# => │ 2 │ Antonio │ Vivaldi │ Composer │
# => ╰───┴──────────┴───────────┴───────────╯That almost looks correct. It looks like there's an extra space there. Let's trim that extra space:
open people.txt | lines | split column "|" | str trim
# => ╭───┬─────────┬─────────┬──────────╮
# => │ # │ column0 │ column1 │ column2 │
# => ├───┼─────────┼─────────┼──────────┤
# => │ 0 │ Octavia │ Butler │ Writer │
# => │ 1 │ Bob │ Ross │ Painter │
# => │ 2 │ Antonio │ Vivaldi │ Composer │
# => ╰───┴─────────┴─────────┴──────────╯Not bad. The split command gives us data we can use. It also goes ahead and gives us default column names:
open people.txt | lines | split column "|" | str trim | get column0
# => ╭───┬─────────╮
# => │ 0 │ Octavia │
# => │ 1 │ Bob │
# => │ 2 │ Antonio │
# => ╰───┴─────────╯We can also name our columns instead of using the default names:
open people.txt | lines | split column "|" first_name last_name job | str trim
# => ╭───┬────────────┬───────────┬──────────╮
# => │ # │ first_name │ last_name │ job │
# => ├───┼────────────┼───────────┼──────────┤
# => │ 0 │ Octavia │ Butler │ Writer │
# => │ 1 │ Bob │ Ross │ Painter │
# => │ 2 │ Antonio │ Vivaldi │ Composer │
# => ╰───┴────────────┴───────────┴──────────╯Now that our data is in a table, we can use all the commands we've used on tables before:
open people.txt | lines | split column "|" first_name last_name job | str trim | sort-by first_name
# => ╭───┬────────────┬───────────┬──────────╮
# => │ # │ first_name │ last_name │ job │
# => ├───┼────────────┼───────────┼──────────┤
# => │ 0 │ Antonio │ Vivaldi │ Composer │
# => │ 1 │ Bob │ Ross │ Painter │
# => │ 2 │ Octavia │ Butler │ Writer │
# => ╰───┴────────────┴───────────┴──────────╯There are other commands you can use to work with strings:
There is also a set of helper commands we can call if we know the data has a structure that Nu should be able to understand. For example, let's look at the first few lines of a Rust lock file:
open Cargo.lock | lines | first 6
# => ╭───┬───────────────────────────────────────────────────╮
# => │ 0 │ # This file is automatically @generated by Cargo. │
# => │ 1 │ # It is not intended for manual editing. │
# => │ 2 │ version = 4 │
# => │ 3 │ │
# => │ 4 │ [[package]] │
# => │ 5 │ name = "adhoc_derive" │
# => ╰───┴───────────────────────────────────────────────────╯The "Cargo.lock" file is actually a .toml file, but the file extension isn't .toml. That's okay, we can use the from command using the toml subcommand:
open Cargo.lock | from toml | get package | select name version | first 3
# => ╭───┬──────────────┬─────────╮
# => │ # │ name │ version │
# => ├───┼──────────────┼─────────┤
# => │ 0 │ adhoc_derive │ 0.1.2 │
# => │ 1 │ aho-corasick │ 1.1.5 │
# => │ 2 │ demo │ 0.1.0 │
# => ╰───┴──────────────┴─────────╯The from command can be used for each of the structured data text formats that Nu can open and understand by passing it the supported format as a subcommand.
Opening in raw mode
While it's helpful to be able to open a file and immediately work with a table of its data, this is not always what you want to do. To get to the underlying text, the open command can take an optional --raw flag:
open Cargo.toml --raw
# => [package]
# => name = "demo"
# => version = "0.1.0"
# => edition = "2024"
# =>
# => [dependencies]
# => adhoc_derive = "0.1.2"SQLite
SQLite databases are automatically detected by open, no matter what their file extension is. To try this out, create a small database with into sqlite:
[[name age]; [Alice 30] [Bob 25]] | into sqlite foo.db --table-name some_tableYou can open a whole database:
open foo.db
# => ╭────────────┬─────────────────────╮
# => │ │ ╭───┬───────┬─────╮ │
# => │ some_table │ │ # │ name │ age │ │
# => │ │ ├───┼───────┼─────┤ │
# => │ │ │ 0 │ Alice │ 30 │ │
# => │ │ │ 1 │ Bob │ 25 │ │
# => │ │ ╰───┴───────┴─────╯ │
# => ╰────────────┴─────────────────────╯Or get a specific table:
open foo.db | get some_table
# => ╭───┬───────┬─────╮
# => │ # │ name │ age │
# => ├───┼───────┼─────┤
# => │ 0 │ Alice │ 30 │
# => │ 1 │ Bob │ 25 │
# => ╰───┴───────┴─────╯Or run any SQL query you like:
open foo.db | query db "select * from some_table where age > 26"
# => ╭───┬───────┬─────╮
# => │ # │ name │ age │
# => ├───┼───────┼─────┤
# => │ 0 │ Alice │ 30 │
# => ╰───┴───────┴─────╯Fetching URLs
In addition to loading files from your filesystem, you can also load URLs by using the http get command. This will fetch the contents of the URL from the internet and return it:
http get https://blog.rust-lang.org/feed.xml
# => ╭────────────┬──────────────────╮
# => │ tag │ feed │
# => │ attributes │ {record 1 field} │
# => │ content │ [table 18 rows] │
# => ╰────────────┴──────────────────╯