Column Selectors
The Selecting rows and columns section of the Dataframes chapter shows a selector in polars select and in polars as. This page covers all the polars selector commands. A selector describes a set of columns by name, by position, or by data type. It holds no data. Polars works out which columns it matches when a query runs, using the schema of the dataframe it runs against.
polars selector on its own prints its help and the list of subcommands.
The examples on this page use this dataframe:
let products = [[id name category price qty in_stock]; [1 apple fruit 0.5 10 true] [2 kale greens 2.25 0 false]] | polars into-df
$products | polars schema
# => ╭──────────┬──────╮
# => │ id │ i64 │
# => │ name │ str │
# => │ category │ str │
# => │ price │ f64 │
# => │ qty │ i64 │
# => │ in_stock │ bool │
# => ╰──────────┴──────╯Using a selector
A selector is a value of type polars_selector. Printing one shows the Polars selector it stands for, which helps when a selector matches something you didn't expect:
polars selector by-name qty id
# => Some(ByName { names: ["qty", "id"], strict: false })The selector commands ignore their input, so $products | polars selector integer returns the same selector as polars selector integer and doesn't select anything from $products. Pass the selector to a command that takes columns instead. These accept selectors:
polars select,polars with-column,polars group-by,polars agg,polars sort-by, andpolars as- the list of columns given to
polars concat-str - the
polars mathcommands, described in Math Expressions polars str-splitandpolars str-strip-chars, described in String Expressions
Dates and Times in Dataframes lists which date commands take a selector. With other commands, such as polars uppercase or polars sum, pick the columns with polars col, which takes several names, a ^...$ regular expression, or --type:
$products | polars select (polars col --type str | polars uppercase) | polars collect
# => ╭───┬───────┬──────────╮
# => │ # │ name │ category │
# => ├───┼───────┼──────────┤
# => │ 0 │ APPLE │ FRUIT │
# => │ 1 │ KALE │ GREENS │
# => ╰───┴───────┴──────────╯Polars resolves a selector when the query runs, so a selector that can't be resolved, such as --strict with a missing column, builds without an error. It fails when you collect the result, or when polars columns or polars schema asks the lazy dataframe for its columns.
The order of the selected columns depends on the selector. polars selector by-name and polars selector by-index return columns in the order you list them. The other selectors return them in the dataframe's order:
$products | polars select (polars selector by-name qty id) | polars columns
# => ╭───┬─────╮
# => │ 0 │ qty │
# => │ 1 │ id │
# => ╰───┴─────╯
$products | polars select (polars selector ends-with y e) | polars columns
# => ╭───┬──────────╮
# => │ 0 │ name │
# => │ 1 │ category │
# => │ 2 │ price │
# => │ 3 │ qty │
# => ╰───┴──────────╯polars select takes several selectors and places their columns one after another. If two of them match the same column, the column appears twice and collecting fails with projections contained duplicate output name.
Selecting by name
polars selector by-name picks the columns you name and skips names that don't exist. With --strict, a missing name makes collecting fail with an error such as "nope" not found.
polars selector exclude picks every column except the ones you name. It ignores names that don't exist. Both commands need at least one name and fail with Missing column names without one.
Four selectors match part of a name:
polars selector starts-withmatches names that start with the given string.polars selector ends-withmatches names that end with one of the given strings.polars selector containsmatches names that contain one of them anywhere.polars selector matchestakes one regular expression in the syntax of Rust'sregexcrate.
The first three treat their strings literally, so polars selector contains . matches only names with a dot in them. Given no strings, they match every column. polars selector ends-with and polars selector contains take any number of strings. For several prefixes, use polars selector matches with a pattern such as '^(p|n)'.
The pattern given to polars selector matches isn't anchored, so polars selector matches e picks every name with an e in it. Add ^ and $ to match the whole name, and (?i) to ignore case. An invalid pattern fails when you collect, with an error such as invalid regex in selector '('.
Three selectors look at the kind of characters in a name. polars selector alpha matches names made only of letters, polars selector alphanumeric names made only of letters and digits, and polars selector digit names made only of digits.
A letter can be any Unicode letter and a digit any Unicode decimal digit, so größe counts as alphabetic and ٣ as a digit. --ascii-only (-a) limits them to a to z, A to Z, and 0 to 9. alpha and alphanumeric also take --ignore-spaces (-s), which allows spaces in the name. An underscore or other punctuation never matches, so q1_2024 isn't alphanumeric:
let sales = [[region "unit price" q1_2024 "2024"]; [north 9.5 3 7]] | polars into-df
$sales | polars select (polars selector alpha --ignore-spaces) | polars columns
# => ╭───┬────────────╮
# => │ 0 │ region │
# => │ 1 │ unit price │
# => ╰───┴────────────╯
$sales | polars select (polars selector alphanumeric) | polars columns
# => ╭───┬────────╮
# => │ 0 │ region │
# => │ 1 │ 2024 │
# => ╰───┴────────╯Selecting by position
polars selector by-index takes column positions, counting from 0. Negative positions count back from the end, so -1 is the last column. A position past either end makes collecting fail with an error such as cannot get the 5-th column when schema has 3 columns, unless you pass --not-strict, which skips it. Given no positions, the selector matches nothing.
polars selector first and polars selector last take a number of columns, which defaults to 1. A number of 0 or less also picks one column, and a number larger than the column count picks every column without an error.
$products | polars select (polars selector by-index 0 -1) | polars columns
# => ╭───┬──────────╮
# => │ 0 │ id │
# => │ 1 │ in_stock │
# => ╰───┴──────────╯
$products | polars select (polars selector last 2) | polars columns
# => ╭───┬──────────╮
# => │ 0 │ qty │
# => │ 1 │ in_stock │
# => ╰───┴──────────╯Selecting by data type
These selectors pick columns by their data type, as polars schema shows it:
| Selector | Matches |
|---|---|
integer | signed and unsigned integers |
signed-integer | i8, i16, i32, and i64 |
unsigned-integer | u8, u16, u32, and u64 |
float | f32 and f64 |
decimal | decimals, such as a column cast to decimal<10,2> |
numeric | integers, floats, and decimals |
string | str, plus categorical columns with --include-categorical (-c) |
boolean | bool |
binary | binary, which Nushell binary values become |
categorical | categorical (cat) columns, such as the output of polars qcut |
enum | enum columns, such as the output of polars cut |
date | date |
datetime | datetimes of any time unit and time zone, unless you narrow it with flags |
duration | durations of any time unit, unless you narrow it with --time-unit |
temporal | dates, datetimes, durations, and times |
list | lists of any element type |
array | fixed-width arrays, or only arrays of one width with --width |
struct | structs, such as nested objects read from a JSON or NDJSON file |
nested | lists, arrays, and structs |
object | object columns, which hold Nushell values with no column type, such as records without a schema |
Each name in the first column is a subcommand of polars selector. Binning values shows polars cut and polars qcut. polars into-df turns Nushell lists into list columns, never into arrays, so you meet array columns only in data from elsewhere.
$products | polars select (polars selector numeric) | polars columns
# => ╭───┬───────╮
# => │ 0 │ id │
# => │ 1 │ price │
# => │ 2 │ qty │
# => ╰───┴───────╯polars selector datetime takes --time-unit and --time-zone, and each takes a list. A time unit is ns, us, or ms. Write us even though polars schema prints μs, because μs is an error. A time zone is a name such as UTC or Europe/Berlin. The list [unset] picks datetimes that have no time zone, and unset has to be the only entry, because [UTC unset] fails with Invalid timezone 'unset'. With both flags, a column has to match both. polars selector duration takes --time-unit the same way.
let events = [[utc_ns berlin_ms naive_us]; [2024-01-02 2024-01-02 2024-01-02]] | polars into-df | polars cast "datetime<ms, Europe/Berlin>" berlin_ms | polars cast "datetime<us, *>" naive_us
$events | polars schema
# => ╭───────────┬─────────────────────────────╮
# => │ utc_ns │ datetime<ns, UTC> │
# => │ berlin_ms │ datetime<ms, Europe/Berlin> │
# => │ naive_us │ datetime<μs> │
# => ╰───────────┴─────────────────────────────╯
$events | polars select (polars selector datetime --time-zone [unset]) | polars columns
# => ╭───┬──────────╮
# => │ 0 │ naive_us │
# => ╰───┴──────────╯
$events | polars select (polars selector datetime --time-unit [ms us] --time-zone [Europe/Berlin]) | polars columns
# => ╭───┬───────────╮
# => │ 0 │ berlin_ms │
# => ╰───┴───────────╯polars selector by-dtype takes one or more type names, written the way polars schema prints them, and picks columns whose type is exactly one of them. "datetime<ms, UTC>" doesn't match a datetime<ns, UTC> column, and a list type needs its element type, as in "list<str>". A * in place of the time zone means no time zone, so polars selector by-dtype "datetime<us, *>" picks naive_us and nothing else. An unknown name fails with Unknown type, and with no names the command fails with Missing data types. To match a type family, such as datetimes of any unit, use the selectors in the table.
Combining selectors
polars selector all matches every column, and polars selector empty matches none. polars selector not takes a selector as input and matches every column the input doesn't:
$products | polars select (polars selector numeric | polars selector not) | polars columns
# => ╭───┬──────────╮
# => │ 0 │ name │
# => │ 1 │ category │
# => │ 2 │ in_stock │
# => ╰───┴──────────╯polars selector not accepts only a selector, so polars col qty | polars selector not fails with Command does not support polars_expression input. To leave out columns by name, use polars selector exclude. There are no commands for the union or intersection of two selectors.
An arithmetic or comparison operator turns a selector into an expression, which applies to each matched column and keeps the column's name:
$products | polars with-column ((polars selector numeric) * 10) | polars collect
# => ╭───┬────┬───────┬──────────┬───────┬─────┬──────────╮
# => │ # │ id │ name │ category │ price │ qty │ in_stock │
# => ├───┼────┼───────┼──────────┼───────┼─────┼──────────┤
# => │ 0 │ 10 │ apple │ fruit │ 5.00 │ 100 │ true │
# => │ 1 │ 20 │ kale │ greens │ 22.50 │ 0 │ false │
# => ╰───┴────┴───────┴──────────┴───────┴─────┴──────────╯For the same reason, and and or work on the values in the columns, not on the sets of columns. (polars selector integer) or (polars selector float) builds boolean columns and doesn't select the integer and float columns. To get both kinds, pass both selectors to polars select, or use polars selector numeric.