Parquet Source

This page lists the data types supported by Data Factory Studio for Parquet files.

This page discusses:

See Also
Indexing Tabular Data

Property Type Support

Binary

Supports a Parquet Byte array, a Parquet string containing data encoded in Base64, or a null.

Parquet Value Resulting Value (in the Index Unit)
"YWJjZGVmZ2gxMjPDqcOgw6ckJsKnwr7jg4jjg4jjg60=" Binary.decodeBase64("YWJjZGVmZ2gxMjPDqcOgw6ckJsKnwr7jg4jjg4jjg60=")
null NULL

Boolean

Supports the standard Parquet booleans values, a json null value, or eventually a json string.

Parquet Value Resulting Value (in the Index Unit)
true true
false false
null NULL
"true" true
"True" true
"TRUE" true
"TruE" true
"false" false
"False" false
"FALSE" false
"FasLE" false
"abcdef" false
"1" false

Date

Supports a Parquet string containing a date or a datetime with ISO 8601 format, a null, or a Parquet Date or Timestamp value.

Note: Parquet Timestamp values do not include timezones, so Data Factory Studio assumes it is UTC timezone.
Parquet Value Resulting Value (in the Index Unit)
ISO 8601
"1970-01-25" (yyyy-MM-dd) Not supported
"1234-05-06T07:08:09" (yyyy-MM-dd'T'HH:mm:ss) Not supported
"2222-02-22T22:22:22Z" (yyyy-MM-dd'T'HH:mm:ssXXX) DateTime.of(2222, 2, 22, 22, 22, 22, 0, ZoneOffset.utc())
"1970-01-01T01:01:01+02:00" (yyyy-MM-dd'T'HH:mm:ssXXX) DateTime.of(1970, 1, 1, 1, 1, 1, 0, ZoneOffset.ofHours(2))
Other formats
"30/05/1978 02:34:56" (dd/MM/yyyy HH:mm:ss) Not supported
"2/4/2010 2:39:28 PM" (M/d/yyyy h:mm:ss tt) Not supported
"6:05:12 a.m." (h:mm:ss tt) Not supported
"23:31:18.345" (HH:mm:ss.s) Not supported
"Tuesday, August 20, 2002" (dddd, MMMM dd, yyyy) Not supported
"Wed, Aug 31 1994" (ddd, MMM dd yyyy) Not supported
"03281999030456" (MMddyyyyHHmmss) Not supported
"Im a random string" Not supported
null NULL

LocalDateTime

Supports a Parquet string containing a date or a datetime with ISO 8601 format, a null, or a Parquet Date, Time or Timestamp value.

Notes:
  • Parquet Timestamp values do not include timezones, so Data Factory Studio assumes it is UTC timezone.
  • Semantic Graph Index does not define a type equivalent to the Parquet Time data type. Therefore, Data Factory Studio maps these to LocalDatetime by filling the date part with epoch date 1st January 1970.
Parquet Value Resulting Value (in the Index Unit)
ISO 8601
"1970-01-25" (yyyy-MM-dd) Not supported
"1234-05-06T07:08:09" (yyyy-MM-dd'T'HH:mm:ss) LocalDateTime.of(1245, 5, 6, 7, 8, 9)
"2222-02-22T22:22:22Z" (yyyy-MM-dd'T'HH:mm:ssXXX) LocalDateTime.of(2222, 2, 22, 22, 22, 22)
"1970-01-01T01:01:01+02:00" (yyyy-MM-dd'T'HH:mm:ssXXX) LocalDateTime.of(1970, 1, 1, 1, 1, 1)
Other formats
"30/05/1978 02:34:56" (dd/MM/yyyy HH:mm:ss) Not supported
"2/4/2010 2:39:28 PM" (M/d/yyyy h:mm:ss tt) Not supported
"6:05:12 a.m." (h:mm:ss tt) Not supported
"23:31:18.345" (HH:mm:ss.s) Not supported
"Tuesday, August 20, 2002" (dddd, MMMM dd, yyyy) Not supported
"Wed, Aug 31 1994" (ddd, MMM dd yyyy) Not supported
"03281999030456" (MMddyyyyHHmmss) Not supported
"Im a random string" Not supported
null NULL

DateTime

Supports a Parquet string containing a date or a datetime with ISO 8601 format, a null, or a Parquet Date, Time or Timestamp value.

Notes:
  • Parquet Timestamp values do not include timezones, so Data Factory Studio assumes it is UTC timezone.
  • Semantic Graph Index does not define a type equivalent to the Parquet Time data type. Therefore, Data Factory Studio maps these to Datetime by filling the date part with epoch date 1st January 1970.
Parquet Value Resulting Value (in the Index Unit)
ISO 8601
"1970-01-25" (yyyy-MM-dd) Not supported
"1234-05-06T07:08:09" (yyyy-MM-dd'T'HH:mm:ss) Not supported
"2222-02-22T22:22:22Z" (yyyy-MM-dd'T'HH:mm:ssXXX) DateTime.of(2222, 2, 22, 22, 22, 22, 0, ZoneOffset.utc())
"1970-01-01T01:01:01+02:00" (yyyy-MM-dd'T'HH:mm:ssXXX) DateTime.of(1970, 1, 1, 1, 1, 1, 0, ZoneOffset.ofHours(2))
Other formats
"30/05/1978 02:34:56" (dd/MM/yyyy HH:mm:ss) Not supported
"2/4/2010 2:39:28 PM" (M/d/yyyy h:mm:ss tt) Not supported
"6:05:12 a.m." (h:mm:ss tt) Not supported
"23:31:18.345" (HH:mm:ss.s) Not supported
"Tuesday, August 20, 2002" (dddd, MMMM dd, yyyy) Not supported
"Wed, Aug 31 1994" (ddd, MMM dd yyyy) Not supported
"03281999030456" (MMddyyyyHHmmss) Not supported
"Im a random string" Not supported
null NULL

Float and Decimal

Supports Parquet Int32, Int64, Float, Double, Decimal, a null, or a Parquet string that can be parsed into a number.

Note: As Int96 is deprecated in Parquet standard, Data Factory Studio does not support it.
Parquet Value Resulting Value (in the Index Unit)
INT32 42 42.
FLOAT 42.3 42.3
INT96 123456789 Not supported
null NULL
"42" 42.
"42L" Not supported
"42.3" 42.3
"42." 42.
"42.3f" 42.3
"42.3d" 42.3
"4.23e3" 4230.
"0x2A" Not supported
"0x2A.5p-0" 42.3125
"" Not supported
"foobar" Not supported

Integer

Supports Parquet Int32, Int64, a null, or a Parquet string that can be parsed into an integer.

Note: As Int96 is deprecated in Parquet standard, Data Factory Studio does not support it.
Parquet Value Resulting Value (in the Index Unit)
Int32 42 42.
Float 42.3 Not supported
Int96 123456789 Not supported
null NULL
"42" 42
"42L" Not supported
"42.3" Not supported
"42.3f" Not supported
"42.3d" Not supported
"4.23e3" Not supported
"0x2A" Not supported
"0x2A.5p-0" Not supported
"" Not supported
"foobar" Not supported

Geo

Supports a Parquet string containing WKT data, or a null.

Parquet Value Resulting Value (in the Index Unit)
"POINT(6 10)" Geo.ofWkt("POINT(6 10)")
"LINESTRING(3 4,10 50,20 25)" Geo.ofWkt("LINESTRING(3 4,10 50,20 25)")
"POLYGON((1 1,5 1,5 5,1 5,1 1))" Geo.ofWkt("POLYGON((1 1,5 1,5 5,1 5,1 1))")
"POINT EMPTY" Geo.ofWkt("POINT EMPTY")
"foobar" Not supported
null NULL

Reference

Supports a Parquet string containing the uri of a referenced item, or a null.

Parquet Value Resulting Value (in the Index Unit)
"FDJSKFDGSFLGKFL" (item uri) A reference to that item
null NULL

Text, String, and FullTextSearch

Supports a Parquet string, a parquet Enum, or a null.

Parquet Value Resulting Value (in the Index Unit)
"Any value" "Any value"
"" ""
null NULL

Property Data Structures

Singleton

Mono-valuated value.

List

An ordered collection of items (which may contain duplicates).

Supports a Parquet list or a null.

Warning: As the representation of lists and maps have recently changed in Parquet standard, there are currently some known issues on these types depending on which library was used to write the Parquet file.

Collection

An unordered collection of items (which may contain duplicates). Collection parsing is identical to list parsing.

Map

An unordered key-value mapping.

Supports a Parquet map or a null.

Warning: As the representation of lists and maps have recently changed in Parquet standard, there are currently some known issues on these types depending on which library was used to write the Parquet file.