Friday, February 26, 2021

Why applications work better on Autonomous Database than on PostgreSQL on AWS

Oracle Autonomous Database, Oracle Database Tutorial and Material, Oracle Database Exam Prep, Oracle Database Prep, Database Preparation, Database Guide

In today’s customer-centric world, organizations find it challenging to grow their user base while addressing the speed, agility, and personalization requirements of their customers—but that’s exactly what LogFire was able to do when the company moved to Oracle Autonomous Database, a next generation cloud platform with a fully automated database. In just one year, LogFire increased its order processing speeds by 55% while growing its customer base by 2x. LogFire also reduced its TCO through Autonomous Database auto scaling technology and went from ten external contractors managing the database to zero—with the database automating all manual tasks. Today, LogFire touches nearly 5 billion packages a year, and is trusted by hundreds of customers globally. With Autonomous Database, LogFire was also able to quickly build more features and customizations to expand into 10 new industries as well.

In this blog, we will discuss how LogFire (now branded as Oracle Warehouse Management Cloud) migrated its cloud native, mission-critical SaaS application to Oracle Cloud Infrastructure (OCI) and moved all its AWS PostgreSQL databases to Oracle Autonomous Transaction Processing (which also runs on OCI) for better performance, higher elasticity, improved security and reduced TCO.

Oracle Warehouse Management Cloud, Born from LogFire

Oracle Warehouse Management Cloud is a leading cloud-based inventory and warehouse management service that helps businesses automate various steps of the supply chain process. Oracle acquired LogFire in 2016 as the foundation for Oracle WMS. LogFire, a pioneer in bringing cloud to supply chain management, was founded by Diego Pantoja-Navajas in 2007. In my conversation with Diego, he said: “With the objective of transforming supply chains by helping enterprises adopt cloud technologies, we developed a standalone SaaS application that integrated various systems and data to remove silos and make supply chains more efficient. We always knew we were designing a heart and lung, mission-critical application. It had to be bullet proof because so many different types of businesses would depend on it.”

Issues with Amazon Relational Database Service (RDS)

LogFire’s application employed modern cloud native development practices with production databases running on PostgreSQL on Rackspace. Arun Murugan, VP WMS Product Development, said that they moved their test databases to Amazon RDS for PostgreSQL to reduce costs but continued running production on Rackspace and outsourced its management, since LogFire did not have an in-house DBA team. “Although RDS simplified scaling, there were performance and reliability concerns that made it unfit for our production databases. These concerns included no performance guarantees, manual tuning of a large number of database parameters to maintain performance levels, no automated bi-directional replication of database, security gaps, downtimes for patching and maintenance, and more,” said Arun.

Oracle Autonomous Database, Oracle Database Tutorial and Material, Oracle Database Exam Prep, Oracle Database Prep, Database Preparation, Database Guide

LogFire’s architecture before moving to Oracle Cloud Infrastructure

Why LogFire Moved their Applications and Databases to Oracle Cloud Infrastructure


Just like any other industry, supply chain is undergoing a huge transformation. The great toilet paper panic of 2020 amid the COVID-19 pandemic highlighted another shortfall in the global supply chain. Exposure to social media, the increasing importance of customer experience, sustainability, ethical vendor behavior, and the circular economy have all had a great impact on transforming supply chain from a back office function to a more strategic differentiator. As a result, supply chain’s significance was amplified across industries and LogFire saw a surge in demand that put a lot of pressure on their prior architecture. As LogFire focused on innovation and modernizing its supply chain for its customers:

◉ They weren’t able to add new features at the rate that their customers were demanding

◉ They were spending an increasing amount of time manually managing their databases to ensure 24/7 availability for their customers.

◉ Most importantly, the production PostgreSQL databases running on Rackspace weren’t able to scale up and down as quickly to respond to the seasonal peaks in demand as well as the yearly increase in shipping volumes of their retail customers. Although their application could scale dynamically, scaling and updating the databases required downtime, making the databases unavailable and slowing down the overall process. They had to plan in advance and purchase additional servers on Rackspace before every seasonal peak and then manually scale down at the end of the season. And on AWS scaling also required downtime.

These fundamental challenges were holding back their growth, so LogFire started evaluating other options that could support its business requirements. In the summer of 2019 after weighing all the benefits, they decided to move their application and databases to OCI. After a thorough assessment of various options of running PostgreSQL databases on OCI, or Oracle Autonomous Database, they migrated all their 700 databases from PostgreSQL to Autonomous Transaction Processing, an Oracle Autonomous Database optimized for OLTP.

Oracle Autonomous Database, Oracle Database Tutorial and Material, Oracle Database Exam Prep, Oracle Database Prep, Database Preparation, Database Guide

Oracle Warehouse Management Cloud architecture on Oracle Cloud Infrastructure

Seamless Migration Process


LogFire’s application and database migration to OCI and Autonomous Database took just over 7 months. The entire migration would have taken under 3 months if not for the delays to avoid any disruptions for its retail customers during their seasonal peaks. By February of 2020 they had moved their entire application stack and all of their customers to OCI. The following factors contributed to a smooth migration for all 700 databases.

1. LogFire built its application using a modern cloud native REST services-based architecture. This made migrating to OCI a simple lift and shift of the application code to Oracle’s cloud native services without much re-architecting or rewriting of the code. They relied on API calls to connect to a variety of non-standard systems for automating picking, packing, sorting, and shipping of packages. Moving their fleet of 700 PostgreSQL databases was just like replacing the underlying database platform without having to worry about complex integrations.

2. LogFire’s application architecture includes the Django object relational mapper (ORM) that provides a database isolation layer, making the queries database-agnostic. This meant that 75% of the SQL queries that were generated by the ORM could be brought in “as is” to Oracle. Only 25% of the queries that were hand-coded required some degree of tuning for Oracle Autonomous Transaction Processing.

3. Since PostgreSQL and Autonomous Transaction Processing are both SQL-based, the LogFire developers and users had a very short learning curve as most of the functions are automated in ATP.

How the Move to ATP from PostgresSQL Propelled LogFire’s Growth


55% increase in order processing speeds: Warehouse management systems are real-time transaction systems where performance and uptime can directly impact order delivery. Several systems, robots, and machines constantly make API calls to the database for completing a single order. As order volumes increase, database outages or slower response times impact the company’s revenue and ultimately its reputation. Only Autonomous Transaction Processing was able to support these high performance requirements. Additionally, migrating to OCI improved LogFire’s customers’ order processing speeds by 55% while packages shipped also increased year-over-year by 45%.

50% increase in warehouse users: Diego pointed out that after moving to OCI, LogFire was able to expand from five industries to fifteen, doubling its customer base and growing warehouse users by 50% in just one year. They were able to quickly launch new features and customizations for these new industries as their employees had access to all data and could focus on development—rather than managing the database and the underlying infrastructure. Autonomous Transaction Processing facilitated this growth by providing unlimited access to compute and storage resources by enabling auto-scaling at the click of a button—even during the seasonal peaks when LogFire’s customers doubled their users. In fact, at the start of the pandemic the shipping volumes of most of their customers went over their seasonal peaks. LogFire was able to meet that unexpected surge in demand as well, easily and without any disruptions. The Autonomous Database’s auto-scaling feature can automatically scale up or scale down the number of CPUs depending on workload demands, resulting in cost savings through true pay-per-use.

TCO Reduction: LogFire reduced its TCO by moving from a remote database management service with 10 contractors constantly monitoring all their databases to a fully automated environment. All of its  tasks were automated by Autonomous Transaction Processing providing error-free 24/7 management, and eliminating ongoing maintenance costs for patching, securing, and backing up the databases.

Zero Downtime: PostgreSQL databases running on AWS or Rackspace required planned downtime for incremental patching, scaling, and maintenance. Autonomous Transaction Processing scales dynamically while the transactions are in flight, applies patches automatically with no downtime, and requires no planned maintenance. It automatically creates and maintains multitenant environments, manages clusters, and implements Oracle Maximum Availability Architecture (MAA). The one-click-enabled Autonomous Data Guard (Auto-DG) feature automatically creates a standby replica of the running database. This enables the database layer to automatically switch over to the standby database in the rare case of any disruption/outage. Rather than waiting for the primary database to recover, application transactions can seamlessly continue to work on the promoted standby database without any performance impact. This feature, called Transparent Application Continuity, further enhances the availability of the database that is already running on Oracle Real Application Clusters (RAC). These capabilities improved LogFire’s uptime SLA, ensuring zero downtime.

Oracle Autonomous Database, Oracle Database Tutorial and Material, Oracle Database Exam Prep, Oracle Database Prep, Database Preparation, Database Guide

Enhanced Security: LogFire hardened its security profile after migrating off PostgreSQL. With AWS RDS for PostgreSQL their data was encrypted only during transmission. Autonomous Transaction Processing provides Transparent Data Encryption (TDE) to encrypt data both at rest and in motion. It uses machine learning to automate monitoring for security threats. It also provides additional granular security to perform assessments, mask sensitive data, and audit activities to protect customer data. Security patches are also automatically applied.

Machine Learning-based automation: LogFire was also able to differentiate and provide unique value to its customers by leveraging the in-database machine learning algorithms available with Autonomous Transaction Processing. They deployed more than 150 different AI/ML algorithms, without any data scientists on their team. Following is an example of a few AI/ML algorithms that they implemented:

◉ Predicting peak hours in distribution centers
◉ Calculating average picking times and expected task time
◉ Replenishment (automatically initiating efficient replenishment tasks)
◉ Calculating expected profit from each order fulfilled or average revenue generated per order
◉ Predicting future target values
◉ Creating predictive pick orders based on historical data
◉ Automatically assigning order fulfillment priority
◉ Clustering employees who work well together to optimize efficiency
◉ Warehouse slotting
◉ Spotting worker efficiencies

Road Ahead

Since Autonomous Transaction Processing is comprised of Oracle Autonomous Database running on Oracle Exadata on OCI and uses Oracle Real Application Clusters (RAC) plus multitenant capabilities, its availability and performance are orders of magnitude higher than PostgreSQL on AWS. And since it automates all the management, it is much simpler to use than PostgreSQL with a significant reduction in TCO. Its broad set of built-in converged database capabilities that enable machine-learning analysis, reporting from day one, low-code application development, elastic and automated scaling, plus auto-indexing make it easy for developers to build and deploy new features. Autonomous Transaction Processing is available in OCI and in customers’ data centers on Oracle Exadata Cloud@Customer or Dedicated Region Cloud@Customer. Oracle Warehouse Management Cloud is available in OCI and Dedicated Region Cloud@Customer.

Source: oracle.com

Thursday, February 25, 2021

Getting started with Oracle Machine Learning for Python

Oracle Machine Learning, Python, Oracle Database Exam Prep, Oracle Database Certification, Oracle Database Preparation, Database Prep, Oracle Database Exam Prep

As noted in Introducing Oracle Machine Learning for Python, OML4Py is included with Oracle Autonomous Database, making the open source Python scripting language and environment ready for the enterprise and big data. 

To get started with OML4Py, log into your Oracle Machine Learning Notebooks account and create a new notebook. If you don't have one yet, you can create an Autonomous Database account using your Oracle Always Free Services and follow this OML Notebooks tutorial.

Load the OML package

In the initial paragraph, specify %python as your interpreter. At this point, you can invoke Python code. However, to use OML4Py, import the package oml. Click the "run this paragraph" button. You can optionally invoke oml.isconnected to verify your connection, which should return true.

%python

import oml

oml.isconnected()

Load a Pandas DataFrame to the database

There are several way to load data into Oracle Autonomous Database. In this first example, we create a table using the sklearn iris data set. We combine the target and predictors into a single Pandas DataFrame and load this DataFrame object into an Oracle Autonomous Database  table using the create function.

from sklearn.datasets import load_iris

import pandas as pd

iris = load_iris()

x = pd.DataFrame(iris.data, 

                 columns = ["SEPAL_LENGTH", "SEPAL_WIDTH", 

                            "PETAL_LENGTH", "PETAL_WIDTH"])

y = pd.DataFrame(list(map(lambda x: {0:'setosa', 1: 'versicolor', 

                                     2:'virginica'}[x], iris.target)), 

                 columns = ['Species'])

iris_df = pd.concat([x,y], axis=1)

IRIS = oml.create(iris_df, table="IRIS")

print("Shape:",IRIS.shape)

print("Columns:",IRIS.columns)

IRIS.head(4)

The script above produces the following output. Note that we access shape and columns properties on the proxy object, just as we would with a Pandas DataFrame. Similarly, we invoke the overloaded head function on the IRIS proxy object.

Shape: (150, 5)

Columns: ['SEPAL_LENGTH', 'SEPAL_WIDTH', 'PETAL_LENGTH', 'PETAL_WIDTH', 'Species']

Out[6]:

   SEPAL_LENGTH  SEPAL_WIDTH  PETAL_LENGTH  PETAL_WIDTH Species

0           5.1          3.5           1.4          0.2  setosa

1           4.9          3.0           1.4          0.2  setosa

2           4.7          3.2           1.3          0.2  setosa

3           4.6          3.1           1.5          0.2  setosa

This table is also readily available in the user schema under the name IRIS, just as any other database table.

Using overloaded functions

Using the numeric columns, we compute the correlation matrix on the in-database table IRIS using the overloaded corr function. Here, we see that petal length and petal width are highly correlated.

IRIS.corr()

With the output:

              SEPAL_LENGTH  SEPAL_WIDTH  PETAL_LENGTH  PETAL_WIDTH

SEPAL_LENGTH      1.000000    -0.109369      0.871754     0.817954

SEPAL_WIDTH      -0.109369     1.000000     -0.420516    -0.356544

PETAL_LENGTH      0.871754    -0.420516      1.000000     0.962757

PETAL_WIDTH       0.817954    -0.356544      0.962757     1.000000

OML4Py overloads graphics functions as well. Here, we use boxplot to show the distribution of the numeric columns. In such overloaded functions, the statistical computations take place in the database - avoiding data movement and leveraging Autonomous Database as a high performance compute engine - returning only the summary statistics needed to produce the plot.

import matplotlib.pyplot as plt

plt.style.use('seaborn')

plt.figure(figsize=[10,5]))

oml.graphics.boxplot(IRIS[:, :4], notch=True, showmeans = True,

                     labels=IRIS.columns[:4])

plt.title('Distribution of IRIS Attributes')

plt.ylabel('cm');

Oracle Machine Learning, Python, Oracle Database Exam Prep, Oracle Database Certification, Oracle Database Preparation, Database Prep, Oracle Database Exam Prep

In-database attribute importance


Let's rank the relative importance of each variable (a.k.a., attribute or predictor) to predict the target 'Species' from the IRIS table.

We define the ai (attribute importance) object, compute the result, and show the attribute importance ranking.

In the result, notice that petal width is most predictive of the target species. The importance value produced by this algorithm provides a relative ranking to be used to distinguish importance among variables.

from oml import ai

# here we use sync to get handle to existing table
IRIS = oml.sync(table = "IRIS")
IRIS_x = IRIS.drop('Species')
IRIS_y = IRIS['Species']

ai_obj = ai()  # Create attribute importance object
ai_obj = ai_obj.fit(IRIS_x, IRIS_y)
ai_obj 

With the output:

Algorithm Name: Attribute Importance

Mining Function: ATTRIBUTE_IMPORTANCE

Settings: 
                   setting name            setting value
0                     ALGO_NAME              ALGO_AI_MDL
1                  ODMS_DETAILS              ODMS_ENABLE
2  ODMS_MISSING_VALUE_TREATMENT  ODMS_MISSING_VALUE_AUTO
3                 ODMS_SAMPLING    ODMS_SAMPLING_DISABLE
4                     PREP_AUTO                       ON

Global Statistics: 
  attribute name  attribute value
0       NUM_ROWS              150

Attributes: 
PETAL_LENGTH
PETAL_WIDTH
SEPAL_LENGTH
SEPAL_WIDTH

Partition: NO

Importance: 

       variable  importance  rank
0   PETAL_WIDTH    1.050935     1
1  PETAL_LENGTH    1.030633     2
2  SEPAL_LENGTH    0.454824     3
3   SEPAL_WIDTH    0.191514     4

Change your service level


In your notebook, you can change the service level of your connection to Oracle Autonomous Database to take advantage of different parallelism options. Available parallelism is relative to your autonomous database compute resource settings. Click the gear icon in the upper right (as indicated by the arrow in the figure), then click individual interpreters to turn them on or off, and click and drag each interpreter box to change the default service level. The 'low' binding runs your functions and queries without parallelism, 'medium' allows limited parallelism, and 'high' allows your functions and queries to use up to the maximum number of compute resources allocated to your Autonomous Database.

Oracle Machine Learning, Python, Oracle Database Exam Prep, Oracle Database Certification, Oracle Database Preparation, Database Prep, Oracle Database Exam Prep

Wednesday, February 24, 2021

Introducing Oracle Machine Learning for Python

Oracle Machine Learning for Python, Oracle Database Certification, Oracle Database Career, Database Preparation

Data scientists and developers know the power of Python and Python's wide-spread adoption is a testament to its success. Now, Python users can extend this power when analyzing data in Oracle Autonomous Database. Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data.

Designed for problems involving both large and small data volumes, Oracle Machine Learning for Python integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for data exploration, statistical analysis, and machine learning on database tables and views using Python syntax. Familiar Python functions are overloaded to translate Python functionality into SQL for in-database processing - achieving performance and scalability - transparently.

Python users can take advantage of parallelized in-database algorithms to enable scalable model building and data scoring - eliminating costly data movement. Further, Python users can develop and deploy user-defined Python functions that leverage the parallelism and scalability of Autonomous Database, and deploy those same user-defined Python functions using environment-managed Python engines through a REST API.

Oracle Machine Learning for Python also introduces automated machine learning (AutoML), which consists of: automated algorithm selection to select the algorithm most appropriate for the provided data, automated feature selection to enhance model accuracy and performance, and automated model tuning to improve model quality. AutoML enhances data scientist productivity by automating repetitive and time-consuming tasks, while also enabling non-experts to produce models without needing detailed algorithm-specific knowledge.

Oracle Machine Learning for Python, Oracle Database Certification, Oracle Database Career, Database Preparation

Access Oracle Machine Learning for Python in Autonomous Database using Oracle Machine Learning Notebooks, where you can use Python and SQL in the same Apache Zeppelin-based notebook - allowing the most appropriate API for the task. Take advantage of team collaboration and job scheduling features to further your data science project goals.

Oracle Machine Learning for Python has a range of template example notebooks included with Oracle Autonomous Database that highlight various features. Zeppelin notebooks illustrating OML4Py features are also available in the Oracle Machine Learning GitHub repository.

Source: oracle.com

Tuesday, February 23, 2021

JSON_TRANSFORM in Oracle Database 21c

Oracle Database 21c, Oracle Database Tutorial and Material, Oracle Database Learning, Oracle Database Career, Oracle Database Preparation

The JSON_TRANSFORM function has more functionality, and is easier to use. 

◉ Setup

The examples in this article use the following table. We are using the JSON data type, introduced in Oracle database 21c. We could have used any supported data type, including VARCHAR2, CLOB or BLOB.

-- drop table t1 purge;

create table t1 (

  id         number,

  json_data  json,

  constraint t1_pk primary key (id)

);

We insert two rows of test data.

insert into t1 (id, json_data) values (1, json('{"fruit":"apple","quantity":10}'));

insert into t1 (id, json_data) values (2, json('{"produce":[{"fruit":"apple","quantity":10},{"fruit":"orange","quantity":15}]}'));

commit;

From the output below we can see row 1 contains a flat JSON object, and row 2 contains an array of JSON objects.

set linesize 100 pagesize 1000 long 1000000

column data format a60

select id, json_serialize(json_data pretty) as data

from   t1;

        ID DATA

---------- ------------------------------------------------------------

         1 {

             "fruit" : "apple",

             "quantity" : 10

           }

         2 {

             "produce" :

             [

               {

                 "fruit" : "apple",

                 "quantity" : 10

               },

               {

                 "fruit" : "orange",

                 "quantity" : 15

               }

             ]

           }

SQL>

◉ SET Operation

The following SET operation updates the quantity value from 10 to 20. The output is returned as a CLOB using the RETURNING clause, with the PRETTY keyword to pretty-printed the output.

select json_transform(json_data,

                      set '$.quantity' = 20

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

--------------------------------------------------------------------------------

{

  "fruit" : "apple",

  "quantity" : 20

}

SQL>

If we use SET to amend an element that isn't already present, the default operation is to create it. Here we use the SET operation to add a new element called "updated_date".

select json_transform(json_data,

                      set '$.updated_date' = systimestamp

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

--------------------------------------------------------------------------------

{

  "fruit" : "apple",

  "quantity" : 10,

  "updated_date" : "2020-12-20T15:35:36.286485Z"

}

SQL>

We can use complex JSON object values using the JSON constructor or FORMAT JSON. Without these the value would just be added as an escaped string, rather than a JSON object. Here we show both methods to add a new element called "additional_info", which has a JSON object as its value.

select json_transform(json_data,

                      set '$.additional_info' = json('{"colour":"red","size":"large"}')

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "fruit" : "apple",

  "quantity" : 10,

  "additional_info" :

  {

    "colour" : "red",

    "size" : "large"

  }

}

SQL>

select json_transform(json_data,

                      set '$.additional_info' = '{"colour":"red","size":"large"}' format json

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "fruit" : "apple",

  "quantity" : 10,

  "additional_info" :

  {

    "colour" : "red",

    "size" : "large"

  }

}

SQL>

The SET operation works equally well for arrays, as shown in the following examples.

-- Set quantity to 20 for first item in the produce array.

select json_transform(json_data,

                      set '$.produce[0].quantity' = 20

                      returning clob pretty) as data

from   t1

where  id = 2;

DATA

------------------------------------------------------------

{

  "produce" :

  [

    {

      "fruit" : "apple",

      "quantity" : 20

    },

    {

      "fruit" : "orange",

      "quantity" : 15

    }

  ]

}

SQL>

-- Add the updated_date element to the first item in the produce array.

select json_transform(json_data,

                      set '$.produce[0].updated_date' = systimestamp

                      returning clob pretty) as data

from   t1

where  id = 2;

DATA

------------------------------------------------------------

{

  "produce" :

  [

    {

      "fruit" : "apple",

      "quantity" : 10,

      "updated_date" : "2021-01-30T08:10:38.368785Z"

    },

    {

      "fruit" : "orange",

      "quantity" : 15

    }

  ]

}

SQL>

-- Add the updated_date element to all items in the produce array.

select json_transform(json_data,

                      set '$.produce[*].updated_date' = systimestamp

                      returning clob pretty) as data

from   t1

where  id = 2;

DATA

------------------------------------------------------------

{

  "produce" :

  [

    {

      "fruit" : "apple",

      "quantity" : 10,

      "updated_date" : "2021-01-30T08:10:55.828378Z"

    },

    {

      "fruit" : "orange",

      "quantity" : 15,

      "updated_date" : "2021-01-30T08:10:55.828378Z"

    }

  ]

}

SQL>

The default behaviour of the SET operation can be altered using the following handlers.

◉ REPLACE ON EXISTING (default), ERROR ON EXISTING, IGNORE ON EXISTING

◉ CREATE ON MISSING (default), ERROR ON MISSING, IGNORE ON MISSING

◉ NULL ON NULL (default), ERROR ON NULL, IGNORE ON NULL, REMOVE ON NULL

For example, to raise an error if we try to amend an item that isn't present, we would do the following.

select json_transform(json_data,

                      set '$.updated_date' = systimestamp error on missing

                      returning clob pretty) as data

from   t1

where  id = 1;

Error report -

ORA-40762: missing value in JSON_TRANSFORM ()

SQL>

Many of the following operations can be replicated using the SET operation with the correct handlers.

◉ INSERT Operation

The INSERT operation is used to add a new element which doesn't already exist. Here we use the INSERT operation to add a new element called "updated_date".

select json_transform(json_data,

                      insert '$.updated_date' = systimestamp

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "fruit" : "apple",

  "quantity" : 10,

  "updated_date" : "2021-01-05T08:44:58.406618Z"

}

SQL>

This differs from the default SET operation, in that is produces an error if the item already exists. Here we use the INSERT operation to add a new element called "fruit". Since this element already exists, it produces an error.

select json_transform(json_data,

                      insert '$.fruit' = 'orange'

                      returning clob pretty) as data

from   t1

where  id = 1;

Error report -

ORA-40763: existing value in JSON_TRANSFORM ()

SQL>

So it's similar to using the SET operation with the ERROR ON EXISTING handler.

select json_transform(json_data,

                      set '$.fruit' = 'orange' error on existing

                      returning clob pretty) as data

from   t1

where  id = 1;

ORA-40763: existing value in JSON_TRANSFORM ()

SQL>

The INSERT operation can also be used to add an element to an array. In these examples we add a new "fruit" to different positions in the "produce" array. Notice this position is specified in the search path.

-- Added to first position in the array.

select json_transform(json_data,

                      insert '$.produce[0]' = JSON('{"fruit":"banana","quantity":20}')

                      returning clob pretty) as data

from   t1

where  id = 2;

DATA

------------------------------------------------------------

{

  "produce" :

  [

    {

      "fruit" : "banana",

      "quantity" : 20

    },

    {

      "fruit" : "apple",

      "quantity" : 10

    },

    {

      "fruit" : "orange",

      "quantity" : 15

    }

  ]

}

SQL>

-- Added to second position in the array.

select json_transform(json_data,

                      insert '$.produce[1]' = JSON('{"fruit":"banana","quantity":20}')

                      returning clob pretty) as data

from   t1

where  id = 2;

DATA

------------------------------------------------------------

{

  "produce" :

  [

    {

      "fruit" : "apple",

      "quantity" : 10

    },

    {

      "fruit" : "banana",

      "quantity" : 20

    },

    {

      "fruit" : "orange",

      "quantity" : 15

    }

  ]

}

SQL>

-- Appended to the end of the array.

select json_transform(json_data,

                      insert '$.produce[last+1]' = JSON('{"fruit":"banana","quantity":20}')

                      returning clob pretty) as data

from   t1

where  id = 2;

DATA

------------------------------------------------------------

{

  "produce" :

  [

    {

      "fruit" : "apple",

      "quantity" : 10

    },

    {

      "fruit" : "orange",

      "quantity" : 15

    },

    {

      "fruit" : "banana",

      "quantity" : 20

    }

  ]

}

SQL>

The default behaviour of the INSERT operation can be altered using the following handlers.

◉ ERROR ON EXISTING (default), IGNORE ON EXISTING, REPLACE ON EXISTING

◉ NULL ON NULL (default), ERROR ON NULL, IGNORE ON NULL, REMOVE ON NULL

◉ APPEND Operation

The APPEND operation is used to add a new element to the end of an array. Here we use the APPEND operation to add a new "fruit" at the end of the "produce" array.

select json_transform(json_data,

                      append '$.produce' = JSON('{"fruit":"banana","quantity":20}')

                      returning clob pretty) as data

from   t1

where  id = 2;

DATA

------------------------------------------------------------

{

  "produce" :

  [

    {

      "fruit" : "apple",

      "quantity" : 10

    },

    {

      "fruit" : "orange",

      "quantity" : 15

    },

    {

      "fruit" : "banana",

      "quantity" : 20

    }

  ]

}

SQL>

This is similar to using the INSERT operation with the [last+1] position.

select json_transform(json_data,

                      insert '$.produce[last+1]' = JSON('{"fruit":"banana","quantity":20}')

                      returning clob pretty) as data

from   t1

where  id = 2;

DATA

------------------------------------------------------------

{

  "produce" :

  [

    {

      "fruit" : "apple",

      "quantity" : 10

    },

    {

      "fruit" : "orange",

      "quantity" : 15

    },

    {

      "fruit" : "banana",

      "quantity" : 20

    }

  ]

}

SQL>

The default behaviour of the APPEND operation can be altered using the following handlers.

◉ ERROR ON MISSING (default), IGNORE ON MISSING, CREATE ON MISSING

◉ NULL ON NULL (default), ERROR ON NULL, IGNORE ON NULL

◉ REMOVE Operation

The REMOVE operation is used to delete an element from an object or an array. Here we use the REMOVE operation to delete the "quantity" element from an object, from an object in an array, and remove a "fruit" from the "produce" array.

-- Remove an element from an object.

select json_transform(json_data,

                      remove '$.quantity'

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "fruit" : "apple"

}

SQL>

-- Remove an element from an object in an array.

select json_transform(json_data,

                      remove '$.produce[0].quantity'

                      returning clob pretty) as data

from   t1

where  id = 2;

DATA

------------------------------------------------------------

{

  "produce" :

  [

    {

      "fruit" : "apple"

    },

    {

      "fruit" : "orange",

      "quantity" : 15

    }

  ]

}

-- Remove an element from the "produce" array.

select json_transform(json_data,

                      remove '$.produce[1]'

                      returning clob pretty) as data

from   t1

where  id = 2;

DATA

------------------------------------------------------------

{

  "produce" :

  [

    {

      "fruit" : "apple",

      "quantity" : 10

    }

  ]

}

SQL>

The default behaviour of the REMOVE operation can be altered using the following handlers.

◉ IGNORE ON MISSING (default), ERROR ON MISSING

◉ RENAME Operation

The RENAME operation is used to rename an element. Here we use the RENAME operation to rename the "fruit" element to "fruit_name".

select json_transform(json_data,

                      rename '$.fruit' = 'fruit_name'

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "quantity" : 10,

  "fruit_name" : "apple"

}

SQL>

The default behaviour of the RENAME operation can be altered using the following handlers.

IGNORE ON MISSING (default), ERROR ON MISSING

◉ REPLACE Operation

The REPLACE operation is used to update the value of an element. Here we use the REPLACE operation to update the "quantity" value from 10 to 20.

select json_transform(json_data,

                      replace '$.quantity' = 20

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "fruit" : "apple",

  "quantity" : 20

}

SQL>

If the element doesn't exist the operation is ignored. A new element is not created. Here we use the REPLACE operation to update the "updated_date" value. The "updated_date" element doesn't exist, so no action is taken.

select json_transform(json_data,

                      replace '$.updated_date' = systimestamp

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "fruit" : "apple",

  "quantity" : 10

}

SQL>

This is similar to using the SET operation with the IGNORE ON MISSING handler.

select json_transform(json_data,

                      replace '$.updated_date' = systimestamp

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "fruit" : "apple",

  "quantity" : 10

}

SQL>

The default behaviour of the REPLACE operation can be altered using the following handlers.

◉ IGNORE ON MISSING (default), ERROR ON MISSING, CREATE ON MISSING

◉ NULL ON NULL (default), ERROR ON NULL, IGNORE ON NULL, REMOVE ON NULL

◉ KEEP Operation

The KEEP operation is used to remove all elements except those included in the comma-separated list or search paths. Using the "$" search path returns an empty JSON document.

-- Remove everything.

select json_transform(json_data,

                      keep '$'

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

}

SQL>

-- Remove everything except the "fruit" element.

select json_transform(json_data,

                      keep '$.fruit'

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "fruit" : "apple"

}

SQL>

-- Remove everything except the "fruit" and "quantity" elements (remove nothing).

select json_transform(json_data,

                      keep '$.fruit', '$.quantity'

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "fruit" : "apple",

  "quantity" : 10

}

SQL>

◉ Combining Multiple Operations

Multiple operations can be combined into a single JSON_TRANSFORM call. They are processed in order, and if one operation fails they all fail.

select json_transform(json_data,

                      set '$.created_date' = systimestamp,

                      set '$.updated_date' = systimestamp,

                      rename '$.fruit' = 'fruit_type',

                      replace '$.quantity' = 20

                      returning clob pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "quantity" : 20,

  "created_date" : "2020-12-20T17:25:34.539480Z",

  "updated_date" : "2020-12-20T17:25:34.539480Z",

  "fruit_type" : "apple"

}

SQL>

◉ Direct Updates Using JSON_TRANSFORM

All the examples so far have performed transformations on the fly as part of SELECT statements, but we could just as easily do the transformations as part of an UPDATE statement.

-- Update the data directly in the table.

update t1

set    json_data = json_transform(json_data,

                                  set '$.created_date' = systimestamp,

                                  set '$.updated_date' = systimestamp,

                                  rename '$.fruit' = 'fruit_type',

                                  replace '$.quantity' = 20

                                  returning json)

where  id = 1;

-- Display the updated data.

select json_serialize(json_data pretty) as data

from   t1

where  id = 1;

DATA

------------------------------------------------------------

{

  "quantity" : 20,

  "created_date" : "2020-12-20T17:39:30.811689Z",

  "updated_date" : "2020-12-20T17:39:30.811689Z",

  "fruit_type" : "apple"

}

SQL>

rollback;

◉ RETURNING Clause

Oracle Database 21c, Oracle Database Tutorial and Material, Oracle Database Learning, Oracle Database Career, Oracle Database Preparation
The output of the JSON_TRANSFORM function depends on the input expression. If they input is a JSON data type, the output is also a JSON data type. All other input types result in a VARCHAR2(4000) return value. The RETURNING clause allows the output to be converted to JSON, BLOB, CLOB or VARCHAR2. Most of the examples above use RETURNING CLOB to make the output readable.

◉ PL/SQL Support

There is no PL/SQL support for direct assignments using the JSON_TRANSFORM function. The following attempt results in an error.

set serveroutput on

declare

  l_json_in   varchar2(32767);

  l_json_out  varchar2(32767);

begin

  l_json_in  := '{"fruit":"apple","quantity":10}';


  l_json_out := json_transform(l_json_in,

                               set '$.updated_date' = systimestamp

                               returning varchar2 pretty);

  dbms_output.put_line(l_json_out);

end;

/

Error report -

ORA-06550: line 8, column 36:

PLS-00103: Encountered the symbol "$.updated_date" when expecting one of the following:

   . ( ) , * @ % & | = - + < / > at in is mod remainder not null

   rem returning with => .. <an exponent (**)> <> or != or ~= >=

   <= <> and or default like like2 like4 likec between error ||

   multiset member empty submultiset lax strict without pretty

   ascii true false absent format allow truncate

The symbol "(" was substituted for "$.updated_date" to continue.

An assignment can be made using a SELECT ... INTO ... FROM DUAL statement, as shown here.

set serveroutput on

declare

  l_json_in   varchar2(32767);

  l_json_out  varchar2(32767);

begin

  l_json_in  := '{"fruit":"apple","quantity":10}';

  select json_transform(l_json_in,

                        set '$.updated_date' = systimestamp

                        returning varchar2 pretty)

  into   l_json_out

  from   dual;

  dbms_output.put_line(l_json_out);

end;

/

{

  "fruit" : "apple",

  "quantity" : 10,

  "updated_date" : "2020-12-21T09:44:33.150459Z"

}

PL/SQL procedure successfully completed.

SQL>

Monday, February 22, 2021

Qualified Expressions Enhancements in Oracle Database 21c

Oracle Database 21c, Oracle Database Tutorial and Material, Oracle Database Preparation, Oracle Database Learning, Oracle Database Career

Oracle database 21c makes it even easier to populate collections using qualified expressions.

◉ Before We Begin

This post makes some assumptions about prior knowledge.

- You have an understanding of the different types of Oracle collections.

- You have an understanding of qualified expressions.

- You have an understanding of the new FOR LOOP iteration enhancements introduced in Oracle database 21c.

- You have an understanding of bulk binds.

◉ Setup

Some of the examples in this post use the following objects.

create or replace function num_to_word (p_num number) return varchar2 is

begin

  return upper(to_char(to_date(p_num, 'j'), 'jsp'));

end;

/

-- drop table t1 purge;

create table t1 as

select level+10 as id,

       num_to_word(level+10) as description,

       trunc(sysdate) as created_date

from   dual

connect by level <= 5;

The query below displays the data in the table.

column description format a20

column created_date format a12

select id, description, created_date from t1;

     ID    DESCRIPTION       CREATED_DATE

---------- -------------------- ------------

        11     ELEVEN               02-JAN-21

        12     TWELVE               02-JAN-21

        13     THIRTEEN            02-JAN-21

        14     FOURTEEN          02-JAN-21

        15     FIFTEEN               02-JAN-21

SQL>

All of the examples are being run in SQLcl and assume you have the server output enabled.

SQL> set serveroutput on

◉ Positional Notation for Associative Array

From 18c onward we've been able to populate an associative array (index-by table) using a qualified expression and named association, but 21c now allows us to populate associative arrays using positional notation, similar to how we can populate nested tables and varrays. When populating an associative array using positional notation, the index is an integer from 1 to N, based on the order of the elements specified.

declare

  type t_tab is table of varchar2(10) index by pls_integer;

  l_tab t_tab;

begin

  -- Pre-18c - Direct assignment to elements of the collection.

  l_tab(1) := 'ONE';

  l_tab(2) := 'TWO';

  l_tab(3) := 'THREE';

  -- 18c - Qualified expression using named association.

  l_tab := t_tab(1 => 'ONE',

                 2 => 'TWO',

                 3 => 'THREE');


  -- 21c - Qualified expression using positional notation.

  l_tab := t_tab('ONE', 'TWO', 'THREE');

end;

/

◉ Basic Iterator

The basic iterator populates the collection as follows.

- index : The value of the iterand.

- value : The value of the expression to the right of "=>". The expression can reference the iterand value.

In this example we populate the collection with indexes 1 to 5, and values of i+10.

declare

  type tab_t is table of pls_integer index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 5 => i+10); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  value=' || v);

  end loop;

end;

/

index=1  value=11

index=2  value=12

index=3  value=13

index=4  value=14

index=5  value=15

PL/SQL procedure successfully completed.

SQL>

The expression can evaluate to a different type. In the following example we use a collection of VARCHAR2, and populate it with the word equivalent of the iterand.

declare

  type tab_t is table of varchar2(50) index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 5 => num_to_word(i)); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  value=' || v);

  end loop;

end;

/

index=1  value=ONE

index=2  value=TWO

index=3  value=THREE

index=4  value=FOUR

index=5  value=FIVE

PL/SQL procedure successfully completed.

SQL>

We can use most of the new features of the FOR LOOP introduced in Oracle database 21c. The following example includes stepped iteration, skipping and stopping. It's a rather silly example, but it demonstrates their inclusion.

declare

  type tab_t is table of varchar2(50) index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 50 by 2 while i < 20 when i > 10 => num_to_word(i)); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  value=' || v);

  end loop;

end;

/

index=11  value=ELEVEN

index=13  value=THIRTEEN

index=15  value=FIFTEEN

index=17  value=SEVENTEEN

index=19  value=NINETEEN

PL/SQL procedure successfully completed.

SQL>

◉ Index Iterator

The index iterator populates the collection as follows.

- index : The value of the expression to the left of "=>". The expression can reference the iterand value.

- value : The value of the expression to the right of "=>". The expression can reference the iterand value.

In this example we populate the collection with indexes 10 to 50, and values 100 to 500.

declare

  type tab_t is table of pls_integer index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 5 index i*10 => i*100); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  value=' || v);

  end loop;

end;

/

index=10  value=100

index=20  value=200

index=30  value=300

index=40  value=400

index=50  value=500

PL/SQL procedure successfully completed.

SQL>

The expressions can evaluate to different types. In the following example we use a collection of DATE indexed by VARCHAR2.

declare

  type tab_t is table of date index by varchar2(50);

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 5 index num_to_word(i) => sysdate+i); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || rpad(i,5,' ') || '  value=' || v);

  end loop;

end;

/

index=FIVE   value=07-JAN-21

index=FOUR   value=06-JAN-21

index=ONE    value=03-JAN-21

index=THREE  value=05-JAN-21

index=TWO    value=04-JAN-21

PL/SQL procedure successfully completed.

SQL>

We can use most of the new features of the FOR LOOP introduced in Oracle database 21c. The following example includes stepped iteration, skipping and stopping.

declare

  type tab_t is table of date index by varchar2(50);

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 50 by 2 while i < 20 when i > 10 index num_to_word(i) => sysdate+i); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || rpad(i,9,' ') || '  value=' || v);

  end loop;

end;

/

index=ELEVEN     value=13-JAN-21

index=FIFTEEN    value=17-JAN-21

index=NINETEEN   value=21-JAN-21

index=SEVENTEEN  value=19-JAN-21

index=THIRTEEN   value=15-JAN-21

PL/SQL procedure successfully completed.

SQL>

◉ Sequence Iterator

The sequence iterator populates the collection as follows.

- index : The value 1 to N. It does not relate to the iterand.

- value : The value of the expression to the right of "=>". The expression can reference the iterand value.

In this example we populate the collection with indexes 1 to 5, even though the iterand is 11 to 15, and values of the i+10.

declare

  type tab_t is table of pls_integer index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 11 .. 15 sequence => i+10); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  value=' || v);

  end loop;

end;

/

index=1  value=21

index=2  value=22

index=3  value=23

index=4  value=24

index=5  value=25

PL/SQL procedure successfully completed.

SQL>

The expression can evaluate to a different type. In the following example we use a collection of VARCHAR2, and populate it with the word equivalent of the iterand. Once again, the index is not related to the iterand.

declare

  type tab_t is table of varchar2(50) index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 11 .. 15 sequence => num_to_word(i)); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  value=' || v);

  end loop;

end;

/

index=1  value=ELEVEN

index=2  value=TWELVE

index=3  value=THIRTEEN

index=4  value=FOURTEEN

index=5  value=FIFTEEN

PL/SQL procedure successfully completed.

SQL>

We can use most of the new features of the FOR LOOP introduced in Oracle database 21c. The following example includes stepped iteration, skipping and stopping. Remember, the iteration control applies to the value of the iterand, not the index.

declare

  type tab_t is table of varchar2(50) index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 50 by 2 while i < 20 when i > 10 sequence => num_to_word(i)); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  value=' || v);

  end loop;

end;

/

index=1  value=ELEVEN

index=2  value=THIRTEEN

index=3  value=FIFTEEN

index=4  value=SEVENTEEN

index=5  value=NINETEEN

PL/SQL procedure successfully completed.

SQL>

◉ Nested Tables and Varrays

The examples so far have focused on associative arrays (index-by tables), but the index and sequence iterator syntax can also be used for nested tables and varrays. The example below shows both iterators with nested tables.

declare

  type tab_t is table of varchar2(50);

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 5 sequence => num_to_word(i)); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  value=' || v);

  end loop;

end;

/

index=1  value=ONE

index=2  value=TWO

index=3  value=THREE

index=4  value=FOUR

index=5  value=FIVE

PL/SQL procedure successfully completed.

SQL>

declare

  type tab_t is table of varchar2(50);

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 5 index i => num_to_word(i)); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  value=' || v);

  end loop;

end;

/

index=1  value=ONE

index=2  value=TWO

index=3  value=THREE

index=4  value=FOUR

index=5  value=FIVE

PL/SQL procedure successfully completed.

SQL>

Attempting to use a basic iterator always results in an error.

declare

  type tab_t is table of varchar2(50);

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 5 => num_to_word(i)); 


  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  value=' || v);

  end loop;

end;

/

Error report -

ORA-06550: line 6, column 18:

PLS-00868: The iterand type for an iteration control is not compatible with the collection index type, use SEQUENCE, or INDEX iterator association instead of a basic iterator association.

Oracle Database 21c, Oracle Database Tutorial and Material, Oracle Database Preparation, Oracle Database Learning, Oracle Database Career
The index iterator does something odd if we don't start from 1, or if we alter the expression on the left side of "=>". It generates empty collection elements.

declare

  type tab_t is table of varchar2(50);

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 6 .. 10 index i => num_to_word(i)); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || rpad(i,2,' ') || '  value=' || v);

  end loop;

end;

/

index=1   value=

index=2   value=

index=3   value=

index=4   value=

index=5   value=

index=6   value=SIX

index=7   value=SEVEN

index=8   value=EIGHT

index=9   value=NINE

index=10  value=TEN

PL/SQL procedure successfully completed.

SQL>

declare

  type tab_t is table of varchar2(50);

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in 1 .. 5 index i+5 => num_to_word(i)); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || rpad(i,2,' ') || '  value=' || v);

  end loop;

end;

/

index=1   value=

index=2   value=

index=3   value=

index=4   value=

index=5   value=

index=6   value=ONE

index=7   value=TWO

index=8   value=THREE

index=9   value=FOUR

index=10  value=FIVE

PL/SQL procedure successfully completed.

SQL>

I'm not sure if this is intentional, or a bug.

◉ Populating Collections From Database Tables

There are a number of existing ways to populate a collection from a database table. Probably the most efficient way is to use a bulk bind, which works well with associative arrays, nested table and varrays. The following example uses BULK COLLECT to build an associative array from the contents of the T1 table. Unfortunately, the index of the associative array is just a sequence from 1 to N, rather than matching the primary key column.

declare

  type tab_t is table of t1%rowtype index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  select id, description, created_date

  bulk collect into l_tab

  from t1;

  -- Display the contents of the collection.

  for i in 1 .. l_tab.count loop

    dbms_output.put_line('index=' || i || '  id=' || l_tab(i).id ||

                         '  description=' || rpad(l_tab(i).description,8,' ') ||

                         '  created_date=' || l_tab(i).created_date);

  end loop;

end;

/

index=1  id=11  description=ELEVEN    created_date=02-JAN-21

index=2  id=12  description=TWELVE    created_date=02-JAN-21

index=3  id=13  description=THIRTEEN  created_date=02-JAN-21

index=4  id=14  description=FOURTEEN  created_date=02-JAN-21

index=5  id=15  description=FIFTEEN   created_date=02-JAN-21

PL/SQL procedure successfully completed.

SQL>

You can read about the performance benefits of bulk binds here.

If we wanted to populate an associative array from the table and make the array index match the primary key column, we would typically have to do something like the following. We define an associative array (index-by table) type (tab_t) based on the row type. We create a variable (l_tab) based on the table type. We use a cursor FOR LOOP to retrieve the rows and use them to populate the collection, using the ID value for the collection index for convenience. We can then display the contents of the collection.

declare

  type tab_t is table of t1%rowtype index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  for cur_rec in (select id, description, created_date from t1)

  loop

    l_tab(cur_rec.id) := cur_rec;

  end loop;

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  id=' || v.id ||

                         '  description=' || rpad(v.description,8,' ') ||

                         '  created_date=' || v.created_date);

  end loop;

end;

/

index=11  id=11  description=ELEVEN    created_date=02-JAN-21

index=12  id=12  description=TWELVE    created_date=02-JAN-21

index=13  id=13  description=THIRTEEN  created_date=02-JAN-21

index=14  id=14  description=FOURTEEN  created_date=02-JAN-21

index=15  id=15  description=FIFTEEN   created_date=02-JAN-21

PL/SQL procedure successfully completed.

SQL>

With the enhancements to the qualified expressions, we can populate a collection in a single step. We define the table type in the same way, but we can populate the collection from the query directly using the table type constructor and an iterator based on a query. We use the INDEX iterator to indicate we want the collection index to be based on the ID column from the query. The operand matches the rowtype of the query.

declare

  type tab_t is table of t1%rowtype index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in (select id, description, created_date from t1) index i.id => i); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  id=' || v.id ||

                         '  description=' || rpad(v.description,8,' ') ||

                         '  created_date=' || v.created_date);

  end loop;

end;

/

index=11  id=11  description=ELEVEN    created_date=02-JAN-21

index=12  id=12  description=TWELVE    created_date=02-JAN-21

index=13  id=13  description=THIRTEEN  created_date=02-JAN-21

index=14  id=14  description=FOURTEEN  created_date=02-JAN-21

index=15  id=15  description=FIFTEEN   created_date=02-JAN-21

PL/SQL procedure successfully completed.

SQL>

The fact we are using an iterator means we have access to the skipping and stopping functionality of the iterator if we need it. The following example uses the WHEN clause to only match rows with an ID column value that is even. Of course, you could argue this would be better done in the query itself.

declare

  type tab_t is table of t1%rowtype index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in (select id, description, created_date from t1) when mod(i.id,2)=0 index i.id => i); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  id=' || v.id ||

                         '  description=' || rpad(v.description,8,' ') ||

                         '  created_date=' || v.created_date);

  end loop;

end;

/

index=12  id=12  description=TWELVE    created_date=02-JAN-21

index=14  id=14  description=FOURTEEN  created_date=02-JAN-21

PL/SQL procedure successfully completed.

SQL>

If we didn't care about the collection index matching the ID column from the query, we could just use the SEQUENCE iterator to let it use a default sequence.

declare

  type tab_t is table of t1%rowtype index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in (select id, description, created_date from t1) sequence => i); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  id=' || v.id ||

                         '  description=' || rpad(v.description,8,' ') ||

                         '  created_date=' || v.created_date);

  end loop;

end;

/

index=1  id=11  description=ELEVEN    created_date=02-JAN-21

index=2  id=12  description=TWELVE    created_date=02-JAN-21

index=3  id=13  description=THIRTEEN  created_date=02-JAN-21

index=4  id=14  description=FOURTEEN  created_date=02-JAN-21

index=5  id=15  description=FIFTEEN   created_date=02-JAN-21

PL/SQL procedure successfully completed.

SQL>

In the previous examples we used an implicit cursor. The iterators work equally well with an explicit cursor, as demonstrated below.

declare

  cursor c_cursor is

    select id, description, created_date from t1;

  type tab_t is table of t1%rowtype index by pls_integer;

  l_tab  tab_t;

begin

  -- Populate the collection.

  l_tab := tab_t(for i in c_cursor index i.id => i); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  id=' || v.id ||

                         '  description=' || rpad(v.description,8,' ') ||

                         '  created_date=' || v.created_date);

  end loop;

end;

/

index=11  id=11  description=ELEVEN    created_date=02-JAN-21

index=12  id=12  description=TWELVE    created_date=02-JAN-21

index=13  id=13  description=THIRTEEN  created_date=02-JAN-21

index=14  id=14  description=FOURTEEN  created_date=02-JAN-21

index=15  id=15  description=FIFTEEN   created_date=02-JAN-21

PL/SQL procedure successfully completed.

SQL>

We can also use a REF CURSOR type. There are two things to notice here. First, we have to open and close the ref cursor ourselves. Second, we've had to explicitly type the operand, as we've used a weakly typed ref cursor.

declare

  type tab_t is table of t1%rowtype index by pls_integer;

  l_tab  tab_t;

  l_sql     varchar2(32767);     

  l_cursor  sys_refcursor;

begin

  l_sql := 'select id, description, created_date from t1';

  -- Populate the collection.

  open l_cursor for l_sql;

  l_tab := tab_t(for i t1%rowtype in values of l_cursor index i.id => i); 

  close l_cursor;

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  id=' || v.id ||

                         '  description=' || rpad(v.description,8,' ') ||

                         '  created_date=' || v.created_date);

  end loop;

end;

/

index=11  id=11  description=ELEVEN    created_date=02-JAN-21

index=12  id=12  description=TWELVE    created_date=02-JAN-21

index=13  id=13  description=THIRTEEN  created_date=02-JAN-21

index=14  id=14  description=FOURTEEN  created_date=02-JAN-21

index=15  id=15  description=FIFTEEN   created_date=02-JAN-21

PL/SQL procedure successfully completed.

SQL>

The documentation suggests it should also be possible to use dynamic SQL using the EXECUTE IMMEDIATE command, but that seems to give errors when I try.

declare

  type tab_t is table of t1%rowtype index by pls_integer;

  l_tab  tab_t;

  l_sql  varchar2(32767);

begin

  l_sql := 'select id, description, created_date from t1';

  -- Populate the collection.

  l_tab := tab_t(for i t1%rowtype in (execute immediate l_sql) index i.id => i); 

  -- Display the contents of the collection.

  for i,v in pairs of l_tab loop

    dbms_output.put_line('index=' || i || '  id=' || v.id ||

                         '  description=' || rpad(v.description,8,' ') ||

                         '  created_date=' || v.created_date);

  end loop;

end;

/

Error report -

ORA-06550: line 13, column 33:

PLS-00801: internal error [*** ASSERT at file pdz4.c, line 3518; Self is null.; Xanon__0x1fc208ea0__AB[10, 38]]

06550. 00000 -  "line %s, column %s:\n%s"

*Cause:    Usually a PL/SQL compilation error.

*Action:

SQL>