Database links in Autonomous Database Serverless are the past - Cloud links are the future

As many of you know, database links are an established mechanism to allow other remote databases to access specific tables or views in your database.

Database links have been around for decades and always require a two-way interaction in one of two ways: The remote (accessing) side contacts you, the data owner, to ask for access and to get the specifics of how to connect to your system. Alternatively, as the data owner, you must proactively contact the remote side and share the access details for the remote side to set up the database link. With Oracle Autonomous Database Serverless, this is a thing of the past.

With Cloud Links, new functionality in Autonomous Database Serverless, the data owner registers a table or view for remote access for a selected audience defined by the data owner. The data is then instantaneously accessible by everybody who got remote access granted at registration time. No further actions are required to set up a Cloud Link, and whoever is supposed to see and access your data will be able to discover and work with the data made available to them.

Sounds almost too cool to be true, doesn't it? Let's step through it how it works.

Cloud Links at work

Let's assume I have central sales information in my autonomous database that other autonomous databases need to access remotely now and then. "Trusted" autonomous databases in the same compartment than my system should be able to access all my detail sales data, whereas other satellite databases within my tenancy should only be able to see the aggregated sales information per sales channel. 

The objects I want to give remote access to look as follows:

You see that there is a base table SALES_ALL and a view SALES_VIEW_AGG defined on top of it that removes the customer information and aggregates the sales information for the other dimensions. The base table will be accessible within my trusted compartment, whereas the aggregated sales information without any customer information should be accessible for everybody in my tenancy.

After the Administrator of my Autonomous Database has given me the privilege to register a table (or view) for remote access with the scope needed for the task at hand, I simply register my table SALES for compartmental access and view SALES_VIEW_AGG for my tenancy (for brevity reasons, pls. consult the documentation for the privilege details):

What is it about this registration? 

Cloud Links introduce a new concept of regional namespace and name for any data that is made remotely accessible. Think of it as something similar to the database today, where one of the most famous Oracle tables ever has the name "EMP" and lives in the namespace "SCOTT". There can only be one SCOTT.EMP in your database. With Cloud Links, it's the same concept, just on a regional level and without being tied to a single database. And since it's not linked to a single database but needs some boundaries of visibility, there's a new concept of scope. The scope defines who can access your table or view through a cloud link remotely. The scope can be a region, tenancy, compartment, individual databases, or a combination of those. 

That was it. My view CLOUDLINK.SALES_VIEW_AGG will be remotely accessible within my tenancy as REGIONAL_SALES.SALES_AGG, and table CLOUDLINK.SALES_ALL as TRUSTED_COMPARTMENT.SALES without exposing its origin.

After a brief period of central metadata synchronization, my trusted databases in my compartment can access all my sales data, whereas all databases in my tenancy can access my high-level aggregated information (it normally takes 5 to 10 minutes). Any future database in my tenancy or the same compartment as my database will be able to access the same data, safely and filtered as required for their work based on the registration policies. 

I can verify what objects I registered for remote access for the different scopes in the data dictionary:

The scope of my two registered objects are, as expected, on the tenancy level for REGIONAL_SALES.SALES_AGG and on the compartment level inside my tenancy for TRUSTED_COMPARTMENT.SALES

On the remote (receiving) end, every autonomous database can see what remote objects they have access to by querying the data dictionary:

select * from dba_cloud_link_access;

Let's see what my trusted autonomous databases (in the same compartment) and others in my tenancy will see.

If I connect to a trusted autonomous database, I will see the following output on the left: I can see both remote data sets. In contrast, when I connect to an arbitrary autonomous database within the same tenancy as my autonomous database that registered the objects, the output will look different, as shown on the right: I can only see the data set shared on the tenant level.

Besides the trusted autonomous database and other autonomous databases in my tenancy, no one else will be able to discover or see the table and view that I registered in this example.

Now it's probably only sometimes known to you what data is made remotely available to you, so you can discover what was made available to you or even find particular data of interest by yourself. If you know the data (namespace, name) you can describe it explicitly, or, the more interesting case, you can see what's out there using free text search.

Voilà, we found the dataset that was shared with everybody without necessarily knowing about its existence.

How to work with registered data?

We registered some objects for remote access and verified that we can see these objects within the scope they were defined, but how do I access them now? I do not have any username/password or other means of authentication and authorization that I shared with a remote database that wants to access my data.

The authentication is done at the registration time of an object. In our example, the trusted autonomous database got access to my sales data by being a trusted database within the same compartment. The same is true for the autonomous database in my tenancy for the aggregated sales data. You only need your Administrator to give you the read privilege on cloud links for authorization (again, please consult the documentation for details here), and you're ready to read the remote data.

After having gotten the proper privilege, any remote object that is made accessible for your autonomous database and your user can be queried with standard "cloud link syntax", namely:

select .. from <namespace>.<name>@cloud$link;

You access remote data without any location knowledge:

That was not too hard to set up. If I can do it, you can do it for sure as well.

Source: oracle.com

Continue reading

Effortlessly set up customized clusters with OCI Big Data cluster profiles

The Oracle Cloud Infrastructure (OCI) Big Data service with Apache Hadoop is a fully managed cloud service from OCI that allows you to process big data workloads using popular open source frameworks, such as Hadoop, Spark, Hive, and HBase. One of the features of OCI Big Data is the ability to create and manage Hadoop clusters of different profiles or types. In this blog post, we discuss the cluster profile feature in OCI Big Data, its benefits, and how to use it.

What is OCI Big Data?

OCI Big Data is a cloud-based big data processing service offered by OCI. The fully managed platform enables customers to process large amounts of data using popular open source big data frameworks, such as Hadoop, Spark, Hive, and Kafka. The service is designed to simplify the process of deploying and managing big data solutions, and you can integrate it with other OCI services and on-premises systems.

Use cases for Oracle Big Data

The following examples show the top use cases for OCI Big Data:

• ETL processing: Extract, transform, load (ETL) is a common use case for OCI Big Data. You can use OCI Big Data to process large amounts of data, transform it into a format suitable for analysis, and load it into a data warehouse or other storage system.
• Data analysis: You can use OCI Big Data for data analysis and data exploration. You can use OCI Big Data to run Apache Spark jobs to analyze data and generate insights—useful for business intelligence, data visualization, and machine learning (ML) applications.
• Machine learning: You can use OCI Big Data for ML applications to train models on large datasets and then use the models for prediction and other tasks.
• Log processing: With OCI Big Data, you can process analyze log data from web servers, application servers, and other sources to identify patterns and trends.
• Batch processing: You can use OCI Big Data to process large amounts of data in batches, for example, to generate reports or perform calculations.
• Real-time processing: You can use OCI Big Data to process streams of data in real-time, for example, to perform fraud detection or anomaly detection.

Why customers love OCI Big Data service

Customers love OCI Big Data, a cloud-based big data analytics service, for the following common reasons:

• Scalability: OCI Big Data can easily scale to handle large amounts of data and processing power, enabling customers to gain insights quickly and efficiently.
• Compatibility: OCI Big Data supports various open source big data frameworks, such as Hadoop, Spark, Hive, and Kafka, which allows customers to use the tools they’re familiar with and use existing code.
• Security: OCI Big Data offers robust security features, including encryption at rest and in transit, and integrated authentication with OCI Identity and Access Management (IAM) service, providing customers with peace of mind that their data is protected.
• Flexibility: Customers can choose to deploy OCI Big Data in various ways, including using preconfigured clusters or creating custom clusters with specific configurations, enabling them to tailor the service to their specific needs.
• Integration with other OCI services: OCI Big Data integrates with other OCI services, such as OCI Data Catalog, OCI Data Flow, and OCI Lake House, and OCI makes it easy for customers to build end-to-end solutions for their big data needs.

What is a cluster profile?

A cluster profile in OCI Big Data represents a preconfigured set of resources optimized for a particular workload or use case. Each cluster profile has a predefined set of configuration parameters that are optimized for a specific data processing job. For example, OCI offers cluster profiles for Hadoop, Spark, HBase, Hive, and Trino (Interactive query), each designed for specific workloads and use cases.

Hadoop

The Hadoop cluster type in OCI Big Data is designed to work with Hadoop Distributed File System (HDFS) and MapReduce. It’s ideal for batch processing, such as data warehousing, log analysis, and ETL. The Hadoop cluster type comes with Hadoop, Hive, Pig, and Oozie preinstalled, making it easy to get started with big data processing.

Spark

The Spark cluster type in OCI Big Data is designed to work with Apache Spark, an open source data processing framework that supports both batch and streaming data processing. It’s ideal for real-time data processing, machine learning, and graph processing. The Spark cluster type comes with Spark, Hive, and Jupyter preinstalled, making it easy to start using Spark for big data processing.

HBase

The HBase cluster profile in OCI Big Data is designed to work with Apache HBase, an open source NoSQL database that runs on top of Hadoop. It’s ideal for storing and retrieving large amounts of structured data, such as sensor data, social media data, and financial data.

Trino

This cluster profile is for interactive querying of large datasets. Trino returns results to you as soon as they’re available. This availability offers data analysts and data scientists the ability to query large amounts of data, test hypotheses, run A/B testing, and build visualizations or dashboards.

Kafka

This cluster profile is designed for streaming data processing and supports Apache Kafka.

Hadoop Etxended

We used this cluster profile before the profile feature was introduced.

Benefits of cluster profiles

The use of cluster profile provides the following benefits:

• Faster cluster deployment: The use of preconfigured cluster profiles speeds up the deployment process by reducing the amount of manual configuration required.
• Better performance: Cluster profiles are optimized for specific workloads, providing better performance compared to a generic cluster setup.
• Simplified management: Each cluster profiles comes with preconfigured services, reducing the need for manual configuration and simplifying cluster management.

How to use cluster profiles in OCI Big Data

To create an OCI Big Data cluster using a specific cluster profile, use the following steps:

1. In the Oracle Cloud Console, navigate to the OCI Big Data service.
2. Click the Create cluster button.
3. Enter the cluster name and admin password.
4. Select the checkbox for Secure and Highly Available (HA) to make the cluster secure and highly available.
5. Select the distribution and version of Hadoop from either Oracle’s Distribution of Hadoop (ODH) or Cloudera’s Distribution of Hadoop (CDH).
6. Select the cluster profile that best suits your use case from the menu. You can also select the version of the cluster type that you want to use.
7. Select from the Compute shape, block storage for master and utility, and the number of Compute shape options for the worker nodes.
8. Provide the network related details, such as CIDR Block, virtual cloud network (VCN), and subnet details.
9. Select your encryption type: Oracle-managed or customer-managed.
10. Click Create to provision the Big Data cluster.

When the cluster is created, off you go! You can use it to process Big Data workloads using the services and tools enabled by default.

Source: oracle.com

Continue reading

Introducing Project Analytics in Oracle Fusion ERP Analytics

Whether managing large enterprise or small projects within an organization, project teams need agility to thrive in a constantly changing business environment. To reach their goals and reduce risk, modern project management organizations (PMOs) are data-driven and recognize the need to analyze their data and understand its impact and correlations for cross-functionally at every stage of the project lifecycle. A critical role includes budget control and margin optimization and for this purpose, they need not only a bird’s eye view of their costs, budgets, forecasts, and spend. They also need to be able to undertake detailed analyses to find hidden insights.

Oracle has introduced Project Analytics as part of Fusion ERP Analytics. Using a prebuilt integrated analytics approach, Project analytics helps project members identify cost optimization opportunities, understand revenue and billing trends, predict variances in budget and forecasting, and integrate all data across departments to enable faster decisions and better project outcomes.

Quickly infer financial health of the project portfolio with prebuilt and best practice KPIs

The biggest challenge is keeping project expenditures on track with allocated budgets as the project progresses. With information spread across multiple systems, project members spend a lot of their time on manual data extraction and manipulation, trying to keep track of and monitor budgets and forecasts, then analyze the variances from what was planned and from the original baseline cost.

Budget and forecast analysis: Project Analytics smooths out the budget and forecast analysis process. With prebuilt KPIs, project teams can keep financial health in check by identifying “at-risk” projects with proactive comparisons of budget and forecasts versus approved original and current baseline costs. Embedded analytics helps to predict budget and forecast overruns by leveraging trends from historical performance. This allows PMOs to make necessary adjustments in costs, expenses, resources, and commitments to reach project goals.

Three key questions that Project Analytics helps users quickly answer are:

◉ What's the overall financial health of projects that I'm managing?

◉ In which cost categories am I overrunning the budget?

◉ What's our current and updated forecast against budget?

Financial Performance Dashboard

Get timely visual insights on revenue and billing  

Organizations often have difficulty getting accurate and timely project performance information for themselves and their clients. With complex project data managed across siloed systems and continued reliance on multiple spreadsheets, it's hard for teams to connect the dots and understand what the up-to-date revenue trends are, who their top customers are, and who isn’t paying on time.

Revenue and Billing Analysis: With interactive dashboards and easy-to-use self-service visualizations, project teams can now follow day-to-day project trends and variances across the project execution phases and billing milestones. With complete visibility in trends across revenue, billings, customers, contracts, and funding allocations, project teams can instantly get the answer to their queries without waiting for IT to extract the reports they need:

◉ Who are my top ten customers, and who's the most profitable?

◉ Which are the top projects driving revenue?

◉ How do current billings compare with billings from the last quarter? 

◉ What's accrued revenue compared with billings right now?

◉ How am I performing against original funding contracts?

◉ Which of my projects are doing well and how do they compare with each other?

Time is of the essence. Particularly with the rise of virtual collaboration, there's an immediate need across project organizations to enable seamless and rapid communications. With built-in natural language processing (NLP) features, PMOs can use speech-driven visualizations to get insights related to revenue and billing at any time from any device. Ask time-sensitive questions such as “how's my revenue trending this month?” or “Which customers haven’t paid this month?” to receive visual insights and share the results with stakeholders.  

Mobile phone with voice

Improve controls over project costs and expenses

PMOs need to constantly monitor the status of their project costs and manage changes to the budget. Always a challenge as the project evolves, this requires project members to analyze various costing reports periodically—whether that's expenses, variances from initial estimates, or profit margins and commitments. All of this can be a time-consuming process, as project-related data and associated costs are often stored in different locations and managed by different people, thus reducing the visibility for teams into project costs and overall spending.  

Project Cost Analysis: Project Analytics provides a telescopic view, powered by prebuilt analysis, enabling dynamic decision-making with self-service visualizations, to quickly assess the impact of project costs on margins and budgets. With the ability to take a deep dive and uncover variances proactively, project members can make project cost adjustments and improve project control. Get detailed insights into the drivers of cost commitments and expenses. Slice and dice data in multiple ways at any time—either by project organization, project type, or category. Deep dive into revenue and cost distributions at account level details by project, or general ledger account and fiscal period using prebuilt analyses and metrics. Some of the queries that the prebuilt cost analysis can answer include:

◉ Where am I forecasting to overspend and how can I control that?

◉ Which resources can I use to maximize profitability and minimize cost?

◉ What's my revenue variance quarter over quarter?

◉ How's my cost trending over the last six months by project and category?

Project Cost Analysis

Gain an integrated view of projects with finance, HR, and supply chain operations—all in one place

Typically, project members have very limited visibility into the project related information across other departments that can either positively or negatively impact their milestones. Project Analytics in combination with Fusion SCM (Supply Chain Management) Analytics provides teams ability to slice and dice supply chain data such as sales orders, procurement requisitions and orders, inventory transactions by project attributes.

Cross-Departmental and Project-Driven Supply Chain: With visually connected insights, project members now have an integrated view across the organization. This helps them not only recognize how to leverage resources but also deliver projects on time. The single, analytical, extensible data model provides integrated visualizations to quickly present correlations of project data across finance, HR, and supply chain operations. This enables members to find correlations in project-driven supply chains to reduce risk and control costs. They can recognize opportunities for next budget allocations and see how much additional labor is required to reach their budget. They can get ahead of any bottlenecks in supply chain operations that will hinder their outcomes. Using the extensibility feature, project members can easily bring together data from different sources or import external files with drag and drop integrations for a secured analysis:

◉ What are my open sales orders by project?

◉ Which orders have been currently shipped out and for which customers?

◉ Which are the current POs that are open by supplier and by project?

◉ How many hours of labor is yet to be processed in projects?

Project-driven Supply Chain Analysis

Source: oracle.com

Continue reading

6 Benefits of a Cloud Data Warehouse

Let’s explore these key topics one by one.

More Info: 1Z0-888: MySQL 5.7 Database Administrator

Question 1 - Do you even want a data warehouse in the cloud?

Of course you do! Look at how fast your data warehouse is growing. Look at the growing number of requests building up for new data warehouse projects, new data discovery sandboxes, new departmental marts, faster query response times, etc. Every IT department is looking for a silver bullet that can magically help them meet the growing demands for data access coming their business units. That silver bullet would be cloud.

Question 2: What can you expect from a cloud data warehouse and what are the key benefits? There are many, but we’ve identified the top six benefits for you.

Data Warehouse Cloud Benefit #1: Lower Costs With Elasticity

The biggest reason most people move to a data warehouse in the cloud is cost. Storing data on-premise, in your own data center, can get very expensive. And expanding your data footprint often makes it harder to support all of your ever-expanding analytical needs.

Why? Well with an on-premise data warehouse, you can’t independently scale compute and storage - at least not that quickly or easily. Typically, if you need more storage the compute will come with it and you end up having to pay for both.

In addition, you need to purchase as much compute as you need for peak times. So if you’re a retail company worried about how much compute you need to handle Black Friday, well, tough luck—you’re stuck with that much compute for the whole year.  

Fortunately, it doesn’t have to be that way. 

With the best kind of data warehouse, your system can instantly and flexibly scale to deliver as much or as little compute is necessary, whenever it is that you need it. And, because compute and storage are separate, you only need to purchase what’s essential. Lastly, you also don’t have as many upfront costs—hardware, server rooms, networking, adding extra staff, etc.

Data Warehouse Cloud Benefit #2: Quick to Deploy

In the past, IT teams had to estimate how much storage and compute power would be necessary for their line of business teams—sometimes three years in advance. Getting this information incorrect would mean buying hardware they didn’t need, or facing complaints if there was a lack of storage.

Today, this complicated, detailed planning-and-estimation process isn’t necessary. With the cloud, business users can build their own data warehouse, data mart, or sandbox in only minutes, at any time (night or day). Having a data warehouse in the cloud allows organizations to pay for only the resources they need—when they need it.

In addition, Oracle’s cloud makes it quicker and easier to roll out new data warehouse projects such as data discovery sandboxes. IT and business teams can develop and/or prototype new services and products without spending large sums of money on infrastructure.

Data Warehouse Cloud Benefit #3: Grow Your Capabilities

Having a data warehouse in the cloud improves the overall value of the data warehouse. It means that business intelligence and other applications can deliver faster, smarter insights to the business since the availability, scalability and performance are better.

As Penny Avril, VP of Product Management said: “The value of the business is driven by data, and by the usage of the data. For many companies, the data is the only real capital they have. Oracle is making it easier for the C-level to manage and use that data. That should help the bottom line.”

With a data warehouse in the cloud, you can engage in the full spectrum of data warehousing from business analytics, data integration, IoT, and more as a complete, integrated solution.

Data Warehouse Cloud Benefit #4: Self-Service Data Warehousing

Self-service is only truly possible if you have a self-driving database. Just as the cloud data warehouse has many benefits, a self-driving, autonomous data warehouse offers even more benefits. Essentially, you don't really have to worry about managing the data warehouse anymore. 

And that means you can benefit from fully automated management, fully automated patching, and upgrades. It means as business user, you don’t need IT to spin up a new data mart for you. You simply log into the cloud and provision a new data warehouse yourself, in minutes.

Data is more available and accessible than ever before.

This allows IT teams to focus attention and resources on more strategic aspects of providing value to the business. But this doesn’t mean that DBAs will be out of work—they still have to manage how applications connect to the data warehouse and how developers use the in-database features and functions within their application code.

Data Warehouse Cloud Benefit #5: More Secure Data

In the past, people were convinced that on-premises data warehouses were more secure. But in the same way that they now trust digital copies more than physical paper copies, some are beginning to see a data warehouse in the cloud as more secure than an on-premises system.

But obviously, it all depends on the database company. So choose a company that has a business model that relies on data security and encryption. Preferably, that company should have over four decades of experience with entire departments to protecting your most valuable asset ... Hmmm, who could that be? 

Just as an aside, with our self-driving database, the Autonomous Data Warehouse, we have strong data encryption switched on by default to ensure your data is fully protected.

Data Warehouse Cloud Benefit #6: The Cloud Itself

A self-driving database makes everything easier: it takes care of much of the dull but highly valuable work that most people don’t want to do. A self-driving database will help you gain even more ability and capability in the cloud.

For many customers, adopting a data warehouse is just one step on a multi-step journey. You need to make sure that your cloud provider offers a complete path to the cloud that encompasses integrated IaaS, PaaS, and SaaS solutions.

You can simplify your IT infrastructure and minimize capital investments by utilizing your cloud’s services for infrastructure, data management, applications, and business intelligence.

When it comes to choosing a cloud, make sure the one you pick allows for flexible deployment models, enabling you to seamlessly migrate your IT workloads from an on-premises data center to the cloud and back again.

Source: oracle.com

Continue reading

Analytics in Finance – Where do YOU stand?

“We have always done it that way”. “Our situation is unique”. Sound familiar? Many people in finance don’t have the time to question the process they follow month in and month out. Usually, we would say finance is all about maximizing revenue and profitability, balancing cash flow and spend, monitoring risk, and return on investments. However, plans have changed, expectations have reset, and 2020 may have even exposed some issues that need to be addressed.

Read More: 1Z0-082: Oracle Database Administration I

One of the keys to success is the ability for organizations to be agile and respond quickly as needed. As a matter of fact, organizations that invested more in digital technology than their peers during the crisis were “twice as likely to report revenue growth than executives at other companies.” (IDG global IT Leaders Research Report, 2020). At the very least, flexible solutions and analytics processes driven by the latest technologies contribute to that needed agility. So where does your Finance organization stand? Have you already modernized your analytics? Does your organization have what it needs to thrive?

We have prepared a short self-assessment tool that allows you to evaluate and take a snapshot of how you use analytics in finance. This assessment should serve as an easy way to get started gathering the proper information and understanding where you stand. Let’s first answer some background questions…

WHEN is the right time to modernize?

Clearly, we are going to say “NOW” is the right time to evaluate and decide on a modernization path. We understand that some businesses may be back in a growth mode while others are still planning their journey. Whether you start to modernize immediately or wait for several more months, preparing yourself and evaluating how modernization can impact your business is worth doing right away.

WHAT financial analytic processes should you be looking to evaluate? 

The answer to this question depends on your role. As mentioned, the self-assessment helps you evaluate how your organization uses analytics in finance. We present questions to help you evaluate and score the financial processes which are most important based on the role you select.

Do you already perform Scenario Modeling? What innovation techniques would you say your finance organization employs? What is most important to you: revenue analytics, cost analytics, or full profitability reporting? Have you undertaken digital transformation or are you still balancing older on-prem systems? This assessment provides quick and easy follow-up information within the areas you might need to improve.

Figure 1: You get to select your series of questions based on your role in finance

 

WHY should you be evaluating your company?

It always helps to take a step back and consider all of your daily tasks. Are they easy and worthwhile or difficult and bureaucratic? Could they be more effective? We want to provide that opportunity and at the same time give you some questions you may not have considered. As you go through the short assessment you will be asked a series of six questions: from data onboarding and preparation to how you handle enterprise analytic reporting or monitoring business performance.

Figure 2: An example of the assessment question and visual.

HOW do you get started? By taking the Analytics for Finance self-assessment.

The assessment is here. The answers are in your hands. Have at it! Take the assessment as many times as you would like or for different roles. The questions will be different based on role, and suggestions for additional information are provided each time.

Figure 3: Links to additional information

With the information provided, we know you will uncover new ideas and help your organization move forward. It may be as simple as improving a process, or as eventful as modernizing your financial software. Please reach out if you need specific follow up and let us know in the comments how you like it!

Source: oracle.com

Continue reading

What we found: Oracle Analytics COVID-19 analysis

This analysis was conducted in the context of the 2021 Gartner BI Bake-Off. All data used is publicly available, and insights highlighted result from ingesting, preparing, and analyzing that data. This does not represent the opinions of Oracle Corporation and should be used strictly as a demonstration of the Oracle Analytics product line. 

More Info: 1Z0-432: Oracle Real Application Clusters 12c Essentials

Every year Gartner invites analytics vendors to present at the Gartner Data & Analytics Summit to show how their products can help solve real-world problems and find insights into real-world data. Last year we looked at life expectancy. As one would expect, our focus this year is on the COVID-19 pandemic. During our analysis phase in March and April 2021, situations changed rapidly. Each day, new data brought new insights, and larger challenges emerged in real-time. 

◉ Questions about vaccine safety arose during April 2021

◉ There have been significant increases in disease incidence around the globe, especially in India, starting in the last few weeks of April 2021

◉ Government restrictions are loosening in key geographies as case numbers begin to fall and vaccine administration picks up steam while tightening in others, leading to a patchwork of guidelines within and across countries

We've drawn some conclusions—many that are reflected in what we hear from news sources, but also some that aren’t. Looking at this data in a week or a month may lead to different conclusions that we can’t yet anticipate. 

One thing is certain, though: This pandemic has shown gaps in response across and within countries and regions, with socioeconomic factors playing a big role in COVID-19’s relentless spread around the globe. 

The phrase “think globally, act locally” is appropriate to the analysis of data about COVID-19 and the vaccines that help combat it. Management of the pandemic has been handled very differently based on geographic, social, political, and economic factors. Here are some insights that we uncovered. 

◉ While vaccination rates are increasing throughout Europe, new case counts have been trending upward from February onward. As late as December 2020, it looked like the number of cases was trending lower. Vaccinations started in the UK in late December, but the rollout in the EU has been slower than in other geographies, leading Europe back into a COVID case-number growth curve.  

◉ North America is also showing an upward trajectory on COVID cases, but only starting in March 2021. In the US, the highest number of vaccinations given was on April 11. Two days later, safety concerns over one vaccine brand were raised, leading to a lower uptake of vaccines in the US—the vaccine rate has not recovered to the same level in the US since that date.

◉ All geographies are showing an uptick in cases, even as the number of vaccinations climbs. While vaccinations are often viewed as the panacea, the spread of the disease is outpacing the ability to get shots in arms. 

◉ High-income and upper-middle-income countries are far ahead in vaccine doses delivered, which is no surprise. But absolute numbers of vaccinations delivered isn’t the key factor in slowing the infection rate. It’s the percentage of the total population getting fully vaccinated—that’s the difference. For example, about 30% of the US is fully vaccinated as of the end of April 2021, whereas, while India has a large number of vaccines administered, only about 3.1% of the population is fully vaccinated. 

◉ Just twelve countries make up more than 50% of the vaccines given based on vaccines per million measures.    

◉ Governments issued thousands of mandates and actions in an attempt to control the spread of COVID-19. Full lockdown, domestic and international travel restrictions and social distancing (including school closures) in aggregate had the biggest influence on the spread of the disease across many countries in EMEA and South America. Responses to government actions were different by country. 

These are just some of the highlights we saw, but this is by no means an exhaustive list. 

◉ Belgium and the UK saw a greater positive impact from lockdown, Argentina and France less so. The first question that comes to mind is “why?” There’s nothing in the data we could find to support different responses by country. But citizens of those countries often point to anecdotal cultural attitudes that may or may not validate the relative impact of lockdowns. 

◉ Social distancing guidelines were not effective in France but were far more effective in Israel.

◉ US results overall weren’t promising in any action as mandates didn’t have desired impacts, especially in the second half of 2020. While we did some drilling down into different states and how they responded, there were markedly different responses based on states and regions. As mandates were set by both states and the federal government, aggregating at the US level did not give clear conclusions. 

◉ The conclusion we draw from all this:  One-size mandates don’t fit all countries and regions.  What works in London may not work in Lyon.  What works in the northeast USA may not work in the southwest USA.  Mandates must be tried and tailored to the geography and the situation on the ground to be effective. 

Tweets about COVID and vaccines are largely neutral to positive. The larger the reach (how many people may be influenced), the more neutral they are. When analyzing Twitter data about specific vaccine brands, there is some variability in tone (positive, neutral, negative) but not enough to say if it’s impacting vaccine hesitancy. As we enter the next phase of vaccinations worldwide, a “charm campaign” may be needed to get past vaccine hesitancy. Tweets from influencers with large reaches may be one avenue to help sway people to get vaccinated. 

Source: oracle.com

Continue reading

Oracle Exadata Cloud vs. AWS and Spark for big data analytics

This blog takes a closer look at why the Moat Reach team moved their data warehouse from Spark on AWS to Exadata on OCI.

What is Moat Reach?

Moat Reach, by Oracle Data Cloud, is a cross-platform TV and digital measurement solution that measures how many people and households your campaigns reach, at what frequency, and how relevant those people are to your marketing objectives.

It integrates Moat Analytics’ digital impression data with TV advertisement impression data from iSpot.tv, which uses 14 million opted-in TV devices, against the people and households in the Oracle ID Graph. This data allows marketers to measure valid and viewable impressions for the audiences that matter to them across TV and digital channels.

In marketing terminology, an impression refers to the number of times an advertisement was seen when browsing the internet on a web browser, watching a sports game on TV, playing a game in an app, and so on. Examples of marketing channels include browsers, apps, traditional cable TV, and streaming video.

For example, a consumer packaged goods (CPG) advertiser can use Moat Reach to measure the unique, people-based reach of its new soft drink campaign using the same metrics across all of its TV and digital channels. Even better, it can also measure its audience based on custom or first-party segments, like soft drink buyers or frequent shoppers, and analyzes key demographics, like age and gender, through each channel.

By comparing the reach, frequency, and viewability of its ads through each channel, the CPG advertiser might discover that it was reaching more of its intended audience through celebrity magazine websites than DIY cable channels. It can then adjust its media strategy to invest more in those channels.

After years of cross-channel confusion, Moat Reach gives marketers a self-service tool to measure the consolidated reach and frequency of their campaigns, so they can see whether they reached their intended audiences through each platform, publisher, and network on a consolidated and de-duplicated basis.

Version 1: Apache Spark on Amazon Web Services (AWS)

Moat Reach answers the impact of the marketing campaign on an individual across TV and digital media. To do that, it counts distinct records for that individual in the data set from Moat and iSpot.tv.

The Moat team developed their first data warehouse solution using Apache Spark, Qubole, and Scala on AWS infrastructure. This solution operated on a dataset with about 97 billion impressions from Moat and 49 billion impressions from iSpot.tv.

The dataset in this first version architecture spanned 34 clients, 275 thousand campaigns, and two years of data. The Moat team loaded this data from object storage into the Apache Spark cluster and ran analytics using Qubole and Scala. This data was precomputed into dimensions to assist with query processing.

Although this method has limitations, other analytics companies in the industry use similar technologies and the same methodology of precomputing the dimensions because it was too hard a problem to solve doing it in real time.

Getting answers out of this v1 data warehousing solution took roughly seven days from when a user asked the question to the return of a response. This solution also costs a lot of money, with each nightly run costing about $5–8K USD, $100–160K USD for the monthly runs, and about $1.2–2.8M USD annually.

Moat Reach v1 relied on a snapshot model where the Moat team ingested impression event, id-graph, demographic, and audience data into a set of parquet tables on AWS S3. They then ran several large Spark jobs to populate reach and frequency metrics in a web UI. The metrics were stored in a database by the nightly job.

The Moat team wrote the job to quickly return user requests without computing any more information. This approach allowed them a responsive site but at the cost of only computing a finite set of metrics.

This lack of flexibility frustrated customers. For example, they computed the number of 18–30 year-olds who saw an ad, the number of women who saw an ad, how many mobile views, or five-second views an ad received. But changing the range of women to 18–29 year olds would have taken at least a day.

A cross-section of 18-year-old women who saw the ad for five seconds on their mobile phones with all other cross-sections was similarly financially infeasible. It would be too expensive to run a job every night for every value of every dimension across every client and campaign. The Moat team set a goal to gain insights into and across all dimensions of the Moat Reach data.

Version 2: Oracle Database on Oracle Cloud Infrastructure and Exadata

While the v1 Moat Reach service was already differentiated in the market because of its uniquely consolidated insights, the team felt there was an opportunity to be more comprehensive and offer more customized insights for customers in near real time. The Moat team faced the challenge of finding technology with enough storage to handle terabytes of data—and over a petabyte a year—and make much faster queries than the existing architecture.

Oracle Cloud Infrastructure (OCI) is the only provider to offer Oracle Database and Oracle Exadata as a service. This offering has vast amounts of space and equally impressive compute capabilities. In the on-demand architecture, they only had to ingest impression event, id-graph, demographic, and audience data into their Exadata appliance, and then let an API do the rest. When users visit the site, an API queries the database for counts, based on the selected filters. Data science methodology is applied to estimate total reach and frequency from the observed data. This architecture extends the product use case from broad, top-line insights to precise and deeply detailed metrics that clients can use to optimize their ad campaigns’ reach in real-time.

To query billions of rows quickly, they used Exadata’s probabilistic data structures and hyperloglog estimation functions to obtain bounded estimates of the needed inputs for their data science methodology. To further optimize performance, they took advantage of Exadata’s tuning abilities and support team. By working with Oracle engineering staff, they tuned Exadata to be more performant than its competitors AWS Redshift and Clickhouse to run a baseline set of queries meant to mimic the live production system requirements. In-house Oracle expertise and the wide availability of Oracle Database and SQL skill sets were other factors to encourage them toward OCI and Exadata.

The following table compares the two approaches:

V1 solution	V2 solution
Warehouse	On-demand
Spark and AWS	Exadata on OCI
Precomputed dimensions for reporting	Flexible dimensions for reporting
Customer answers in days	Custom answers in seconds
Limited reports	Unlimited number of custom reports
Long development cycles	Rapid development cycles
Delayed innovation turnaround	Real-time access for Innovation
$$$	$$

While improved customer experience was the Moat team’s top priority, cost was also a factor in their decision to use Exadata over the V1 architecture and AWS’s data warehouse, Redshift. An AWS Redshift instance cost roughly 2 million dollars for a baseline of 912 CPUs and 940 TB of storage. For the same cost, Exadata on OCI offered much faster performance and real-time data queries with an optimized database shape with 150 CPUs, 23 TB of RAM, persistent memory for an in-memory database, and close to 1 PB of flash and disk storage for historical data. Exadata features like smart scan, storage indexes, and smart flash cache were some of the factors for the faster query performance.

V1 solution	Alternative	V2 solution
Warehouse	Warehouse	On-demand
Spark and AWS	AWS Redshift	Exadata on OCI
Customer answers in days	Customer answers in hours	Customer answers in seconds
Limited reports	Less limited reports	Unlimited number of custom reports
$$$	$$	$$

Source: oracle.com

Continue reading