 |
| Data Warehousing |
Data Warehousing Object/Vision: Data Warehousing is the same term as material warehousing, where we are storing the information in the form of formatted and labeled data. The objective or vision is to make a data warehouse to store big data from all sources of information, and that can be accessed from anywhere with full security. This can centralize all information of an organization with multiple layers of security.
Data
Data is information in an encrypted or decrypted form, it can be gathered from any source, manual, mobile, website, campaigns, transactions, third party, etc. Data is power, who has as much big size of data, that has that much power, but if you don't know how to use your data, then that is just garbage for you, to make that data meaningful, you need to make that in a structured and relational form.
World's giant companies are running on data, e.g. Google, Amazon, Yahoo, etc.
Sources and Destination
The source is the place or thing from where you are collecting or getting the data, and the destination is the place where you are saving the data, for example, if you are running a Facebook survey campaign then your source of data collection is Facebook, internet, cache memory, mobile devices, etc. and the destination can be the place where you are receiving and saving the data or information.
On-Premise, On Cloud
The source to collect the data or destination to store the data can be on-premise or cloud-based, which means either you can store the data in a location space or you can store the data on a server or virtual space. Both options have pros and cons if you will know that in detail. But for now, let me tell you on-premise database is more secure and fast while cloud-based data is accessible from anywhere in the world, on-premise required more budget and space, while cloud-based doesn't require that much space and budget.
Data Lake
A destination of data from multiple sources of data but in a very unstructured and casual format, In the data lake, data size can be huge, from GBs to TBs, or more. It can be cloud-based or noncloudy, but usually, we find that cloud-based. Data can be collected from anywhere and it can be fed somewhere and then it can be centralized somewhere but becomes a data lake, but it is hard to understand and identify due to its unstructured form.
Cleaning
Need to clean the big data, it brings so much junk and useless values with it, especially when the data was in an encrypted form, or while manual or automated data feeding. Data Cleaning makes data fast, and clear to understand so that we can set the rule of calculations on the schema for the best outcome.
Labeling
Data labeling is a kind of data naming, where you are giving some identities to some data or relational data. With the data labeling, it is easy to identify the data with some direct or code names.
With the data labeling, you can proceed the process of data warehousing.
RDBMS
Relational Database Management System is mandatory to connect multiple data tables using various joining, with different methods of joining, usually, we find a primary key or come corelative things to connect tables.
Data Warehousing
After all the above processes we centralize the cleaned data in a structured and labeled form with different formats, algorithms, and formulas. It can be present in different visuals and structures depending on the module.
Centralization
With the data warehouse, we can make a central database where you can save the big data at a central point collected from various sources. The data can be accessed from anywhere and it can be used for a big purpose because all data are integrated with each other.
e.g. In a factory, there are many functions and departments, knowing about the raw material you can make changes in the production's data, and that data can be co-related to procurement, sales, and finance.
After making a data warehouse how does data store in the data warehouse -
ETL
Extract, Transform, and Load (ETL) is the step for data feeding in a data warehouse, firstly it extracts the data from any source, then it gets cleaned and labeled, and transform according to the objects of data warehouse schemas, there can be many things, like calculation, parsing, mapping, etc., and it gets load into the data warehouse.
Replication
Then the data replicate in the data warehouse according to the schema and calculations, replicated data can be the information from data sources, and it can be replicated, manually or with scheduling.
Automation Chain
The ETL and Replication can be automated in the data warehouse by making any process chain for it, with the automation it will be one-time activity until you want any change in the schema or formulas or mapping, it will work automatically, just keep collecting the information and keep using that, and become strongest one.
Data Warehouse Architecture
|
A |
B |
C |
D |
|
CRM, Website, CSV, Flat Files, Campaigns, Manual, Mobile, Computer, other devices, |
ETL |
Data Warehouse |
OLTP, OLAP, BI/BO |
Data Warehousing Applications (Company/Functions)
Data Warehousing can be beneficial for multiple applications or functions in every industry.
e.g. Operations, Management and Business, C-Suite, Procurement, Inventory, Supply Chain, Purchase, Sales and Marketing, Human Resources, Finance, Accounts, Education, Medical & Healthcare, Information Technology, Engineering and Technical, Design, Legal, Ground Level,, Customer Support
Manufacturing, Media and PR, Administration, Business Development, Consulting, Quality Assurance, etc.
Cost
The exact cost can depend on the size of space, implementation, the complexity of data, efforts, and time, but to understand the cost, we can say there are two costs -
One-time cost
This is the implementation cost for the data warehouse solution which can be fixed and paid once (in multiple delivery milestones) you finalize the scope of the work, so based on efforts, time, complexity, resource, quality, etc., it can be finalized and fixed.
Recurring Cost
This cost you have to pay on a rental or usage basis, it can be monthly, quarterly, or most probably annual. Depends on how much space you are acquiring, how many transactions are happening, what technology you are choosing, etc.
Technologies
There are many technologies in the market that allow you to create your own data warehouse for your efficient business process.
e.g. Azure, AWS, SAP, Oracle, Redshift, Snowflake, BigQuery, Db2, Firebolt, Hadoop, and Many Open-source, etc.
Data Warehouse Impact
Time Management: With good data warehousing you can easily manage the time in any department and industry, which can help to make the process better and smooth.
Information Management: With the data warehouse solution the organization can manage all small and big information in a proper manner and with protocols.
Great Progress: With better time management you can improve the progress results for any work or project, this can be a great impact on any department.
Cost Saving: It can save a lot of money to invest or waste, which makes your organization more profitable.
Stabled System: A data warehouse can balance the things in the organization, so it can manage the stability of any organization or function.
Teamwork: It is always easy to do amazing teamwork as well as team collaboration with the data warehouse solution.
Professionalism: With great management, teamwork, stability, and progress, the organization can be more profitable and professional to do work and follow the protocols and instructions.
Work Efficiency: A data warehouse can increase work efficiency for any department or industry. That is the biggest impact of data warehousing on any organization.
ROI: There are higher chances to get the Return On the Investment in the earliest stage of any process because here you will be more focused and well-managed.
Financial Profit: A data warehouse can be a reason for a profitable or well-established organization because they know where they are good and where they are lacking.
Organizational Growth: When you are in the profit and you will understand the pros and cons, you know what is overstock and what is the shorted, you know where you are standing, then it is easy to get the right path and speed for the growth, a data warehousing helps in the organizational behavior and growth.
Business Scalability: You know where you are best, what is your next step, and which is the best path for you, and you are already growing, now it is easy for you to extend your business or scale up the business.
Reporting - BO/BI
This is an advanced level reporting by business intelligence and business objective. How Business Intelligence can help you?
Reports: From the data warehouse or from the data sources you can get the aggregated report to make decisions for your departments or organization to get some conclusion, there are two ways to get the reports to know the insights -
OLAP - Online Analytics Processing - Where you get some descriptive report based on historical data, it helps to make big decisions.
OLTP - Online Transaction Processing - Where you get real-time reports based on the current transactions, it can help you to make instant decisions.
Dashboards - With the dashboards, you can get visualization with deep insights from the data, there you can get KPIs (Key Performance Indicators) to make some decisions or get satisfied.
