Big data and predictive analytics

-
Today ‘big data’ has become a buzz word. Everyone is talking about it. Big data are characterized by three attributes, which are high volume, high velocity and high variety. A few researchers also recently proposed "high volatility" as the fourth attribute of big data. We can define big data as the collection of data which are so large, complex and ever growing that they cannot be processed and stored using traditional methods. The size of big data is greater than petabytes. This makes storage of big data very difficult. Examples of the big data are web data, telecom data, sensor data of jet engines, RNA-DNA data etc.

The challenges in processing big data led to the development of new technologies such as Hodoop and Map Reduce. People claim that big data can be processed in reasonable time using these technologies. But I have yet to use the above technologies. Hence I cannot judge the efficiency of these technologies.

When we deal with any data, then we come across two kinds of professionals. The first kind is the database architect and infrastructure developer. He/She designs and models the database to store the data. They also write programs to extract the raw data and convert the data into usable format. To acquire expertise in this field, a person can enroll in courses such as database engineering and design, computer networking etc. To learn about a specific database such as Oracle, Db, Red shift etc, a person can opt for a certificate program offered by many international organizations.

The second kind of the professionals are data scientists or analysts. They use data to derive useful information for research, business or governance. Earlier business leaders made decisions based on their convictions, but now they also try to verify their hypothesizes using the data. Professionals who are experts in processing data and studying them to get information out of it are called data scientists. They also develop new methods of data mining and learning algorithms.  Data scientists use several statistical algorithms to derive underlying information. These algorithms are also called machine learning algorithms.

Data driven decisions are made mostly in the banking, telecom, retail, and pharmaceutical and internet industry. In business, data are used to derive information for marketing, customer retention and growth, credit history assessment and clinical trials. Data analysis helps a business leader to make decisions which deliver their business goals with minimum costs. 


Comments

Post a Comment

Popular posts from this blog

How to check whether a SAS dataset exist or not and throw an error in the log ?

-
/* if the data set exists, then conditionally execute the step(s). %sysfunc checks the availability of the data in the library location. It means it checks historical sas data. It generates the system error if the not is data available at the defined library  */ %let dsname=&dataval.MKT_DATA;  /**** declaring data set name with library*****/ %macro warning1(name); %if %sysfunc(exist(&name)) %then %do; Proc printto log="&logfile.\WarningLog..txt" new;     /*** printing log in the text file at specifile location &logfile****/ run; %Put WARNING :- &name does not exists; Proc printto  ; Run; %end; %else %do; Proc printto log="&logfile.\WarningLog.txt" ; run; %put &name exists; Proc printto  ; Run; %end; %mend ; %warning1(&dsname); /**** Pros:  If any sas process is dependent on the existence of the data then we can use this macro***/ /**** Cons:  *Even if the SAS data set is a...
Read more

Solution for ERROR: Some character data was lost during transcoding in the dataset

-
When I search for the mentioned error I get the link solution provided administration oriented but developers do not have access.     http://support.sas.com/kb/52/716.html Please see my solution below proc options option=config; run; proc options group=languagecontrol; run; /* Show the encoding value for the problematic data set */ %let dsn=item_information_may16; %let dsid=%sysfunc(open(&dsn,i)); %put &dsn ENCODING is: %sysfunc(attrc(&dsid,encoding)); /*Renaming item desc file  (encoding=any) allowed reading****************************/ data tmp.item_info_curr; set hc.item_information_may16 (encoding=any); run; /*gave error*/ data item_info_curr; set item_information_may16; run;
Read more

2018 plan for getting expertise in Machine Learning and Deep Learning

-
Machine Learning and deep learning are next frontier in the world of innovation. These skill sets are high in demand and demand is going to increase further as we are moving towards a world with connected systems. Although I am experienced professional with SAS having multiple SAS certifications and  using SAS for programming, predictive model building, optimization and data visualization for more than seven years, I will be foolish I do not recognize how important it is to adopt open source platforms for innovation.  2018 plan to acquire expertise in these are are 1. Choose a programming language and I have chose python as my language of preference  2. Do hands on practice to be efficient in Pandas and Numpy. These two libraries are useful and very important for data exploration,wrangling and data cleaning. I find Kernals in Kaggle and below blogs and books to be helpful tutorial for  Pandas  a. https://www.dataquest.io/blog/pandas-tutorial...
Read more