Big data, big analytics, big opportunity

Published Jul 25, 2012 04:03pm

Data, data, every where And not a byte to think

The world today is awash with data. Corporations, governments, and individuals are busy generating petabytes of data on culture, economy, environment, religion, and society.  While data has become abundant and ubiquitous, data analysts needed to turn raw data into knowledge are in fact in short supply.

With big data comes big opportunity for the educated middle class in the developing world where an army of data scientists can be trained to support the offshoring of analytics from the western countries where such needs are unlikely to be filled from the locally available talent.

In a 2011 report, McKinsey Global Institute revealed that the United States alone faces a shortage of almost 200,000 data analysts. The American economy requires an additional 1.5 million managers proficient in decision-making based on insights gained from the analysis of large data sets. And even when Hal Varian, Google's famed chief economist, profoundly proclaimed that "the real sexy job in 2010s is to be a statistician," there were not many takers for the opportunity in the West where students pursuing degrees in statistics, engineering, and other empirical fields are small in number and are often visa students from abroad.

A recent report by Statistics Canada revealed that two-thirds of those who graduated with a PhD in engineering from a Canadian University in 2005 spoke neither English nor French as mother tongue. Similarly, four out of 10 PhD graduates in computers, mathematics, and physical sciences did not speak a western language as mother tongue. Also, more than 60 per cent of engineering graduates were visible minorities, suggesting that the supply chain of highly qualified professional talent in Canada, and to a large extent in North America, is already linked to the talent emigrating from China, Egypt, India, Iran, and Pakistan.

The abundance of data and the scarcity of analysts present a unique opportunity for developing countries, which have an abundant supply of highly numerate youth who could be trained and mobilised en masse to write a new chapter in offshoring. This would require a serious rethink for thought leaders in developing countries who have not taxed their imaginations beyond thinking of policies to create sweat shops where youth would undersell their skills and see their potential wilt away while creating undergarments for consumers in the west. The fate of the youth in developing countries need not be restricted to stitching underwear or making cold calls from offshored call-centres in order for them to be part of the global value chains. Instead, they can be trained as skilled number-crunchers who would add value to otherwise worthless data for businesses, big and small.

A multi-billion dollar industry

The past decade has witnessed a major change in the sectorial evolution of some very large manufacturing firms known in the past for mostly hardware engineering and now evolving into firms delivering services, such as business analytics. Take IBM for example, which specialised as a computer hardware company producing servers, desktop computers, laptops, and other supporting infrastructure. That was IBM's past. Today, IBM is focussed on analytics. It is spending hundreds of millions of dollars in advertising, trying to rebrand itself as a leader in business analytics. In fact, it has divested from several hardware initiatives, such as manufacturing laptops, and has instead spent billions in acquisitions to build its analytic credentials. For instance, IBM has acquired SPSS for over a billion dollars to capture the retail side of the Business analytics market. For large commercial ventures, IBM acquired Cognos to offer full service analytics.

In 2011 alone, the business analytics software market was worth over $30 billion. Oracle ($6.1bn), SAP ($4.6 bn), IBM ($4.4 bn), and Microsoft and SAS each with $3.3 bn in sales led the market. It is estimated that the sale of business analytics software alone will hit $50 billion by 2016.  Dan Vesset of IDC, a company specialising in watching industry trends, aptly noted that business analytics had "crossed the chasm into the mainstream mass market" and the "demand for business analytics solutions is exposing the previously minor issue of the shortage of highly skilled IT and analytics staff."

In addition to the bundled software and service sales offered by the likes of Oracle and IBM, business analytics services in the consulting domain generated several billion dollars more worldwide. While the large firms command the lion’s share in the analytics market, the billions left at the bottom are still a large enough prize to take the analytics plunge.

Several billion reasons to hop on the analytics bandwagon

While the IBMs of the world are focused largely on large corporations, the analytics needs for small and medium-sized enterprises (SMEs) are unlikely to be met by IBM, Oracle, or other large players. Cost is the most important determinant. SMEs prefer to have analytics done on the cheap while the overheads of the large analytics firms run into millions of dollars thus pricing them out of the SME market. With offshoring comes the access to affordable talent in developing countries who can bid for smaller contracts and beat the competition in the West on price, and over time on quality as well.

The trick therefore, is to beat the IBMs of the world in the analytics game by not competing against them. Realising that business analytics is not a market, but an amalgamation of several types of markets focused on delivering value-added services involving data capture, data warehousing, data cleaning, data mining, and data analysis, developing countries can carve out a niche for themselves by focusing exclusively on contracts that large firms will not bid for because of their intrinsic large overheads.

Leaving the fight for top dollars in analytics to top dogs, a cottage industry in analytics could be developed in the developing countries that may strive to serve the analytics need of SMEs. Take the example of the Toronto Transit Commission (TTC), Canada's largest public transit agency with annual revenues exceeding a billion dollars. When TTC needed to have a large database of almost a half million commuter complaints analysed, it turned to Ryerson University, rather than a large analytics firm. TTC's decision to work with Ryerson University was motivated by two considerations. First the cost; as a public sector university, Ryerson believes strongly in serving the community and thus offered the services for gratis. The second reason is quality. Ryerson University, like most similar institutions of higher learning, excels in analytics where several faculty members work at the cutting edge of analytics and are more than willing to apply their skills to real life problems.

Why now?

The timing had never been better to undertake such an endeavour on a very large scale. The innovations in Information and Communication Technology (ICT) and the ready availability of the most advanced analytics software as freeware allows entrepreneurs in developing countries to compete worldwide. The Internet makes it possible to be part of global marketplaces with negligible costs. With cyber marketplaces such as Kijiji and Craigslist individuals can become proprietors offering services worldwide.

Using the freely available Google Sites, one can have a business website online immediately at no cost. Google Docs, another free service from Google, allows one to have a web server for free to share documents with collaborators or the rest of the world for free. Other free services, such as Google Trends, allow individual researchers to generate data on business and social trends without needing subscriptions for services that cost millions. The graph below is generated using Google trends showing daily visits to the websites of leading analytics firms. Without free access to such services, access to the data used to generate the same graph would carry a huge price tag.

Similarly, another free service from Google allows one to determine, for instance, which cities registered the highest number of search requests for ‘business analytics’. It appears that four of the top six cities where analytics are most popular are located in India, which is evident from the following graph where search intensity is mapped on a normalised index of 0 to 100.

The other big development of recent times is freeware that is levelling the playing field between haves and have-nots. In analytics, one of the most sophisticated computing platforms is R, which is available for free. Developers worldwide are busy developing the R platform, which now offers over 3,000 packages for free for analysing data. From econometrics to operations research, R is fast becoming the lingua franca for computing. R has evolved from being popular just amongst computing geeks to having its praise sung by the New York Times.

R has also made some new friends, especially Paul Butler, a Canadian student who became a worldwide sensation by mapping the geography of Facebook. While being an intern at Facebook, Paul analysed gigabytes of data to plot how Facebook’s friends were linked globally. His map (see the image below) became an instant hit worldwide and has been reproduced in publications thousands of times. If you are wondering what software Paul used to generate the map, wonder no more, the answer is R.

R is fast becoming the preferred computing platform for data scientists worldwide. For decades the data analysis market was ruled by the likes of SAS, SPSS, Stata and other similar players. R has taken over the imagination of data analysts as of late who are fast converging to R. In fact, most innovations in statistics are first coded in R so that the algorithms become available to all immediately and for free.

Source: http://r4stats.com/articles/popularity/

The fact R is freely available should not be taken lightly. A commercial license of a similarly equipped version of SPSS may cost up to US$7,500. The other big advantage of using R is the fact that thousands of training documents on the Internet and videos on YouTube are also available for free by volunteers.

Where to next

The private sector has to take the lead for business analytics to take root in developing countries. The governments could also have a small role in regulation. However, the analytics revolution has to take place not because of the public sector, but in spite of it. Even public sector universities in developing countries cannot be entrusted with the task where senior university administers do not warm up to innovative ideas unless they involve a junket in Europe or North America. At the same time the faculty in public sector universities in developing countries is often unwilling to try new technologies.

The private sector in developing countries may want to launch first an industry group that takes upon the task of certifying firms and individuals interested in analytics for quality, reliability, and ethical and professional competencies. This will help build confidence around national brands. Without such certification, foreign clients will be apprehensive to share their proprietary data with individuals hidden behind computer monitors thousands of miles away.

The private sector will also have to take the lead in training a professional workforce in analytics. Several companies train their employees in the latest technology and then market their skills to clients. The training houses would therefore also double as consulting practices where the best graduates may be retained as consultants.

Small virtual marketplaces could be setup in large cities where clients can put requests for proposals and pre-screened, qualified bidders can compete for the contract. The national self-regulating body will be responsible for screening qualified bidders from its vendor-of-record database, which it would make available to clients globally through the Internet.

The IBMs of the world see the analytics market to hit hundreds of billions in revenue in the next decade. The abundant talent in developing countries can be polished into a skilled workforce to tap into the analytics market to channel some revenue to developing countries while creating gainful employment opportunities for the educated youth who have been reduced to making cold calls from offshored call centres.

 


Murtaza Haider, Ph.D. is the Associate Dean of research and graduate programs at the Ted Rogers School of Management at Ryerson University in Toronto. He can be reached by email at murtaza.haider@ryerson.ca

 


The views expressed by this blogger and in the following reader comments do not necessarily reflect the views and policies of the Dawn Media Group.


Murtaza Haider is a Toronto-based academic and the director of Regionomics.com.


The views expressed by this blogger and in the following reader comments do not necessarily reflect the views and policies of the Dawn Media Group.

More From This Author

Comments (51) (Closed)


liver,brother
Jul 27, 2012 07:15am
Dear Mr.Raza, Thanks for sharing knowledge.
ali shah
Jul 27, 2012 07:19am
great article.....wonderful read......have forwarded to some friends already
Murtaza Haider
Jul 27, 2012 07:16am
What do you think my day job as a professor in Canada entails?
Kumar
Jul 25, 2012 02:32pm
Really nice article. Good Business Opportunity for developing worlds including Pakistan.
Mandeep
Jul 26, 2012 12:35pm
Very informative and well presented. This is kind of information that is missing from media outlets. We have an overdose of politics, religion and current affairs everywhere. Very refreshing to read, Mr Murtaza.
chakraborty
Jul 27, 2012 11:00am
I think you have never names of S.N Bose (On whose name Boson Particle is named), J.C Bose (The man who proved Plants have life & Patented wireless communication), Meghnad Saha (Known for Saha Equation), CV Raman (Known for Raman Effect_Nobel winner), Har Gobind Khorana (Nobel winner) I have just named a few from scientific community who are from pre-partition/ Born much before partition era . If you want I can name numerous people from every field where Indians have excelled. So dont make such statements. You are insulting yourself
@smohkim
Jul 26, 2012 09:23am
Who knocked some sense here? I hope and expect a lot more written on the same. Analytics is a very serious business and the most essential item to base your progress upon. I think if we can have more of this talked about by the experts (and i mean real Experts) to show the value it has to offer. The way around for this should and must come from the Private Sector as relying or hoping for the Public Sector Participation would only ensure an absolute failure. Off topic, but my last conversation with an industry expert mentioned the lack of exploring the same sector (along with a few others). Good read.
BRR
Jul 25, 2012 02:50pm
An extremely informative, revealing article, which contains a lot of information someone like me with 25 years in IT knows about, but few in the developing world are aware of. Great article, good job educating the masses.
_Jamal_
Jul 27, 2012 07:13am
Slightly off-topic but looking at that FaceBook map shows no users in KP, FATA or Balochistan, shame there is such a huge intensification of users in Punjab and Sindh. Here in UK offices FaceBook is banned, blocked by the FireWall, at one company where I used to work when such measures were bought in, we had a mini-revolt from some workers, FaceBook is a productivity destroyer and should be avoided at all costs.
Apoorv
Jul 27, 2012 06:48am
Aamir, Can you enlighten me from where did you get this information that Pakistan was ahead of India in Science and Tech some 30 years back? PS - Please provide me sources links or book or articles wherever possible.
Definitely
Jul 25, 2012 02:01pm
nice to see a fairly technological and computer oriented piece on Dawn.
p1845
Jul 26, 2012 12:04pm
A warning.. The material in there is not for the light heart or useful on day-1. However it is great starter to know, where to start learning. There is so much one has to know before actually being able to claim oneself even a primitive analyst. Happy to run into this article. Please also check out cloud computing that enable big-data analytics on hundreds of computers spread over the net. (Amazon EC2). Enough tuition for free : -) My thoughts are: let us stop blaming the past, countries, ministers, terrorists, adversaries and just do some thing for ourselves and that in turn will be for the society and others around us. Good luck to anyone who actually ventures into this and pls do not forget to thank the other after you make your first billion. ..Best
p1845
Jul 26, 2012 11:52am
Folk, A free (and hopefully useful) advice. People interested in large data analysis should check out: Wiki Big Data and Wiki Hadoop (Apache open source platform) A great starter for people wanting to learn. Royalties welcome : -) ..Best
Faran Ali
Jul 26, 2012 11:15am
quite well written. good job, I hope the youth of pakistan improves upon this.
chakraborty
Jul 27, 2012 01:01pm
I think you have never " heard" ... sorry
inam
Jul 27, 2012 07:08am
Well written :) Can any one tell me the name of some firms that are already working on Analytics except TD and IBM in islamabad. Thank you :)
Muhd. Sunil Bokhari
Jul 27, 2012 12:34am
Big data has become a trendy thing in the IT world, however, things are picking up in Pakistan but at a very low pace specially because of the state of our economy. Just to give a glimpse, you can view this IBM case study on Sui Southern Gas Company (SSGC) which has implemented Cognos among whole other things ( I can say this because I was involved). But because of lack of projects, the skill pool is very limited and that acts as a deterrent for the companies to buy these expensive products as well as for IT professional to get them trained. One more thing, you won't be targeting Big Data through products like IBM Cognos or Oracle BI or SAP Business Objects, but products similar to for example Oracle Endeca which work on unstructured data, because most of the Big Data is unstructured ! Regards, Bokhari
Enigma
Jul 27, 2012 04:39am
Nice to see an article on technology in Dawn
Mehdi Maloof
Jul 27, 2012 04:40am
This is a very biased Article. QlikView is obviously the best analytics tool ever made but it is not mentioned here. What a shame.
Usman
Jul 27, 2012 02:56pm
Good work Murtaza. I would like to meet you and discuss more when i come to Toronto as i am also a Ryerson alumni.
Adnan
Jul 30, 2012 09:31am
Disagree with the baseless rant of Aamir. The problem with Pakistan is the mentality like Aamir who just throw blame on others and do nothing themselves. Zia died 20+ years ago, a youth born and developed in due course which faced the secular extremism of PPP and Musharraf but nothing substantial achieved in due course. Due to mindset like Aamir Pakistan is in dark ages.
uthmanmalik
Jul 25, 2012 03:47pm
I am glad that you wrote about it. I just moved back to Pakistan to work on it here at a University. And since you have mentioned it, probably I should get in touch with you in a few months time to discuss a Phd thesis :)
Rishad Shaikh
Jul 25, 2012 02:08pm
Analytics is a win-win, Intellectually as well as economically, and its really sexy. Thanks for writing such a well researched and relevant article and presenting it as it is- The Next Big Thing.
nobody
Jul 25, 2012 05:34pm
Murtaza should look for the interest of Canada first not developing countries at the expense of Canada. You live here, do something for Canada first, guide the youth here.
Rashid
Jul 25, 2012 05:52pm
Very informative article. Just want to add one thing here, readers who read this article and motivated enough to jump into the bigdata market should also look into Hadoop and MapReduce. Good luck friends.
kamran
Jul 25, 2012 06:02pm
This article would be more apt for publication in a scientific journal rather than a newsapaper.
Goga Pagla
Jul 25, 2012 06:20pm
Wow!! Great article and more importantly I am impressed that Dawn would carry such a piece.
Ash
Jul 25, 2012 06:58pm
Murtaza - Can you please shorten the length of your articles and stop giving so many links in bold.
Naseema Perveen
Jul 26, 2012 07:58am
excellent one...found something worthy to go through!
Falcon
Jul 30, 2012 02:46am
Murtaza Sahab - Thanks a lot for such a brilliant ad well articulated research. After reading your article, I am really interested in learning more about business analytics.
Falcon
Jul 30, 2012 02:49am
@nobody: An enlightened person is first committed to humanity and then the individual countries. If that was not the case, humanity won't have not gotten anywhere. If this article can help bring bread to even one more poor person's home in any part of the world, that would be the true benchmark of success.
Rayhan Abdulmughnee
Jul 25, 2012 08:49pm
Agree with your view on the importance of data analysis, especially when it comes to SMEs (Small and Medium Size Businesses). Those who are interested in getting into data analysis as a business opportunity should look into Google Analytics. A free, but very powerful tool from Google. So powerful in fact that most business owners are not able to fully appreciate its power. Is there a market for entrepreneurs to help SME understand the data from the Google Analytics? I think so. If you can analyze the data and write a jargon-free report that can help entrepreneurs run their business better, I believe you can find a niche that can be profitable.
irfan
Jul 25, 2012 10:42pm
m not surprised that indians are doing far better in the world ..
Abdul
Jul 25, 2012 11:18pm
A good one. However it really doubt that without an active Public sector support, this complex division of technology could make significant inroads in Pak. Still as writer suggests, private sector can provide for those in search of excellence in this field.
Saracen
Jul 25, 2012 11:32pm
An absolutely stunning article with so much insight and vision. If only Pakistanis like this could be found in Pakistan with the unadulterated support of the authorities. He has really tapped into the new growth area for Pakistan. If only the criminals who run Pakistan had any sense they would give this man overarching powers to bring this about. Well done this man!
Darwin
Jul 26, 2012 12:03am
Interesting.
BNS
Jul 26, 2012 12:31am
Murtuza, I read the article first and guessed that it is authored by you. It had your hallmark. Good job. You just dont bring up something you talk about the potential solution as well. If someone does not agree to your solution thats another point but on your part you come up with a complete proposal not just the tag line. Thank you for doing this service to your countrymen. Now it is on us, others, to evaluate your proposal and to exploit it, or not. Again thanks very much.
Anup
Jul 26, 2012 01:10am
Thanks Dr. Murtaza for valuable info on this developing situation. I am sure it is going to be great start for many aspiring Engineers, statisticians and computer programmers.
Zaim Raza
Jul 26, 2012 01:57am
very good article about big data.Murtaza, could you please give applied example regarding analytics and importance of analytics for mid market.specially if you give the example from microsoft product that would be great for me because i am working on microsoft product. cheers, Zaim Raza.
Ehsun
Jul 26, 2012 02:14am
Informative and with references; excellent post.
Ali I.
Jul 26, 2012 04:57am
Hi Dr. Murtaza, Thanks for writing this article and sharing this new idea. I am an entrepreneur and was already researching for another idea to work on in a new startup. This one definitely looks promising. I will further research on this. Thank you and please keep writing.
Al
Jul 26, 2012 05:48am
As Thomas Friedman has written in his World is Flat and NY Times articles, computer/network can be performed from anywhere in the world. So, those numerate foreign workers can do the analytics work without having to move to North America or anywhere else, hence avoiding any political interference from host countries. No?
Asif Khan
Jul 26, 2012 07:41pm
Pakistan was ahead of India only during pre partition years. Peshawar and Lahore ranked alongside Delhi and Kolkota in terms of intellecual capital. Eversince the partition there is a gradual decline of educational standards. The noted magazine Economist published a supplement on Pakistan and they compared pre partition to post partion. The percentage of people who could write english in peshawar prepartition was around 38% and much higher for Lahore and Karachi. Now, the estimate around 3% in Peshawar. It could be due to various factors but generally should hold good. Some other extraneous factors are responsibe including people who created high value jobs, and the only validitity in your statement is Yes, Zia made it much worse---setting up madrasas and accelearting the slow decline.
Atul
Jul 26, 2012 06:07am
Mr. Murtaza, while i agree on the importance of big data and analytics, i cant dis-agree more on your statement that youths are wasting time making cold calls from call centers and selling undergarments for consumers in the west. Big data and analytics opportunities dont fall from the sky. Take India, we started with cold calling, graduated to data work in BPO...this work was the foundation for us getting access to data for the corporations for whom we work. So its not a surprise to me, who works in this field that 4 of the top 6 cities where analytics is most popular are in India. Similarly, if the chinese youth are manufacturing undergarments today- that is the essentially building block for them to graduate to become tomorrows Armani designers. In other words, developing economies ought to walk before they run.
Muhammad Arif Saleem
Jul 26, 2012 06:17am
Good Haider, you always come up with new perspectives
Nut Case
Jul 26, 2012 07:23am
Hi Murtza, Very nice article , In fact data analytics is already big here in India. I would like to point out that jobs going away due to call centre outsourcing is a caricature created by politicians / media in USA. If you look at volumes in India most of the software business is from domains such as VLSI , System Integration , ERP , DB and now data analytics. Having said that training manpower for data analytics is very different then say for Database development or for writing code for BFSI segment. Manpower for former was provided by short term course mean to cross train basically any gradudate where as for data analytics you need atleast a seed manpower with masters / P.hd level education who can visualize the problem and then a team around them of Engineering business graduates who can use the avialble tools and packages. Here in India , Indian Statistical Research Institue , IISC and other universities have created a seed base since past many decades. Though Engineering graduates study statistics for couple of semisters and also MBA and Executive programs have atleast one course on statistics it is mostly forgotten by the time they pass out of the couse.
aamir
Jul 26, 2012 10:24am
India is doing much better and we are busy making and breaking terrorists. This difference was created in the last 30 years (Zia take over). Before that, Pakistan was a head of India in science and technology.
Nut Case
Jul 26, 2012 08:30am
Atul, Would like to correct that cold calling and BPO work came much later after 2000 , It started from software development. Today also BPO and Call Centre is smaller part of the pie but it gets disproportionante flack due to social impact in west. India's IT revolution started in 1969 when FC Kohli started TCS which is 10 billion $ company today. A host of companies mainly in to outsourced software set shop in 80s. Then in 1984 Texas Instrument was first MNC to set shop in Bangalore for captive software development unit followed by Motorola in 1991.
nyla
Jul 26, 2012 08:52am
hmm really nicely written, though our country is slowly coming towards this side but of course it will take some time though its being done and is an attractive stuff
Venkat
Aug 05, 2012 10:06am
Such a coincidence, I started reading a book on Hadoop by Tom White. Very, very interesting. I am a big fan of R. Lot can be acheived in R for free. You have so many libraries, you could easily be confused, what model you want to appliy for a given issue. The other issue, is when you have to handle large data, you might encounter memory issues.
Farooq
Aug 20, 2012 06:12pm
Interesting one. My hunch says that one day this data machine(Internet) will destroy everything on earth.