What’s the deal with “Big Data?” Part 1

Part I: The History

Big data. You hear about it at every turn. From Google to Facebook, Amazon to Twitter, it appears that everywhere you look, it Terminalsseems that someone is tracking your online activities. There is no doubt that this information can be very valuable and, in some cases, very embarrassing to the person involved if it were to come to light. Few people realize just how much these companies can learn about you just by watching what you click, what you search and where you go on the Internet. Further, there is not a lot of protection for the data that is collected. For many, it is no wonder that this is a great cause of concern.

Yet, there can be a very positive side to “Big Data,” particularly in the realm of higher education.

Before we get to what “Big Data” is today and how it can be used positively in higher education, let’s first take a look at history and see how we got here:

In the not-so-distant past, once computers were first created, databases were soon developed to manage transactional operations. Instead of using vast numbers of people to do such things, we found that computers could do these tasks not only more efficiently, but more accurately. The cost savings were also not insubstantial. For example, in the higher education context, we developed systems to manage the registration for classes. Instead of doing this task by hand as was done in the past, we created data systems to manage these tasks. Over the y ears, these systems were aggregated and bundled into cohesive wholes and marketed together under various brand names such as Datatel, Banner and PeopleSoft. In time, these new systems were collectively named ERPs or Enterprise Resource Planning systems. This is how all these systems are categorized today.

Almost as soon as these systems came into being, people began to want to generate reports from them. This turned out not to be so easy. A system that was designed to run operational transactions efficiently and accurately might not have data in a format that is conducive to longitudinal reporting.

Thus was born the concept of a “Data Warehouse.” In general, the purpose of a data warehouse is to aggregate, normalize and transform data from various transactional system sources and move it into an authoritative central source to make reporting easy. To turn that complex definition into something more like English, it means, at least in the university context, that the information is taken from all these various transactional systems and then reformed. This reformation is done in such a way that it now makes it possible to run simple queries or searches across the data and get meaningful results without the need for complex programming. In other words, it makes the data easily accessible. At least it does in theory.

With such a promise, many universities in the 1990s began ambitious projects to develop data warehouses for reporting purposes. Many spent millions on the endeavors with the idea of getting great insight from the transactional data that they had been collecting for years. Unfortunately, many of these projects did not quite turn out as planned. Indeed, many turned out to be absolute disasters.

The reasons for the failures were many. However, amongst the biggest reasons were the sheer number of different data sources that needed to be brought together. At the time, universities often did not yet have aggregated ERP systems, but a collection of different, smaller systems that were linked together through something called “batch processing.” This meant that normalizing the data (making the data match across systems so that it meant the same thing regardless of the system being used) was incredibly complex. Because the definitions of data are often used in reporting, changing that definition to get it to match across systems was no easy task and often took a great deal of negotiation amongst various departments. For example, what is an applicant? For a university, that definition can have huge consequences.

Another challenge was that the technology used to build data warehouses was relatively new at the time. The tools, processes and procedures to build these systems were simply not all that well developed. In addition, in a lot of cases, university IT departments were ill equipped to develop such systems with the often limited resources available to them.

Thus, a great many of these data warehouse projects were abandoned or dramatically scaled backMoney Pit. The promise that they offered remained just that, a promise. Many an IT career was derailed because of the money-pit many of these projects had become. Thus for years, at any gathering of university IT professionals, you could still illicit cringed looks with the mere mention of the term “data warehouse.”

But, as is the case with many things in technology, what might seem like an old or even bad idea, often gets reborn in some new way. So it has come to be with data warehouses. Today, they have been redefined and reconstituted as “Big Data” with all the same promises and opportunities and yet with some of the same challenges too. In some ways, “Big Data” may even have new, more significant issues.

In Part II, we will take a close look at this modern take on the data warehouse and how “Big Data” can make a large, positive difference in higher education while also presenting some daunting challenges.

Photo Credits:

Photo 1:

http://www.flickr.com/photos/8399025@N07/3300683055/in/photolist-62ES5x-4xjF6L-9tmQrm-4xjZVu-6ZhcKK-88XKCJ-717Wbe-9tiNNe-51CVki-51CVUg-9tmMqQ-aBchPi

Photo 2:

http://www.flickr.com/photos/60057912@N00/4404695769/in/photolist-7HedAv-uvWfh-51Ys8K-7bYqCN-2kmuW2-uvWdP-uvWeP-uvWd9-7vpAUr-9A37wt-78yDYU-e6BPLD-dKLm3U-d3F8j3-d3FcJy-d3F947-d3F7eE-d3F1Ej-d3F9tG-d3F2T5-d3F8HU-d3F5BG-dqMxba-2FU8tq-jXtiMZ-daWsnN-cqkBfw-9MKokD-dqMx4V-dqMGCA-dqMwPT-dqL9zJ-dtzYYF-dqMwT6-523CXd-523KLC-51YrzP-51YtC6-51YAhX-51YqLv-523Goy-51YzJB-8w7E4V-51YwSH-51Yqh6-51Yy5R-523Eg1-523PUj-523HVb-523Hoj-51YzmB

For the Love of Technology…

loveIt is that very special time of year… Yes, Valentine’s Day is here and people across the world are thinking and talking about love. Taking that as a cue, I thought it would be a good time to write on what I love about technology. While I probably won’t be sending my computer a valentine per se, believe it or not, there is actually quite a bit to like!

If you like change, then you have to love technology. Looking back over my lifetime, it is amazing to see the pace of change. Indeed, the pace only seems to be getting faster. Some technologies that were a staple of my youth (like video cassettes) are basically gone out of existence. In its place, whole new products that were just a dream to science fiction writers a few years before have come into being. Everything from HD TVs to cell phones, from Facebook to Google were just a dream not that long ago. Indeed, the desktop computer itself is an invention of my lifetime.

We in technology have a name for the basis of all this rapid change. It is called Moore’s Law. Moore’s Law is named after Gordon E. Moore, the co-founder of microchip manufacturer, Intel. In a paper he wrote in 1965, Moore noted that the number of transistors in integrated circuits doubles approximately every two years. Over the last half century, he has turned out to be exactly right. From that time until this, what we can do at the very basic core of technology has increased exponentially. Indeed, it is directly because we can fit more and more tiny transistors onto microchips that everything you see today has now become possible. The more tiny we can go, the bigger things we can do. It may sound ironic, but it is absolutely true.

EniacTo put this into sharp context, computers that used to fill rooms can now fit in your smartphone… multiple times over. To me, that is just absolutely amazing. I remember just how primitive the graphics were on my first video game system, the Atari 2600. Yet, look what we can do today! When Steve Jobs first demonstrated the iPhone, I was literally stunned by what I was seeing on the screen. It almost looked like magic, but, in fact, it was simply what had become possible all because of Moore’s Law.

While some have said that our capability to double the number of transistors on microchips will soon come to an end, this prediction has yet to materialize despite being predicted multiple times. That said, it is quite true that we will face the ultimate limits of physics. It is hard to imagine transistors smaller than the size of an atom. Of course, as a kid, it was hard to imagine all we have accomplished today despite a vivid imagination. Thus, future breakthroughs could continue Moore’s Law indefinitely.

So, what does all this have to do with IT in an educational setting? Frankly, I would say everything.

It means that what we do today will not be what we do tomorrow. It means those of us who work in the field of IT have to expect disruptive technologies to come along. When they do, we need to help our institutions understand the context, to discern between what might be a fad and what might be truly transformative. While this sometimes might feel like we are walking on quicksand, I find the experience exhilarating. We are always having to learn new things and approach matters in different ways. It makes for a job that is never boring.

Yet, despite all the new technologies and the change that comes with it, there is actually a constant. This constant is another aspect of what I do that I love so much. Despite the never ending change, there is a highly developed methodology or pathway in which to look at and implement new things. It is this methodology that has become the core of what we call MIS (Management in Information Sciences) that now guides everything that we do. In this way, we can bring a little control to what can sometimes admittedly be chaotic change.

What exactly is MIS? That is a blog for another day.

In short, I love what I do. There is no other way to put it. I love all the new technologies and the opportunities and, yes, challenges that come with it. However, I also love the fact that there is a way to manage all this correctly. It isn’t just chaos. It is, at least, controlled chaos. How much more fun could a person have?

So, in the end, maybe it isn’t such a crazy idea to send my computer a valentine.

Photo Credits:

Photo 1:

http://www.flickr.com/photos/37150010@N03/4188507599/in/photolist-7o8cog-7CZKLD-7CZKU4-7D4B19-7CZM3e-7D4B8J-7CZMwz-7CZKs2-7CZLji-7CZLav-7CZLSk-7CZL1k-7D4A3d-7CZMGe-6WTrX2-6PFUkz-5UV17P-9vd9C7-xErLq-c4Brk-9m6bZ-3cHwYg-549vCt-5DbjPo-sY5rX-eELYWn-6cvQPQ-aDBqtY-2gcf8B-7C4NCy-9PW1cc-iEcWE8-BXwtj-8y49oC-a5kEPj-eg4Top-6oy3FU-84mgSf-84mgYU-6ows1C-7Cr21R-4XbHxb-ifJ24W-ifHG4k-2e4Cyj-3GnGiw-8yZ2bL-gM2k5X-4xF9Hy-9EEkuc-5V5KcZ

Photo 2:

http://en.wikipedia.org/wiki/File:Eniac.jpg

“Arrogant Humility”

As members of my team handshakeknow all too well, I like to describe myself as something I call “arrogantly humble.” At first breath, that phase seems completely contradictory. Traditionally, it doesn’t make any sense. How can you be arrogant and humble at the same time?

Yet, for me, this seemingly nonsensical phrase has become one of my very core values. Indeed, it is a lesson that I hope to impart not only upon my team members but also those that call on me for mentorship. It is a phrase that I constantly repeat to myself as I go about my day and, as a result, it has become a guiding principle that helps me in almost every situation.

So, what is “arrogant humility”?

A bit over ten years ago, I started teaching Management in Information Sciences (MIS) for Emory’s Rollins School of Public Health. The goal of the class was to teach students how you go from various technology parts to systems and then from systems to infrastructure. Public Health is the study of disease prevention and a lot of the technology used in the discipline is unique to the field. Another challenge in Public Health is that, unlike medicine, funding can be hard to come by. So, you sometimes have to be very clever in the way you go about doing things. You have to think outside the proverbial box if you actually want to get anything done.

I wanted to teach these skills to my students.

As I thought about it, I found the need to be flexible sometimes conflicted with the culture that had been developed in the field of information technology (IT) over the years. Do not get me wrong: a lot of the processes and procedures that IT has developed are really great. If properly followed, they almost guarantee you get projects done on-time and on-budget. The core intent is to prevent the absolute disasters that IT projects can become if they are not managed properly. Who hasn’t heard horror stories were a company spent billions on some software project only to throw it all away?

However, these processes and procedures can become the end upon themselves for many of the people who work in IT. It was almost as if they are just painting by the numbers. Once the project was outlined, that is what they did. This was OK if circumstances did not change or unexpected issues did not arise. However, with modern projects, this never happens! No project these days gets done without facing a lot of twists and turns before you get to the end. It is just the nature of things.

Yet, with the rise of the Internet, all of this uncertainty, all of the unknowns became the norm with IT projects and it was driving IT people absolutely nuts. The ground seemed completely unstable. Where was the certainty? stressWhere was the planning? And, I have to admit, fifteen years ago when all of this was really coming to the fore, I was one of those IT people. It was really stressful.

Still, the writing was on the wall: the services we in IT were offering would no longer meet our community’s needs unless something changed. We needed to be flexible. We needed to be open to change. We needed to be agile. We had to look at the unexpected requests not as a burden, but an opportunity. We needed to listen. However, we also needed to do all this while following best practices the IT field had developed. After all, those processes still worked if they could be matched to needs.

When I thought about it, in order to do all these things, I felt I had to be different. I had to reconcile these seeming contradictions. This is what led me to “arrogant humility.”

So, exactly what do I mean?

To date, I have been in the field of higher education IT for over 20 years. I have seen, done and experienced a lot in all that time. I have taught and mentored countless team members and students. I know the processes, the procedures and the rules.  I think I know how to do what I do very well. And yet, through that knowledge has come a deep sense of knowing what I don’t know. For example, I have team members whose technical skill is way beyond and more current than my own. There are people here who have talents that I can only admire and wish I could have. Indeed, I view my job now as an enabler: I help my team members be successful by giving them the resources and the support they need to get the job done. I want people to work here who know more than I do. That just makes us better!

But, the humility goes further than that. It also means listening to our community. It means hearing what they say and making those things important. After all, we in Technology Services are a service unit. We are here to meet the needs of our faculty, staff and students. That sometimes means you need to hear things you may not want to hear and make adjustments that are difficult or, sometimes, even painful. It may mean that our best laid plans get tossed out the window. For me, that is absolutely OK if it helps us do what needs to be done for our community.

On the flipside, because of the constant flux that is the nature of technology, we sometimes have to be the purveyors of change even if it is unwelcome. In those cases, listening, communication, creativity and flexibility are even more important. After all, you may not be able to revert to the past, but you can often find that you can make things better in the newly created present. Even if you can’t, people appreciate when they are heard.

So, in the end, what is “arrogant humility”?

I think it is having the confidence and knowledge that you can work through almost anything that happens and yet knowing that you don’t and can’t ever know it all. With that knowledge comes openness and a willingness to listen and incorporate other’s ideas into your own. Once you open yourself up to what others say, you might find that you can actually be far more creative and flexible in meeting needs than you ever thought possible. I find myself welcoming that even if that means best laid plans get changed.

And, in the end, that is what we are all here to do.

Photo Credits:

Photo 1:

http://www.flickr.com/photos/57567419@N00/7496765660/in/photolist-cqsTfm-6rUPaH-8R6pwz-8R9wEE-8R9wHo-8R9wCE-8R6ptF-4HJx7M-5E1X1j-8Ree5g-8RedFB-8RhmpC-8RedC6-8RedUr-8Reeut-8zeG3z-6gG1Uf-oLV21-6FADUs-a9GfJu-6dEndk-6dJvFS-6dJvJw-6dEnfZ-6dEniD-686uvZ-7Pwbtm-yXKoU-817VX2-4uezqw-9B8maB-7VHu5X-5gBQyo-5teuPz-faRfEy-3v2wux-3v2xfn-3v2w6R-3v2wRF-wWcfd-7MnYfa-wWcrU-7pPKxE-78y5gh-6g4KNY-8H4CYm-7K4pkQ-3WeQ3G-4fgaGY-4PowDs-2LMQMc

Photo 2:

http://www.flickr.com/photos/22964099@N05/2204059683/in/photolist-4mLoa2-b4riTB-egdAKS-4bee1t-8GRidY-6cTJYv-9BMsED-isQYLw-isQHWc-9jctx9-dxi7Hk-sZDh8-cEJPoW-fyLg6i-f7kW2b-aajmg4-auQjtm-auQjsf-auQjsU-86AtMA-3g3Xrv-afDxkr-6KU8oq-3g8ehU-4t7wtb-93R2VM-cwDpgy-eBGJZh-6GvsEd-8X29ZM-7zexHP-5S6zTR-9dV6gQ-77RspM-fNUBFd-hHMmHy-bcPivt-dAM1Zg-dAM1wa-dAStb9-dAStfA-dAStws-dAM1Dr-dAStU5-dAM1PM-dAM1z8-dAvMhv-dAStCS-dAStPE-dAStVY-6qpvod

 

Winter is coming…

winterWinter is coming… winter break that is. And with it, many in our community will soon be taking off for a much deserved break to hopefully recharge and renew for the up-coming spring semester.

However, for those of us who work in academic information technology (IT) units, winter break is actually a very busy time. It is one of only two periods (the other being summer) during the year when we can make significant system updates, do campus-wide upgrades and replace core technology without disrupting the community during an academic semester. After all, when classes are taking place, the last thing we in IT want to do is have something we do disrupt the key reason we are all here in the first place, to give students a quality education.

So what sorts of things do we do during winter break? A lot. What most people do not realize is just how complex back-end IT operations are to support. After all, what happens on the back-end directly affects what goes on with your desktop, laptop, iPad or Android device. They are intimately interconnected.

Here at Puget Sound, for example, we have literally hundreds of servers, most virtual these days, with many hundreds of terabytes of raw disk storage to support what you do on daily basis. So, for example, when you click to access our administrative system (PeopleSoft/Cascade), you actually might be touching literally dozens of different servers all doing different things to provide you what you see on your screen. It seems so simple, but it is actually very complex.

And just as all these servers working together might seem complex, the individual servers themselves might need to be built using a multitude of different technologies in order for them to do what we need them to do. We in IT call these layers. In a way, they are like a cake. Each layer is needed to support and glue together what comes above it. If something goes wrong at the bottom, the whole thing can quickly fall apart.

So, in a way, all these things working together is like a well-choreographed ballet. It looks magical if everything is going correctly. However, if even one part does not work quite right, the whole thing can fall apart. No one wants that to happen.

Thus, for us, winter break is a rare opportunity when we in IT can take the time toSONY DSC make sure all those parts are working correctly and will continue to do so through the next semester. We might do things such as “patch” (install fixes) for our servers and network. We will upgrade various software to keep up with the latest supported technology. We will replace aging machines with newer, more efficient ones. We might even entirely replace a service with a brand new one such as you are seeing with the Cascade to PeopleSoft migration.

Still, we try to be very careful doing all these upgrades too. We do not want to change so much that we actually create problems down the road. That means we have to test what we do to make sure that all the systems working together continue to produce that well-choreographed ballet that you, our community, expect when you return for the new semester.

So, enjoy your time away! We hope that each and every one of you come back refreshed and ready for the next semester. While you are gone, please know that we in IT will be busy here working for you to make sure your new semester starts off right. For us, having the time to get everything right is the best gift of all.

Change: It is the one thing that is constant…

Change. In the world of technology, it is an absolute constant. Over a five year period, technology that was once the hot new thing can suddenly be obsolete. This is true even in our own lives. Just recently, Blockbuster, a staple of the American landscape since the 1980s, announced that they were closing literally all of their domestic stores. Why? ThumbnailBecause media, whether it be music, movies or even first run TV shows, are all moving to an online streaming model. Few people buy physical media like CDs, DVDs or Blu-Rays much less rent them. Why run down to Blockbuster when you can simply rent from the comfort of your couch?

Universities are not immune to technological changes and, as a result, have experienced similar dramatic changes over the last few years. We have seen the rise in higher education of “big data” and analytics with a strong focus on outcomes. Students, faculty and staff are literally bringing their own devices (a trend called BYOD) in ever greater numbers at colleges nationwide. Indeed, the average student now has three wireless devices all connected simultaneously to the network. The media has documented the rise of massive open online courses (MOOCS) and the push for digital collaboration amongst colleges as ways to help make higher education more affordable. With these and many other trends, many say that they have not ever seen so much technological change in higher education before. No doubt, the resulting strain of all this rapid change has been difficult for everyone in higher education to manage.

All this change is no less true here at Puget Sound. While we are a liberal arts college that places great value on small classes in an in-person environment, technological changes over the last few years have been dramatic. We have seen:

  • The university’s Internet bandwidth quadruple in less than four years to meet needs.
  • The need to replace Cascade with a new ERP whose core was not in danger of being end-of-life and yet also provided significant new critical capabilities like analytics to the university.
  • We rolled out new services like vDesk, Moodle, Mahara, SoundNet and PrintGreen.
  • We are working to deploy a new library, medical record and admission system.
  • We just collaborated with Communications to deploy the new website.
  • We completely rebuilt and redesigned the backend server infrastructure to not only be more resilient and cost effective, but far more energy efficient.
  • We rolled out a new phone system to replace our legacy end-of-life solution.
  • We completely reorganized Technology Services (TS) with the goal to deliver better services to you. (Does anyone remember Perceptis?)
  • And much, much more…

Yet, with change comes disruption. While we in TS work very hard to minimize the effects of change on our community, sometimes there is nothing we can do to avoid the disruption that comes with change.

For example, in the case of the ERP replacement, we know that this conversion has been difficult. This is true for all universities that have to do such a thing as we here at Puget Sound have had to do. Further, big conversions like this just take time. While we are now running all our core systems on the new ERP, it will be some time before the system gets to where it was with Cascade both for our users and for the people who have to support the system. The good news is that we picked a system with enormous potential, one that has capabilities that few other universities like Puget Sound have. In the complex and competitive market that is higher education today, this new system will serve us well.

In the end, we in TS are a service unit. Our ultimate goal is to do what we can to further the teaching mission of the university. With all this technological change and with the disruption we are now seeing in the higher education market, we are doing our best to help give the university the best tools to not only compete, but also provide our faculty the resources to give our students the most relevant liberal arts education that we can. It is quite the task!

However, if we have to change, we are also taking the time to not just replace what we have with something equivalent. Instead, we are making the effort to think strategically. How can we use this required change to make things better? After all, we want to make the disruption that comes with change worth it.

We in TS appreciate and thank you for your patience and support as we have undertaken all these tasks. Indeed, your support is what has made all of this possible. After all, change is never easy. We appreciate your trust in knowing that we do not make changes just for change’s sake. Just as when you build a new building or a new pathway on campus and experience all the disruption that comes with it, we have faith that, in the end, you will see that what we have been doing these last few years was all worth it.