How to Manage Research Data with a Limited Headcount

How to Manage Research Data with a Limited Headcount

If an HE institution loses staff in the middle of a project, it can be hard to replace them. If you’re struggling with growing research data volumes and less manpower to manage them, this blog can help you take back control of even the largest research data sets.

With large volumes of unstructured research data, and the growing need to share this securely among geographically dispersed campuses, higher education (HE) institutions already face tough data challenges.

From our recent work with the University of Dundee, we saw another challenge that has the potential to impact on HE data centres around the world: a lack of headcount.

In the following video James Martin talks further about the big data solution at the University of Dundee

The recruitment challenge in higher education.

People leave jobs. That’s a fact of working life. But when projects are funded between a particular start and end date as they are in higher education, finding a replacement for a departing staff member can be tough.

After all, if you were offered a job that could only guarantee pay for three months – up to the project’s end date – and one that would support you for years into the future, which one would you choose?

If an HE organization finds itself in this predicament, it can put a lot of pressure on the data centre. Each employee can only really manage so much data at any one time, so if you have dwindling manpower and several petabytes of data to manage, things can start looking bleak for your backup and data management processes.

The impact of reduced headcount

HE institutions have to be serious about their disaster recovery, and that means backing up regularly. Anything less frequent than a daily backup puts essential research at risk – but when you only have a handful of employees trying to manually backup petabytes of data, those backup times can slip.

The moment it becomes impossible to back up everything in a 24-hour period, large portions of data get left out each time, and the entire backup process is put at significant risk.

There is also a cost issue that comes with decreased manpower. HE institutions with a mature data management strategy will be moving research data to different storage tiers based on its usage: regularly used data will stay on faster, more expensive storage, while cold research is moved to a more cost-effective tape array.

Doing this manually with large volumes of data becomes a huge challenge when you have fewer staff – resulting in missed opportunities to optimise storage resources, and cut data management costs.

Solving the headcount challenge with Spectrum Scale

HE organizations unable to hire additional staff to ease the data management burden need a way to streamline and automate key data processes.

Enter Spectrum Scale: a solution that lets administrators automate essential data management and backup processes, helping reduce the administrative burden placed on members of staff.

Spectrum Scale lets users set parameters that automatically tier data to a new type of storage. So, you could set it to move any data that hasn’t been looked at in six months to tape. And to move it back to flash or disk when that data is requested again. This saves your staff from manually moving storage between tiers, helping you do much more (and save more) with fewer people.

As a global, parallel file system, Spectrum Scale also accelerates backup and DR processes. So even if you are managing many petabytes of data, backups can still be achieved in a matter of hours rather than days – and with fewer people than before.

More importantly, as a software-defined storage solution, it can work with any combination of storage arrays from any vendor, and doesn’t require any costly, up-front investment in new storage hardware. It even allows HE institutions to seamlessly scale data to cloud tiers, aiding existing and future cloud strategies.

Spectrum Scale – one way to overcome resource hurdles

As funding and talent become key issues for universities and other HE institutions, and as research data volumes continue to grow, smart new solutions will be needed to tackle the inevitable burden placed on IT.

Spectrum Scale is one way of overcoming these hurdles, but it’s important to note it isn’t a one-size fits all solution. A partner with rich experience in your particular sector will be able to advise you on what solutions are the best fit for your data centre, and how you can get the most out of them.

To learn more about the big data challenge, and how to overcome it, take a look at our eBook: Managing Data at Scale.

Or, learn more about how HE institutions are successfully managing petascale data environments in this blog.